The Same Webhook Can Knock Twice: Test an AI-Written Consumer

Build a local webhook lab for raw-body signatures, replay windows, duplicate delivery, out-of-order events, and one-time side effects.

Ask an AI coding assistant for a webhook consumer and it can produce a plausible route in seconds: parse JSON, switch on an event type, update a record, send a message, and return 200. The happy-path demo may work on its first request.

That is not the difficult part.

A real sender can deliver the same event more than once. A valid retry can carry a new attempt timestamp or signature. Two different events can refer to the same business object. Events can arrive out of order. Your process can stop after changing the database but before acknowledging the request—or after calling an external service but before recording that the call succeeded.

Those are not obscure edge cases. They are the delivery model.

The useful review artifact is therefore not a polished route handler. It is a deterministic fixture bench that asks whether the consumer remains correct when delivery is duplicated, delayed, reordered, interrupted, or ambiguous. The goal is not to prove “exactly once.” A generic HTTP consumer cannot promise that across every database and external service. The goal is to produce duplicate-tolerant, convergent effects with evidence.

This article uses only fictional payloads, fixed test credentials, and illustrative storage interfaces. It does not connect to a provider, reveal a real secret, exercise production traffic, or certify a production integration.

Freeze the Sender Contract First

“Webhook” describes a delivery pattern, not one universal protocol. Do not let generated code infer a signature format or retry policy from another provider’s example. Stripe and GitHub make the mismatch concrete.

Contract question	Stripe	GitHub	Bench consequence
What bytes are authenticated?	The signature covers a timestamp and the actual request body. Stripe requires the unmodified raw body and recommends its official libraries.	X-Hub-Signature-256 is an HMAC-SHA256 signature over the unmodified payload body.	Capture bytes before JSON middleware and invoke a provider-specific verifier.
What identifies a repeat?	Stripe recommends recording processed Event IDs. It also notes that separate Event objects can sometimes represent duplicates, for which object ID plus event type may matter.	X-GitHub-Delivery supplies a GUID. GitHub’s redelivery examples state that the GUID remains constant across redeliveries of the same delivery.	Define the event key from documented provider semantics. Do not deduplicate every provider by one guessed field.
What retry horizon exists?	Live automatic delivery can retry for up to three days. Manual resend is available for up to 15 days in the Dashboard and 30 days through the CLI.	Failed deliveries are not automatically redelivered. Current delivery records can be manually or programmatically redelivered for three days.	A five-minute signature window must not become a five-minute deduplication-retention rule. Recovery also differs by provider.
Is order guaranteed?	No. Stripe advises consumers to handle missing objects and retrieve them through its API when appropriate.	GitHub says deliveries can arrive in a different order from the underlying events.	Arrival order is not business order. Test stale events explicitly.
How quickly must the endpoint answer?	Stripe says to return a successful response quickly, before complex processing.	GitHub records a failure if it does not receive a response within ten seconds.	Verify and durably receive first; perform slow work asynchronously.

The community Standard Webhooks specification offers a coherent profile using webhook-id, an attempt timestamp, and the raw payload in the signed material. It is useful reference material, but it is not evidence that an arbitrary sender implements that profile.

Likewise, CloudEvents 1.0.2 says a conforming producer makes the combination of source and id unique for distinct events and may retain that pair for a duplicate resend. That rule applies when the source actually emits CloudEvents. It is not a fallback interpretation for unrelated JSON.

For each endpoint, create a small contract sheet containing:

• the exact signed bytes and signature header;
• the trusted key source and endpoint scope;
• event, delivery-attempt, and business-object identifiers;
• automatic and manual retry behavior;
• response timeout and success codes;
• ordering or version guarantees;
• schema and API version;
• supported event types; and
• the recovery and reconciliation path.

If a generated consumer cannot point to this sheet, it is still guessing. Keep a dated link to the relevant provider documentation beside the adapter tests so a future reviewer can see which contract the implementation intended to follow.

Keep Three Guarantees Separate

Webhook implementations often collapse three independent controls into one cache entry. Keep them separate.

Attempt authenticity and recency answer: “Was this particular HTTP attempt signed under a trusted provider contract, and is its timestamp acceptable?” A five-minute tolerance can limit replay of an old signed attempt. It does not prove that the underlying event is new. A legitimate provider retry can carry a new attempt timestamp while describing the same event.

Event deduplication answers: “Have we durably accepted this documented event key before?” Retention needs to account for provider retries, manual redelivery, audits, and the application’s recovery model. Stripe’s documented manual resend horizon can reach 30 days. A short cache can be an optimization, but it must not become the only guard protecting a lasting business effect.

Domain idempotency answers: “Can this business transition or benefit happen twice?” A unique event ID cannot, by itself, prevent two distinct events from granting the same credit, shipping the same order, or sending the same entitlement. Protect the business invariant with legal state transitions and unique effect keys.

These controls operate on different identities and clocks. An attempt timestamp is not an event occurrence time. An event ID is not necessarily a resource ID. A resource ID is not a delivery-attempt ID. Put each value in a deliberately named field instead of a generic variable called id.

Build a Synthetic Delivery Fixture

Start without a tunnel, provider account, or production payload. Define a fictional sender called BenchPay. Its test-only signature covers an event ID, attempt timestamp, and exact raw body. This resembles common designs but deliberately claims compatibility with none of them.

import {
  createHash,
  createHmac,
  timingSafeEqual
} from "node:crypto";

const FIXTURE_SECRET =
  Buffer.from("fixture-only-secret-never-use-in-production");

export function makeDelivery({
  eventId = "evt_fixture_0042",
  attemptTime = 1787637600,
  rawBody
}) {
  const signed = eventId + "." + attemptTime + "." + rawBody;
  const signature = createHmac("sha256", FIXTURE_SECRET)
    .update(signed)
    .digest("hex");

  return {
    headers: {
      "x-bench-event-id": eventId,
      "x-bench-attempt-time": String(attemptTime),
      "x-bench-signature": signature
    },
    rawBody: Buffer.from(rawBody, "utf8")
  };
}

export function verifyFixture({ headers, rawBody, nowSeconds }) {
  const eventId = headers["x-bench-event-id"];
  const timestamp = Number(headers["x-bench-attempt-time"]);
  const supplied = headers["x-bench-signature"];

  if (!eventId || !Number.isSafeInteger(timestamp)) {
    throw new Error("missing fixture metadata");
  }
  if (!/^[0-9a-f]{64}$/.test(supplied ?? "")) {
    throw new Error("malformed fixture signature");
  }
  if (Math.abs(nowSeconds - timestamp) > 300) {
    throw new Error("stale fixture attempt");
  }

  const signed = eventId + "." + timestamp + "." +
    rawBody.toString("utf8");
  const expected = createHmac("sha256", FIXTURE_SECRET)
    .update(signed)
    .digest();
  const received = Buffer.from(supplied, "hex");

  if (received.length !== expected.length ||
      !timingSafeEqual(received, expected)) {
    throw new Error("invalid fixture signature");
  }

  return {
    provider: "benchpay",
    endpointId: "checkout-test",
    eventKey: eventId,
    payload: JSON.parse(rawBody.toString("utf8")),
    payloadHash: createHash("sha256").update(rawBody).digest("hex")
  };
}

Use a body whose business effect is easy to count:

{
  "type": "invoice.paid",
  "invoice": {
    "id": "inv_fixture_17",
    "version": 3,
    "state": "paid"
  },
  "effect": {
    "kind": "grant_receipt",
    "key": "receipt:inv_fixture_17"
  }
}

Now alter only the JSON whitespace or key order after signing. The verifier should reject the changed bytes even though parsing both bodies yields equivalent objects. That catches frameworks which parse and reserialize before verification. Also send invalid hexadecimal, a missing header, an attempt just inside the tolerance, and one just outside it. Every failed verification must leave the database and effect ledger unchanged.

In provider-facing code, replace verifyFixture with an adapter that follows the provider’s current documentation and preferably calls its maintained verification library. A useful adapter accepts headers, rawBody, and receivedAt, then returns normalized fields such as provider, endpoint, event key, event type, subject, and any contract-defined version.

The adapter must not invent missing ordering fields or pretend every provider signs the same material. It should also isolate endpoint-specific keys. Stripe documents that registered endpoint secrets and local CLI secrets differ, and test and live configurations must not be treated as interchangeable.

Make Receipt Durable Before Acknowledgement

A queue is useful only if enqueueing is durable before the endpoint acknowledges success. “Return quickly” should mean:

1. preserve the raw bytes;
2. verify the provider-specific signature;
3. validate a bounded envelope;
4. insert or identify a durable inbox record;
5. commit;
6. return the provider-appropriate response; and
7. process business work asynchronously.

Acknowledging before step five creates a silent-loss window: the sender believes delivery succeeded, while the consumer has no durable record. Performing slow email, payment, or fulfillment work before the acknowledgement creates the opposite problem: an otherwise valid delivery can time out and be repeated.

A compact relational bench might use:

CREATE TABLE webhook_inbox (
  provider       TEXT NOT NULL,
  endpoint_id    TEXT NOT NULL,
  event_key      TEXT NOT NULL,
  event_type     TEXT NOT NULL,
  payload_sha256 TEXT NOT NULL,
  received_at    TIMESTAMP NOT NULL,
  status         TEXT NOT NULL DEFAULT 'pending',
  attempts       INTEGER NOT NULL DEFAULT 0,
  last_error     TEXT,
  PRIMARY KEY (provider, endpoint_id, event_key)
);

CREATE TABLE invoices (
  invoice_id      TEXT PRIMARY KEY,
  state           TEXT NOT NULL,
  last_version    INTEGER NOT NULL
);

CREATE TABLE webhook_outbox (
  effect_key   TEXT PRIMARY KEY,
  effect_type  TEXT NOT NULL,
  payload      TEXT NOT NULL,
  status       TEXT NOT NULL DEFAULT 'pending'
);

The composite inbox key prevents unrelated providers or endpoints from colliding. The payload hash detects a more serious condition: the same documented event key arriving with different content. Do not silently accept that as an ordinary duplicate. Quarantine it as a contract violation or collision and preserve only the authorized evidence needed to investigate.

The receiver’s transaction can be expressed as:

await db.transaction(async (tx) => {
  const existing = await tx.findInbox(verified);

  if (existing) {
    if (existing.payloadSha256 !== verified.payloadHash) {
      await tx.quarantineCollision(existing, verified.payloadHash);
      throw new Error("event key reused with different payload");
    }
    return;
  }

  await tx.insertInbox({
    ...verified,
    status: "pending"
  });

  await tx.applyInvoiceTransition({
    invoiceId: verified.payload.invoice.id,
    nextState: "paid",
    version: verified.payload.invoice.version
  });

  await tx.insertOutboxIfAbsent({
    effectKey: verified.payload.effect.key,
    effectType: "grant_receipt",
    payload: JSON.stringify({
      invoiceId: verified.payload.invoice.id
    })
  });
});

The example omits database-specific syntax, transaction isolation, authorization, payload-size limits, schema validation, and HTTP error mapping. Those must be selected deliberately. The important property is that the inbox claim, local state transition, and outbox intent commit together.

A separate “seen IDs” write is insufficient. If the consumer records an ID and then stops before changing the invoice, a retry may be discarded and the effect lost. If it changes the invoice and stops before recording the ID, a retry may repeat the effect. A transaction connects the receipt with the local consequence that receipt authorizes.

Run the Duplicate Storm

Do not test duplicates sequentially through an in-memory set. Send concurrent requests against the same database constraint. A sequential test can pass even when two workers can both read “not seen” and then perform the same work.

test("fifty valid duplicates create one effect", async () => {
  const delivery = makeDelivery({
    rawBody: JSON.stringify(FIXTURE_EVENT)
  });

  const responses = await Promise.all(
    Array.from({ length: 50 }, () => postFixture(delivery))
  );

  expect(responses.every((response) => response.status === 202))
    .toBe(true);
  expect(await countInbox("evt_fixture_0042")).toBe(1);
  expect(await invoiceState("inv_fixture_17")).toBe("paid");
  expect(await countOutbox("receipt:inv_fixture_17")).toBe(1);
});

The exact success status is a local bench choice, not a universal provider rule. The durable assertions matter:

• one inbox identity;
• one permitted state transition;
• one effect intent;
• no unique-constraint error exposed as a delivery failure; and
• no dependence on which request won the race.

Then send two distinct event IDs for the same invoice and different legitimate event types. Both belong in the inbox. Deduplicating solely by resource ID would incorrectly discard real history. Conversely, send the same event ID with one changed payload byte and assert that it enters a visible quarantine path rather than being labeled a harmless duplicate.

Run the storm with the real database engine and the isolation level intended for the service. A mocked map cannot reveal unique-index races, deadlocks, rollback behavior, or restart persistence. Repeat it with several worker processes if production topology uses more than one process, but keep this article’s fixture isolated from any production system.

Inject the Three Crash Windows

A generated handler can look idempotent while failing at a process boundary. Add deterministic failpoints and stop the worker at each one.

Failpoint	Required observation	Why it matters
After verification, before inbox commit	No inbox row or effect exists; the endpoint does not report durable success.	A retry or reconciliation path must still be able to recover the event.
After inbox commit, before worker execution	One pending row survives restart; the worker later completes it.	Durable receipt makes a quick acknowledgement safe.
After an external effect, before marking it complete	A retry does not blindly repeat the effect; ambiguity is surfaced or reconciled.	A local transaction cannot atomically include an arbitrary remote API.

The first window exposes a provider-specific recovery issue. A non-2xx response can trigger Stripe’s automatic retry behavior. GitHub explicitly does not automatically redeliver failed deliveries, so a GitHub integration also needs a delivery audit and manual or API-driven redelivery process. “The sender will retry” is not a portable assumption.

The second window is the reason for a durable inbox. On restart, a worker selects pending records and resumes them. Test this with a real process stop, transaction failpoint, or forced connection loss, not merely an exception caught inside the same function. The record must remain recoverable when all process memory is gone.

The third window is the hard boundary. Suppose the worker calls an email, payment, or fulfillment API, the remote service succeeds, and the worker stops before recording success locally. On retry, the local database cannot know whether the remote effect occurred.

Prefer a stable downstream idempotency key when the remote contract supports one. Otherwise store an explicit unknown state and reconcile against the remote system before trying again. Never convert ambiguity into a confident “failed” status merely because the AI-generated call threw a timeout. A timeout describes what the caller observed, not what the remote service committed.

This is why “exactly-once webhook delivery” is the wrong headline. The testable claim is narrower: duplicates do not violate the local invariant, recoverable work survives a crash, and uncertain external effects have an idempotency or reconciliation path.

Make Out-of-Order Events Converge

Next deliver invoice version 3 before version 2. The final state must remain paid; the stale event must not regress it.

Use a provider-defined resource version when one exists. If the provider supplies an authoritative occurrence time for the relevant comparison, use it under that contract. Do not manufacture order from arrival time, database insertion time, or an unrelated signature timestamp. Two events can also share a coarse timestamp, so a timestamp comparison needs an explicit tie policy.

Stripe documents that event order is not guaranteed and that consumers can retrieve missing objects through its API. GitHub advises using timestamps contained in its payload when relative event time matters. Neither statement creates a universal ordering algorithm.

Useful strategies include:

• accept only legal state-machine transitions;
• compare a documented monotonic resource version;
• retrieve the current authoritative resource;
• defer an event until a prerequisite exists;
• make operations commutative where possible; and
• reconcile current provider state periodically.

Test two equal-looking timestamps, a missing version, and an older event that arrives after a newer one. The consumer should record why it applied, ignored, deferred, or reconciled each event. “Last request wins” is not a business rule.

Complete the Fixture Ledger

The finished bench should cover more than one duplicate test.

Fixture	Expected result
Correct signature over exact raw bytes	Durable receipt succeeds.
Parsed and reserialized body	Signature fails; no row or effect is created.
Missing, malformed, or wrong signature	Provider-appropriate rejection; no business work.
Old signed attempt outside tolerance	Rejected as stale under that signature contract.
Legitimate retry with a fresh attempt signature	Signature passes; stable event identity deduplicates it.
Fifty concurrent copies	One inbox row, transition, and effect intent.
Same event key with a different payload hash	Quarantined rather than silently deduplicated.
Same object with distinct valid event IDs or types	Both retained and evaluated.
Newer object state followed by stale state	Final state does not regress.
Unknown but authentic event type	Audited no-op or documented failure according to provider policy.
Current and previous signing keys during rotation	Accepted only for the approved overlap; an unknown key is rejected.
Test-mode event sent to a live endpoint adapter	Rejected or isolated under endpoint-specific configuration.
Slow business worker	Durable receipt remains inside the sender’s acknowledgement limit.
Redelivery after cache expiry	The persistent business invariant still blocks a duplicate benefit.

Measure receipt latency separately from worker latency. Record event key, provider, endpoint, type, disposition, attempt count, and payload hash only where authorized. Do not log signing secrets, signature headers, or unrestricted sensitive bodies. If raw-payload retention is required for recovery, define encryption, access, minimization, and deletion rules rather than letting a debug table become a permanent archive.

Add a reconciliation test that compares the provider’s current authoritative state with local state after simulated downtime. It should find a deliberately omitted event and repair the local projection without granting the same effect again. This test matters most for senders that do not automatically retry.

Finally, test acknowledgement policy by outcome. An invalid signature should not enter the inbox. A transient database failure must not be reported as durable success. An authentic but unsupported event should follow the provider-specific documented policy—often a recorded 2xx no-op, but never a universal assumption copied between integrations.

Give AI a Bounded Role

AI is useful at this bench. It can draft fixture factories, generate concurrency-test scaffolding, enumerate crash points, propose state-machine properties, and compare an adapter with a cited provider contract. It can also produce readable failure messages and turn a contract table into parameterized test names.

It should not:

• invent a signature profile;
• treat every timestamp as replay protection;
• choose a deduplication key without provider evidence;
• assume failed deliveries are automatically retried;
• infer event order from arrival;
• equate an event ID with a business invariant;
• promise exactly-once remote effects; or
• decide that an ambiguous timeout means nothing happened.

Ask the model to attach each generated rule to the contract sheet and a failing test. If it cannot, keep the rule out of the consumer. A fluent explanation is not a delivery guarantee, and a generated unit test that repeats the implementation’s assumption is not independent evidence.

A webhook route is finished only when the team can stop it at every boundary, send the same delivery again, reverse the delivery order, and explain the resulting state.

The second knock should be ordinary—not a surprise that ships twice.

Source and Version Note

Sources were checked on August 25, 2026. Provider behavior can change, and endpoint configurations can differ. Confirm the current documentation and exact event-destination mode before implementing or revising a consumer.

• Stripe: Receive events in your webhook endpoint documents raw-body verification, quick responses, retry and manual-resend horizons, unordered delivery, duplicate handling, asynchronous processing, signature timestamps, and secret rotation.
• GitHub: Validating webhook deliveries documents X-Hub-Signature-256, the original payload body, HMAC-SHA256, and constant-time comparison examples.
• GitHub: Handling failed webhook deliveries states that failed deliveries are not automatically redelivered and describes manual or API-based recovery.
• GitHub: Troubleshooting webhooks documents the ten-second acknowledgement limit and out-of-order delivery behavior.
• GitHub: Redelivering webhooks documents the current three-day redelivery window.
• GitHub: Automatic redelivery example states that the delivery GUID remains constant across redeliveries of the same delivery.
• Standard Webhooks specification, version 1.0.0 describes a community webhook profile with signed ID, attempt timestamp, raw payload, rotation support, constant-time comparison, and idempotency guidance. Its existence does not establish provider adoption.
• CloudEvents specification, version 1.0.2 defines source plus id semantics for conforming CloudEvents producers.

The synthetic code is illustrative and conceptually reviewed against the stated fixture properties. It is not a complete HTTP server, provider SDK replacement, cryptographic audit, database implementation, or production-readiness guarantee.