API tests

How to test webhooks: signatures, duplicates, ordering, and delayed delivery

A webhook is a request through which an external service tells your application about an event: a carrier changes the status of a parcel, an invoicing system issues a document, or a CRM updates a contact. One successful request in Postman does not show that the integration can handle real delivery behaviour. This process helps engineering and QA teams test signatures, repetition, ordering, and delays without presenting a functional check as a security audit.

Why receiving a 200 response is not enough

A webhook connects two separate concerns. First, the endpoint must receive and verify the HTTP request. The application must then make the correct business-state change, such as marking a parcel as delivered, creating an invoice, or adding a contact to a campaign.

A test that checks only the status code can miss an event stored twice, an older state overwriting a newer one, or a side effect taking place before the message has been saved safely. Conversely, the correct database change does not show that the endpoint will reject a request with an invalid signature. The test model therefore needs to observe receipt, processing, and the outcome.

Example: a parcel returns to an older state

Consider an online shop that marks an order as complete and emails the customer after a “parcel delivered” event. The endpoint receives and stores the event, but a network timeout prevents its response from reaching the carrier. The carrier sends the same event again. If two workers process both copies concurrently without shared protection, they may send two emails or award loyalty points twice.

Several minutes later, an older “parcel in transit” event arrives after being delayed in a queue. Code that simply replaces the status with the value from the last message received moves the completed order backwards. Every individual request may still return 200 and appear successful in a technical log.

The expected result of this test is not a particular implementation but a set of observable rules: the final status remains “delivered”, one customer email is created, both deliveries can be traced, and an unknown or older input does not cause an invalid transition. These expectations remain useful even if the team later replaces its database or message queue.

Record the provider’s contract first

There is no universal webhook protocol. Providers use different headers, signature algorithms, event identifiers, retry rules, and expected responses. Before designing scenarios, record the following from the official documentation:

• which event types you subscribe to and which fields you use;
• which bytes and algorithm are used to verify the signature;
• which HTTP response acknowledges receipt and when it must be returned;
• whether the provider retries failed deliveries, can send duplicates, or does not guarantee ordering;
• which identifier represents the event, delivery, and business object.

For example, Stripe documents the original request body, automatic retries, duplicate events, and non-guaranteed ordering. This is one provider’s contract, not a rule that can be transferred to a carrier or CRM without checking.

Translate the contract into a small decision table. For each event type, record the required fields, object to be found, permitted starting state, new state, and side effects. Define separate behaviour for an unknown type, an unknown object, and a correctly signed but incomplete message. The endpoint may technically accept a message while marking it unprocessed and sending it to a controlled error queue. The team then avoids treating “HTTP request received” as equivalent to “business change completed”.

Scenarios that reveal the most defects

1. Verify the signature over the original body

Create a valid event with the provider’s test data. Then separately change the body, signature, header, and secret. A valid request should proceed to processing; an invalid one must not affect business data.

The signature is often calculated over the exact request body received. If a framework parses and reserialises JSON first, it can change spacing, order, or encoding and break verification. Stripe therefore requires the raw body; GitHub’s procedure also calculates a hash-based message authentication code (HMAC) from the secret and original body and compares it with the signature header. Test the algorithm, header, encoding, and any timestamp tolerance exactly as specified by the chosen provider.

2. Send the same event more than once

Deliver the same message twice in sequence and then concurrently. Do not check only the number of rows in an event table. Inspect every side effect: an invoice, email, inventory movement, or status change should occur according to the business rule, not again on every receipt.

Idempotent processing means that repeating the same event does not create a second business outcome. Recording an already processed identifier with a database constraint that also works under concurrency is a common approach. Two different events concerning the same object are not necessarily technical duplicates, however, so the permitted business-state transition also needs a check.

3. Change the order only when the contract allows it

If the provider does not guarantee ordering, send the newer state before the older one. An application should not return a delivered parcel to “dispatched” simply because it receives an older message later. The solution may use an event version, a timestamp from a trusted source, state-machine rules, or a request for the current object through the provider’s API. Its contract and your business rules determine the right choice.

4. Simulate a delay, timeout, and retry

Delay an event by minutes or hours and verify that the application can still reach a consistent state. Then simulate an endpoint timeout or error and use the redelivery mechanism provided by the sandbox, administration interface, or provider command-line interface (CLI). Do not infer its retry schedule yourself.

Test the boundary between receipt and processing separately. If the endpoint stores a message in a queue before returning the contractually required response, simulate a failure before storage, after storage, and during processing. An acknowledged event should not disappear, while a repeated event should not execute the effect twice.

5. Check both the outcome and diagnostics

For each input, define the expected HTTP response, event record, final business state, and permitted side effects. Include the delivery or event identifier in logs so the team can connect a request with its processing. Do not automatically store sensitive fields or entire bodies; diagnostics still need to follow the rules for personal data and credentials.

Close the test with evidence, not just a delay

Asynchronous processing may not finish when the HTTP response is returned. A test should therefore not use an arbitrary long sleep after sending a webhook. It should poll with a fixed deadline for an observable result: a record marked “processed”, the expected change through an API, or a message in a test output for the queue. If the deadline expires, it preserves the intermediate states and fails rather than waiting indefinitely.

Verify four levels for every scenario:

1. Receipt: the status code and signature-validity decision.
2. Record: the event identifier, attempt count, and processing status.
3. Business result: the correct object, final state, and values intended to change.
4. Negative evidence: no second invoice, second email, or backwards transition exists.

For a concurrency test, use a mechanism that releases both requests at the same instant instead of sending them quickly in one loop. Wait for both attempts to complete and inspect both the database constraint and business outcome. Finally, run the reconciliation process if the system has one: it may retrieve the current state from the provider after a lost or incomplete event. A separate test then verifies both normal delivery and the route back to consistent data.

Combine three test layers

A local test with a controlled message quickly covers invalid signatures, concurrency, and error paths. The provider’s sandbox verifies the real format, headers, and supported redelivery. A small integration scenario then confirms the complete path to the business outcome. Mock, sandbox or real service explains the differences between these layers.

A signature test demonstrates the implementation’s behaviour for the covered contract. It is not an assessment of the endpoint’s overall security, secret management, network, or permissions. Similarly, a successful sandbox scenario does not establish a production provider’s speed or availability.

What you gain

This suite reveals defects that a happy path misses: a duplicate invoice, a backwards state transition, a lost event, or an invalid message accepted for processing. The team also gains repeatable evidence of the behaviour the integration handles and the provider contract for which that evidence applies.

Next step

Choose one webhook with a real business effect and create a matrix of five inputs: valid, incorrectly signed, duplicate, delayed, and delivered out of order if the provider does not guarantee ordering. Add the expected response, state, and side effects to each. Only then add the scenarios to your automated API tests.