Webhook retries and idempotency go together. A sender that gets no 2xx answer in time sends the same event again, so your receiver will sometimes see one event twice. Make the handler idempotent: acknowledge fast, queue the work, and record each delivery id so that a repeat is recognised and skipped.
On the receiving end of an integration, I treat every webhook as something that may arrive more than once. A handler that does all the work inside the request will, sooner or later, do it twice.
Why do webhook senders retry the same event?
A sender retries because it cannot tell a lost request from a lost answer. If your server did the work and then the connection dropped, or your answer came after the sender’s timeout, the sender records a failure and sends the event again.
RFC 9110, the HTTP semantics standard, lists PUT, DELETE and the safe methods as idempotent; POST is not among them. A client should not retry a non-idempotent request automatically unless it can know the request is idempotent, or detect that the original was never applied. A sender that retries a POST relies on the receiver to make it idempotent, and a delivery id that stays the same on every attempt lets the receiver do that.
Why use a queue for webhooks?
A queue lets the handler answer within milliseconds and do the slow work afterwards, so the sender’s timeout never races your business logic.
- Verify the signature against the raw body, if the sender signs its calls.
- Insert the delivery id and the raw body into a table with a unique key on the delivery id.
- Answer 2xx. RFC 9110 defines 202 Accepted for this case: accepted for processing, processing not completed.
- A worker reads the stored event, does the work and marks the row done.
If the insert hits the unique key, you already have the event, so answer 2xx and do nothing else. An error answer to a duplicate buys more retries. It is the same idea as the unique key on device and timestamp in an MQTT database writer. If the insert fails because the database is down, answer with an error, so the sender tries again later.
What happens if a webhook endpoint is down?
The sender retries on its own schedule and then stops. Attempts and waits differ from sender to sender, so read its documentation before you plan a maintenance window.
Retried events arrive late and can arrive out of order, so order them by the timestamps in the payload, not by arrival. For a long outage, keep a catch-up read through the sender’s API, as in webhook vs WebSocket vs polling.
Hermetry connects field devices to live dashboards, rules and alerts: IoT telemetry and SCADA in one platform. Its webhooks carry an X-Webhook-Delivery id that is the same on every retry and redelivery, and deliveries are at least once, with no guaranteed order. By default a delivery gets 3 attempts, the first included, with waits of 60,000 ms and 300,000 ms before the retries. After 10 failures in a row the endpoint is switched off; once an administrator switches it on again, what failed can be redelivered or replayed.
Before you write the worker, look up in the sender’s event documentation, such as a webhook event reference, which field holds the event time.
How do you test a webhook on localhost?
A sender on the internet cannot reach localhost, so replay recorded payloads against your local handler first, then do one end-to-end check through a public HTTPS address.
- Record one real delivery, headers and raw body, and replay it twice with a command-line HTTP client. The second call should get a 2xx and cause no side effect.
- If your handler rejects old timestamps, sign the replays with a test secret, and send one stale timestamp to see it refused.
A tunnel that gives your machine a temporary public HTTPS address covers the end-to-end check. I close it when the test passes.
Frequently asked questions
Is a webhook POST idempotent by default?
No. RFC 9110 lists PUT, DELETE and the safe methods as idempotent, and POST is not among them. A webhook POST becomes idempotent only when the receiver makes it so, for example by storing the delivery id under a unique key and skipping repeats.
How long should I keep delivery ids?
At least as long as the sender can still retry or redeliver the event, and a manual redelivery, where a sender offers one, can come days later. Each id is a short string, so I keep them for weeks.
Can I deduplicate on the payload instead of a delivery id?
Only as a fallback. Two genuine events can carry identical payloads, for example a value crossing the same limit twice. Without a delivery id, build a key from the event type, the source id and the event’s own timestamp.

