Webhook Retries and Idempotency

Why retries exist

Webhooks cross the public internet. Timeouts, deploys, load balancers, and database outages happen. A provider cannot know whether your application processed an event unless your endpoint returns a successful response.

That means duplicate delivery is a feature, not a bug.

Store before side effects

The reliable pattern is:

1. Verify the signature.
2. Parse the event.
3. Insert the event ID into a table with a unique constraint.
4. If the insert conflicts, return 200.
5. Apply side effects.
6. Mark processing complete.
7. Return 200.

For high-throughput systems, enqueue after the unique insert and process asynchronously.

Example schema

create table webhook_events (
  provider text not null,
  event_id text not null,
  event_type text not null,
  received_at timestamptz not null default now(),
  processed_at timestamptz,
  payload jsonb not null,
  primary key (provider, event_id)
);

This lets retries hit the same primary key instead of running the side effect twice.

Monotonic email status

Email events can arrive out of order. For example, you may see delivered after a temporary deferred, or an old accepted event after a final bounced event.

Define status precedence:

Only move the stored message state forward unless the new event is a correction you explicitly support.

Response codes

Return:

1. 2xx after durable acceptance.
2. 4xx only when the request is invalid and should not be retried.
3. 5xx when retrying later may succeed.

Do not return 500 after a duplicate event has already been processed. That creates unnecessary retry noise.

Operational checklist

1. Verify signatures from the raw body.
2. Record event IDs with a unique key.
3. Use database transactions for status updates.
4. Make downstream notifications idempotent too.
5. Monitor webhook failure rates.
6. Keep old and new provider handlers during migration.
7. Reconcile daily counts with provider dashboards.

Pair this with webhook signature verification before processing production traffic.

References

• RFC 2104: HMAC
• Postscale webhooks documentation
• Postscale inbound email webhooks guide