Handling Webhook Race Conditions and Out-of-Order Delivery at Scale

The Illusion of Ordered Webhook Streams

Most engineering teams build their first webhook endpoint under an intuitive but fatal assumption: that events arrive sequentially and exactly once. A customer initiates a checkout, the billing provider fires checkout.completed, and subsequent subscription lifecycle events follow in orderly succession.

In production, this assumption can create incorrect state. Real-world networks experience packet retransmission, regional gateway jitter, and concurrent retry storms. When a provider fires an event, network lag can cause a subsequent subscription.canceled event to arrive and execute before the original subscription.created payload reaches your server. If a database blindly upserts the latest payload, the record can reflect an incorrect subscription state.

The Anatomy of Naive Failures

Consider what happens when a billing provider re-transmits an event due to a temporary TCP reset. If your handler directly triggers inventory reservation or sends an invoice:

• Double Fulfillment: A single customer purchase results in duplicate inventory depletion or multiple automated fulfillment tickets.
• Lock Contention & Race Conditions: Concurrent worker processes attempt to update the same customer row simultaneously, resulting in deadlock timeouts (PostgreSQL error 40P01) or lost updates.
• Timeout Cascades: Heavy synchronous operations inside the HTTP webhook handler cause response latency to exceed provider thresholds (often 5 to 10 seconds), prompting the provider to flag the endpoint as failing and retry with exponential volume.

Architecture Pattern: The Three-Stage Ingestion Pipeline

For consequential workloads, consider separating webhook ingestion from business processing with a staged architecture:

1. Fast Ingestion and Signature Verification (< 150ms)

The HTTP handler performs exactly three operations before returning HTTP 200 OK:

1. Validates the cryptographic HMAC signature using the shared webhook secret.
2. Generates a deterministic hash of the payload or extracts the provider's unique event_id.
3. Inserts the raw payload into an append-only webhook_events queue table with status pending.

CREATE TABLE webhook_events (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  provider VARCHAR(64) NOT NULL,
  event_id VARCHAR(128) NOT NULL,
  event_type VARCHAR(128) NOT NULL,
  payload JSONB NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status VARCHAR(32) NOT NULL DEFAULT 'pending',
  processed_at TIMESTAMPTZ,
  error_message TEXT,
  CONSTRAINT uq_provider_event UNIQUE (provider, event_id)
);

2. Idempotent Asynchronous Processing

A background queue worker (such as Cloudflare Queues, SQS, or a Redis-backed worker) pulls pending events. Before applying any mutation, the worker checks:

• Has this event already been processed? Checked via the unique index on (provider, event_id).
• Is this event older than current record state? For lifecycle updates, compare the event's created_at timestamp against the entity's version or last_event_timestamp in your database. If the event is older, mark it as ignored_stale and exit cleanly.

3. Transactional Outbox for Side Effects

If processing the webhook requires triggering external side effects (e.g., sending an email via Resend or notifying an ERP), never call the third-party API directly inside the database transaction. If the transaction rolls back, the email was still sent. Instead, write the outgoing action into an outbox_messages table within the same ACID transaction, and let an outbox runner dispatch it safely.

Production Checklist for Webhook Reliability

• Consider retaining raw payloads: Subject to privacy and retention requirements, raw payloads can support investigation and replay when third-party schemas change or a regression occurs.
• Implement dead-letter queues (DLQ): Payloads that fail after 5 exponential retries must be routed to a dead-letter queue with alerting for engineering review.
• Enforce rate-limiting and IP allowlisting: Protect your webhook ingress from DDoS attacks by verifying IP ranges where supported and caching signature verification keys.