Resolving Out-of-Order Webhook Delivery Issues

You hit this problem the first time an invoice.payment_succeeded lands before the invoice.created that should have preceded it, and your state machine either rejects a payment for an invoice it has never heard of or silently creates a phantom one. Asynchronous payment providers make no ordering guarantee across HTTP deliveries, so any handler that treats receipt order as truth will eventually corrupt entitlements or double-charge. This page is the sequencing companion to Webhook Retry & Timeout Strategies: retries solve duplication, but they do nothing for ordering. The fix is to decouple ingestion from mutation and apply events in provider-defined order, not arrival order.

The reordering is not a rare edge case you can defer. Providers fan out webhook delivery across a pool of worker nodes, and two events emitted 40 ms apart can travel different network paths, hit different TLS handshakes, and land at your endpoint in the reverse order. Under retry, the gap widens: a customer.subscription.updated that failed its first delivery and succeeds on the third attempt can arrive minutes after the customer.subscription.deleted that logically followed it. If your handler applies the deleted event, marks the subscription_id as cancelled, and then applies the late update that flips status back to active, you have just re-granted entitlements to a churned customer. The blast radius is real money and real access, which is why arrival order is disqualified before the analysis even begins.

The subtle failure is not the loud one. A handler that crashes on an unknown invoice_id at least surfaces the problem in your error tracker. The dangerous case is the handler that quietly upserts a partial row from the payment event, fills in defaults for the fields the invoice event would have carried, and then never corrects them when the invoice finally arrives because the row “already exists.” That row now reports an amount_cents of 0 against a customer who was in fact charged 4900, and nobody notices until a finance reconciliation flags the discrepancy weeks later. Ordering correctness is therefore a data-integrity property, not a latency optimization.

Trade-offs

The ordering strategies form a spectrum from trusting arrival order (never, for money) to a full event-sourced log (total order, high cost). The pragmatic middle — a provider-sequence buffer plus a version-guarded upsert — gives strong ordering at moderate cost. The map ranks them by ordering strength.

For most SaaS billing, the pragmatic answer is a per-subscription buffer keyed on a monotonic provider sequence, backed by a version-guarded upsert so anything that slips the buffer still cannot regress state. Wall-clock created_at alone is a trap: providers run multiple emitters and clocks drift, so timestamps regress.

The reason the version-guarded upsert earns its own row in the table is that it degrades gracefully. A buffer is a coordination mechanism with moving parts — a Redis instance, a drain worker, a TTL — and any of those can be down during an incident. The upsert guard is a static invariant enforced by the database itself, so even if the buffer is bypassed entirely (a hotfix that writes directly, a replay script run by hand, a second consumer you forgot about), the WHERE sequence_number < EXCLUDED.sequence_number predicate still refuses to let a stale event overwrite fresh state. That is why the recommended architecture is not “buffer or guard” but both: the buffer optimizes for applying intermediate events in order, and the guard is the last line that protects the end state when the buffer fails or is circumvented.

Full event sourcing sits at the far end of the spectrum for a reason worth naming: it buys you total order and perfect auditability, but it moves the ordering problem from write time to read time. Every query against subscription state becomes a fold over the event log, and you now own snapshotting, log compaction, and schema evolution of historical events. For an audit-critical ledger where you must be able to answer “what did we believe about this customer_id at 14:03 on the day of the dispute,” that cost is justified. For a typical entitlements service that only needs the current truth, it is a large standing tax to solve a problem the buffer already handles. Choose it deliberately, not by default.

Where the buffer latency actually comes from

The 2–5 second figure in the table is not a fixed sleep; it is the worst-case wait for a missing predecessor to arrive under normal jitter. In steady state, when events land roughly in order, the drain worker releases each event within milliseconds of receipt because the head of the sorted set already equals last + 1. The buffer only adds visible latency when it is doing its job — holding event 2 because event 1 is genuinely still in flight. Tuning the TTL is therefore a bet on your provider’s tail: set it to roughly the 99th-percentile inter-event delivery gap you observe, commonly 3–10 seconds, so that ordinary reordering resolves inside the buffer and only true losses escalate to reconciliation.

Step-by-Step Implementation

The mechanism decouples ingestion from mutation: acknowledge fast, buffer per subscription scored by sequence, then release only contiguous sequences into a version-guarded upsert. The timeline shows why a payment that arrives before its invoice waits in the buffer until its predecessor lands.

1. Diagnose sequence drift before you build

Extract the provider’s sequence id or created_at from the raw payload — never your server’s receipt time. Log every event where created_at < last_processed_created_at and emit it as a sequence_gap metric. This separates true ordering violations from ordinary network jitter and from retries (which reuse the same created_at).

Do this measurement before you write a line of buffering code, because the shape of the drift tells you which mechanism you actually need. If the diagnostic shows that fewer than one event in ten thousand arrives out of order and the reorder distance is never more than one position, a version-guarded upsert alone may be sufficient and the buffer is premature complexity. If it shows bursts of deep reordering correlated with provider incidents — ten events shuffled across a thirty-second window — you need the per-subscription buffer and a generous TTL. Instrument first so the architecture matches the observed distribution rather than a guessed one. It is also worth breaking the sequence_gap metric down by event_type, because reordering is rarely uniform: subscription lifecycle events tend to reorder against each other far more than isolated invoice.payment_succeeded events, and knowing which pairs collide tells you exactly which invariants your tests must cover.

def classify(event: dict, last_processed_created_at: float) -> str:
    if event["created_at"] == last_processed_created_at:
        return "retry"            # same event, dedupe handles it
    if event["created_at"] < last_processed_created_at:
        return "out_of_order"     # ✗ genuine sequencing violation
    return "in_order"             # ✅ normal forward progress

2. Acknowledge fast, then buffer per subscription

Return 200 immediately so the provider does not retry, then park the payload in a Redis sorted set scored by sequence. This gives O(log N) insert and ordered reads, and isolates one subscription’s drift from every other.

Keying the buffer per subscription_id rather than using one global queue is the decision that makes this scale. Ordering only matters between events that touch the same aggregate: a payment on subscription A has no causal relationship to an invoice on subscription B, so forcing them through a single ordered channel would make one slow customer’s missing predecessor stall every other customer’s events. Per-subscription keys turn one large ordering problem into millions of tiny independent ones, and a stall in one key never propagates. The trade is a proliferation of small sorted sets, which is cheap in Redis but does mean you must sweep empty keys — set a TTL on the buffer key itself so a subscription that goes quiet does not leave a tombstone behind forever.

The MAX_BACKLOG cap and the Lua script matter more than they first appear. The script makes the “add, then check size” a single atomic operation, so two workers inserting concurrently cannot both observe a below-cap size and push the set over the limit. Without atomicity you would either need a separate lock or accept an unbounded buffer, and an unbounded buffer under a provider outage is how a reordering safeguard turns into an out-of-memory incident. Fifty is a deliberately low cap: a healthy subscription almost never has more than two or three events in flight at once, so a backlog of fifty is already a strong signal that a predecessor was lost rather than merely delayed, which is exactly why overflow routes to the dead-letter queue for reconciliation instead of silently growing.

import redis, json

r = redis.Redis()
MAX_BACKLOG = 50

def buffer(subscription_id: str, payload: dict, sequence: int) -> None:
    key = f"webhook_buffer:{subscription_id}"
    lua = """
    redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])
    if redis.call('ZCARD', KEYS[1]) > tonumber(ARGV[3]) then
      return -1
    end
    return 1
    """
    if r.eval(lua, 1, key, sequence, json.dumps(payload), MAX_BACKLOG) == -1:
        route_to_dlq(subscription_id, payload, "backlog_overflow")  # ⚠️ reconcile

3. Release only contiguous sequences

A worker pops the lowest-scored event and releases it only when it is exactly one past the last processed sequence. Anything ahead waits for its predecessor; a TTL converts an indefinite stall into a reconciliation trigger.

Note the three branches in drain are not symmetric, and the middle one is easy to omit and expensive to omit. The sequence <= last branch drops events the ledger has already applied — these are retries or duplicates that slipped past upstream dedup, and leaving them in the set would wedge the drain loop forever because it keeps re-reading a head it can never advance. The strict sequence == last + 1 equality is what enforces contiguity: it is tempting to relax it to sequence > last so the worker “catches up faster,” but that reintroduces exactly the gap you are trying to prevent, applying event 3 while event 2 is still missing. The only correct way to advance past a gap is through reconciliation, never by loosening the release condition.

One concurrency subtlety hides in get_last_sequence and apply_event: if two drain workers process the same subscription_id key at once, both can read the same last, both can see the head as last + 1, and both can call apply_event. The version-guarded upsert in the next step makes the duplicate application harmless — the second write is a no-op because the sequence is no longer strictly greater — but you still waste work and risk log noise. Serialize draining per key with a short-lived Redis lock (SET buffer_lock:{subscription_id} nx ex 5) so only one worker owns a subscription’s drain at a time, and let the lock’s own expiry handle a worker that dies mid-drain.

def drain(subscription_id: str) -> None:
    key = f"webhook_buffer:{subscription_id}"
    while True:
        head = r.zrange(key, 0, 0, withscores=True)
        if not head:
            return
        payload_raw, sequence = head[0][0], int(head[0][1])
        last = get_last_sequence(subscription_id)
        if sequence == last + 1:
            apply_event(json.loads(payload_raw))   # ✅ in order
            r.zrem(key, payload_raw)
        elif sequence <= last:
            r.zrem(key, payload_raw)               # already applied, drop
        else:
            return                                 # ⚠️ gap, wait or reconcile

4. Upsert with a version guard

Even with a buffer, defend the storage layer. A composite unique constraint blocks duplicates and the WHERE sequence_number < EXCLUDED.sequence_number clause rejects any stale write that bypassed the buffer.

The ON CONFLICT (provider_event_id) target is doing double duty and both jobs matter. The first is idempotency: the same event replayed twice collides on provider_event_id and updates in place rather than inserting a second row, so a retry can never inflate amount_cents by appending a duplicate. The second is the ordering defense in the WHERE clause, which fires only when the incoming sequence_number strictly exceeds the stored one. Together they give you a write that is safe to run with any event, in any order, any number of times — the same physical row converges to the highest sequence it has ever seen and stays there. That convergence property is what lets the rest of the system be sloppy about delivery without the ledger ever being wrong.

Be deliberate about which columns the guard protects. If your ledger row carries fields that legitimately change out of band from the sequence — a dispute_status updated by a separate webhook stream, say — folding them into the same guarded upsert means a stale billing event could stomp a fresher dispute update, or vice versa. When two independent event streams write the same row, either give each its own sequence_number column and guard them independently, or split them into separate rows joined by subscription_id. Cramming unrelated update cadences behind one monotonic guard is a common way to reintroduce the exact regression the guard was meant to stop.

INSERT INTO subscription_ledger (
  ledger_entry_id, subscription_id, provider_event_id,
  event_type, sequence_number, amount_cents, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, now())
ON CONFLICT (provider_event_id) DO UPDATE
   SET sequence_number = EXCLUDED.sequence_number,
       amount_cents    = EXCLUDED.amount_cents,
       updated_at      = now()
 WHERE subscription_ledger.sequence_number < EXCLUDED.sequence_number;

Verification & Testing

The core invariant is that a shuffled event stream produces the same ledger as its in-order replay. The tests prove it three ways: an explicit reorder, a replay for idempotency, and a stale upsert that must be rejected. The panel lists them.

Assert the core invariant directly: feed invoice.payment_succeeded (sequence 2) before invoice.created (sequence 1) and confirm the payment is held until creation applies, then both land in order. Replay any event twice and assert one ledger row. Push a stale sequence_number through the upsert and assert the row is unchanged. Run a daily reconciliation query that diffs internal state against the provider’s authoritative API.

-- Reconciliation: events we hold that the provider considers superseded
SELECT l.subscription_id, l.provider_event_id, l.sequence_number
  FROM subscription_ledger l
 WHERE l.status <> 'reconciled'
   AND l.updated_at < now() - interval '1 hour'
 ORDER BY l.subscription_id, l.sequence_number;

Integration tests should shuffle a known event stream, inject random latency and duplicates, and assert the final ledger state is identical to the in-order replay.

Make the shuffle test property-based

A single hand-written reorder proves the mechanism works for one permutation; the real guarantee you want is that it works for every permutation. Drive the headline invariant with a property-based test that generates random permutations of a fixed event set — say the six lifecycle events for one subscription_id — and, for each permutation, feeds them through the full ingest-buffer-drain-upsert path and asserts the final ledger row matches the in-order replay byte for byte. A good generator also duplicates a random subset of events and interleaves a second subscription_id to prove per-key isolation. When such a test fails it hands you the exact seed and permutation that broke, which is dramatically faster to debug than a flake reported once a week in staging. Convergence under arbitrary permutation is the property; enumerate a few hundred permutations per run and you will surface the off-by-one in the release condition long before production does.

The reconciliation query above is the safety net for everything the tests cannot cover, namely genuinely lost events. It surfaces rows that have sat unreconciled past a threshold, which is the signature of a predecessor that never arrived and a buffer that already gave up and dead-lettered. In production, wire the count of such rows to an alert: a slow, steady trickle is normal background loss you resolve by pulling authoritative state from the provider API, but a sudden spike means the provider is dropping deliveries in bulk and you should stop trusting the webhook stream and fall back to a full sync for the affected accounts. The test suite proves the code is correct; the reconciliation query proves the world outside the code is still delivering what the code assumes.

Gotchas & Production Pitfalls

The pitfalls here are all about the buffer’s edges: what it trusts for ordering, what happens when a predecessor never arrives, and how rejections stay visible. The map groups them so each fix is one rule.

• Clock skew makes created_at regress. Two emitters with drifting clocks produce timestamps that go backwards. Prefer a monotonic provider sequence id; fall back to created_at only when no sequence exists.
• Buffers stall forever on a lost predecessor. If event 2 never arrives, event 3 waits indefinitely. A TTL must escalate to provider-API reconciliation, not block the pipeline.
• Backlog overflow hides a deeper outage. A buffer growing past its cap usually means the provider dropped an event, not that traffic spiked. Dead-letter and reconcile rather than raising the cap.
• Version guards silently swallow late events. A rejected stale upsert is correct but invisible. Log and count rejections, or a real bug looks identical to normal late-arrival handling.
• Multiple gateways share no sequence space. If two providers are active, normalize into one internal schema with namespaced sequence ids before sequencing — never compare sequences across providers.

The sequence id is often not what you think

Before you trust a field as a monotonic sequence, confirm it actually increases per aggregate and not merely globally or per account. Some providers expose a global event counter that is monotonic across the whole account but skips wildly between two events for the same subscription_id, because thousands of unrelated events landed in between. That is still usable — you only need last + 1 to become “the next value we have actually seen for this key,” which means storing the last applied sequence per subscription and comparing against the buffer head, not asserting the raw numbers are consecutive integers. Other providers give you no numeric sequence at all, only a created_at and a causal hint like a previous_attributes diff or a parent object id. In that case you reconstruct order from the object graph — an invoice.payment_succeeded references its invoice_id, and you refuse to apply it until that invoice_id exists — which is a per-object dependency wait rather than a numeric one, but the buffering shape is identical.

Deleting the aggregate is the ordering edge case that bites hardest

Terminal events deserve special thought because they change what “in order” even means. When a customer.subscription.deleted applies, you may be tempted to purge the ledger rows and the buffer key for that subscription_id. Do not purge eagerly: a late invoice.payment_succeeded for a final proration can still be in flight, and if you have torn down the buffer it has nowhere to wait and no predecessor to reference. Keep the aggregate’s row and its guard active for a grace window past the terminal event — long enough to cover your provider’s worst-case retry horizon, often 24 to 72 hours — and let the version guard reject anything that tries to resurrect status to active after the delete. Treat deletion as just another sequenced state transition, not as a signal to drop the safeguards, and the “cancelled customer regains access” bug simply cannot occur.

Frequently Asked Questions

Why does out-of-order delivery happen at all? Retries, parallel delivery, and network variance mean two events sent seconds apart can arrive in either order. Providers generally do not guarantee ordering.

Is a timestamp enough to order events? Not reliably. Clock resolution and skew make timestamps ambiguous for events generated close together; a monotonic sequence or version per object is stronger where the provider offers one.

What should happen to a stale event? It should be recorded and ignored rather than applied. Silently dropping it loses the evidence that ordering issues are occurring at all.

Can ordering be enforced by processing serially? Per subscription, yes, and that is usually sufficient. Serialising globally destroys throughput for a guarantee nothing actually needs.

Related

• Webhook Retry & Timeout Strategies
• Building Idempotent Webhook Handlers in Node.js
• Outbox Pattern & Event Publishing
• Syncing Subscription Status Across Microservices
• Webhook Processing & Backend State Management