Idempotency & Event Deduplication

Idempotency is the property that processing the same billing event twice produces the same result as processing it once — and it is the single most important guarantee in Webhook Processing & Backend State Management. Payment providers deliver at least once: a network blip, a slow 200, or an aggressive retry policy means the same invoice.paid can land two, five, or fifty times. Without a deduplication layer, each delivery is a fresh charge, a fresh ledger entry, a fresh dunning email. Event deduplication turns that unreliable stream into exactly-once effects so that ledger integrity holds through charge successes, dunning failures, and tax recalculations alike. This page covers the key architecture, the storage choices that enforce it, and how it composes with state machines and the ledger.

It helps to separate two ideas that are easy to conflate. Idempotency is a property of a handler: calling apply_payment(invoice_id, 4990) a second time must not move the balance a second time. Deduplication is the mechanism that recognizes a repeat delivery and short-circuits it before the handler runs at all. You need both, and they defend against different failures. A perfectly idempotent handler with no dedup layer will still re-run expensive side effects — a second call to a metered tax API, a second row in an append-only ledger, a second entry in the outbox — even if the net balance ends up correct. A dedup layer with a non-idempotent handler will usually be correct until the one race where two deliveries slip past the filter concurrently and both mutate state. The rest of this page treats the two as a pair: the key claim is the deduplication gate, and the transactional effect behind it is written to be idempotent so that the gate can fail open without corrupting money.

The stakes are asymmetric, which shapes every design decision below. A false negative — admitting a duplicate — double-charges a customer, triggers a chargeback, and costs support hours plus a dispute fee that is often larger than the transaction itself. A false positive — dropping a distinct-but-similar event — silently loses a real payment or a real cancellation, and because nothing errors, you find out weeks later during a reconciliation run. Given that asymmetry, the design leans toward a durable database constraint as the authority rather than a probabilistic in-memory filter, and toward keys derived from provider-supplied identity rather than from fields your own code computes.

Prerequisites

Exactly-once effects are not one mechanism but a stack of them: a deterministic key, an atomic claim store, a versioned aggregate for ordering, and an outbox for downstream reliability. Each prerequisite closes a gap the others cannot. The stack shows the dependencies before the checklist.

• A deterministic idempotency key derivable from every inbound event
• An idempotency store: a Postgres UNIQUE-constrained table or Redis with SET NX
• The webhook signing secret available to the handler for signature verification
• A versioned subscription table for monotonic out-of-order event handling
• A transactional outbox table for reliable downstream publishing
• All event timestamps normalized to UTC

The ordering of these prerequisites is not cosmetic. Signature verification must happen before the key claim, because an attacker who can reach your endpoint can otherwise poison your dedup store: by replaying a captured event ID they could pre-claim a key so that the legitimate delivery is treated as a duplicate and dropped. Verify the HMAC first, and only a party holding the signing secret can influence what lands in webhook_ingest_log. Likewise, the versioned aggregate must exist before you rely on it for ordering — retrofitting a version column onto a live subscription_id table means backfilling every existing row and choosing a starting value that will not collide with the next in-flight event, which usually forces a brief write freeze on the table.

The tenant dimension deserves explicit attention in multi-tenant billing systems. The unique constraint below is on (tenant_id, idempotency_key), not on idempotency_key alone. Two different customers on two different Stripe accounts can legitimately produce the same provider event ID namespace, and a single-column unique index would cause tenant B’s event to be silently deduplicated against tenant A’s. Scoping the key to the tenant also keeps the index locally dense, which matters when you partition by month: each tenant’s keys cluster together and the renewal-day burst for one tenant does not thrash the whole index.

Architecture & Data Flow

The idempotency gate sits between signature verification and business logic. The inputs are verified, possibly-duplicate events; the processing step claims the key atomically and runs the effect in a transaction; the outputs are exactly-once ledger entries and outbox events. A duplicate never reaches business logic — it is acknowledged and dropped at the gate.

Why the gate acknowledges rather than errors

The single most consequential behavioral choice at the gate is what HTTP status a duplicate receives. It must be a 200, not a 409 or a 500. Providers interpret any non-2xx response as a failed delivery and schedule another retry with backoff, so returning an error on a duplicate does not protect anything — it guarantees the provider keeps re-sending the same event, widening the window in which a concurrent worker might race the claim. Returning 200 tells the provider the event is settled and stops the retry sequence. The mental model is that your endpoint acknowledges receipt and terminal handling, not first-time processing: from the provider’s perspective there is no difference between “I applied this” and “I already applied this,” and there should not be. The only responses that should ever be non-2xx are a failed signature check (401) and a genuine internal failure where you want the retry (500), because in the latter case the key claim will have rolled back and the retry is exactly the recovery path you want.

The gate is synchronous, the effect can be deferred

A common refinement at scale is to split the gate from the effect. The webhook handler claims the key, writes the raw verified payload into an ingest row, returns 200 within the provider’s timeout budget (Stripe expects a response inside roughly 20 seconds, and a slow handler that blocks on tax APIs will blow through it), and a separate worker pool drains the ingest table and runs the business logic. This keeps the request path fast and bounded while preserving exactly-once semantics, because the key claim — the part that must be synchronous with the HTTP request to dedup concurrent deliveries — is cheap, and the expensive, retry-prone effect runs behind it where a slow tax lookup cannot cause a provider-side retry. The trade-off is that “processed” now has two states — claimed and completed — and your stale-row sweeper must distinguish an ingest row that is mid-flight in a worker from one orphaned by a crash, which the status column and a claimed_at timestamp handle.

Implementation Walkthrough

The four steps move from identity to effect: derive a key, claim it atomically, run the effect in one transaction, then gate concurrent ledger writes. The claim is the lock — everything after it assumes the current worker is the sole owner of this event. The sequence shows the flow.

1. Derive a deterministic key

Prefer the provider’s event ID. When the provider does not supply one, hash stable fields so retries of the same logical event collide.

import hashlib

def idempotency_key(event: dict) -> str:
    if event.get("id"):
        return event["id"]                                  # provider event id, best case
    raw = f'{event["customer_id"]}:{event["amount_cents"]}:{event["currency"]}:{event["created_utc"]}'
    return hashlib.sha256(raw.encode()).hexdigest()         # deterministic fallback

The field selection in the fallback hash is where subtle bugs live. Every field you include must be stable across retries of the same logical event, and every field that distinguishes two different events must be present. Including a mutable field — a retry_count the provider increments, a received_at your own edge stamps on arrival, or a load-balancer request ID — breaks the collision and lets a retry through as if it were new. Omitting a discriminating field does the opposite: if a customer legitimately makes two identical 4990 charges in the same second and your hash covers only customer_id, amount_cents, and currency at second resolution, the second charge collides with the first and is wrongly dropped. This is why the provider’s own event ID is always preferable: it is opaque, monotonic per event, and the provider guarantees it is stable across the retries of one delivery and distinct across separate events. Reach for the hash only when the provider truly gives you nothing, and when you do, prefer the finest timestamp resolution the provider exposes and include a provider sequence number if one is available.

One further nuance: the key must be derived from the event content, never from anything about the delivery attempt. A retry of an event is a fresh HTTP request with a fresh TCP connection and often a fresh trace ID, so any identity you compute from the transport layer will differ on every retry and defeat the whole scheme. Treat the verified JSON body as the sole source of identity.

2. Claim the key with a unique constraint

The insert is the lock. Concurrent duplicates race for the constraint; exactly one wins.

CREATE TABLE webhook_ingest_log (
  idempotency_key    VARCHAR(255) NOT NULL,
  tenant_id          UUID         NOT NULL,
  provider_event_id  VARCHAR(255),
  status             VARCHAR(16)  NOT NULL DEFAULT 'pending',
  created_at         TIMESTAMPTZ  NOT NULL DEFAULT now(),
  CONSTRAINT uq_tenant_idempotency UNIQUE (tenant_id, idempotency_key)
);

-- Claim: zero rows returned ⇒ duplicate, ack and stop.
INSERT INTO webhook_ingest_log (idempotency_key, tenant_id, provider_event_id)
VALUES (:idempotency_key, :tenant_id, :provider_event_id)
ON CONFLICT (tenant_id, idempotency_key) DO NOTHING
RETURNING idempotency_key;

The reason INSERT ... ON CONFLICT DO NOTHING ... RETURNING is the right primitive — rather than a SELECT to check existence followed by an INSERT — is that the check-then-act pattern has a race window between the two statements. Two workers processing concurrent duplicates both SELECT, both see no row, both proceed to INSERT, and one gets a constraint violation while the other has already started the effect: you now have two workers believing they won. The single-statement upsert collapses the check and the claim into one atomic operation guarded by the unique index, so exactly one worker gets a non-empty RETURNING result and every other gets zero rows. The database’s own locking on the index does the mutual exclusion for you, with no application-level lock to leak on a crash.

Note that the claim row starts in status = 'pending', not 'done'. The row’s existence proves someone claimed the key; a separate transition to 'done' (or deleting the row, depending on your retention model) proves the effect committed. Keeping that distinction is what lets the crash-recovery sweeper tell an orphaned claim from a completed one. If you instead marked the key done at claim time, a crash between claim and effect would leave a done row with no effect behind it, and the event would be swallowed forever — the exact failure the transactional pattern in step 3 exists to prevent.

3. Process the effect inside one transaction

The dedup claim and the state change must commit or roll back together; otherwise a crash leaves a claimed key with no effect, permanently swallowing the event. The subtle requirement here is that the INSERT into webhook_ingest_log and the mutation of the subscription_id row must share one transaction. If the claim commits in its own transaction and the effect in another, then a process death in the gap — an OOM kill, a deploy rollout draining the pod, a Postgres failover — lands you in precisely the swallowed-event state: the key is claimed, so every retry deduplicates and stops, but the balance never moved. Wrapping both in a single BEGIN/COMMIT makes that gap non-existent; either the retry finds no claim and re-runs cleanly, or it finds a committed claim backed by a committed effect.

async function handleEvent(event: { subId: string; sequence: number; delta: Record<string, unknown> }) {
  const version = await db.getSubscriptionVersion(event.subId);
  if (event.sequence <= version) {
    return { action: 'DEDUPLICATE', version };              // ⚠️ stale / replay
  }
  const tx = await db.beginTransaction();
  try {
    await tx.updateSubscription(event.subId, event.delta);
    await tx.insertOutbox(event.subId, 'subscription.updated', event.delta);
    await tx.commit();
    return { action: 'APPLY', version: event.sequence };    // ✅ exactly once
  } catch (err) {
    await tx.rollback();                                    // ✗ key not retained
    throw err;
  }
}

Notice that the TypeScript handler above carries a second dedup dimension beyond the key claim: the event.sequence <= version guard. The key claim answers “have I seen this exact event before?”; the version guard answers “is this event still relevant, or has a newer one already superseded it?” These are different questions. A subscription.updated that lowers the seat count can arrive after a later subscription.updated that raised it — both are distinct events with distinct keys, so the key claim admits both, but applying the older one last would corrupt the current state. The monotonic version check discards the stale delta without treating it as a duplicate. Keeping the two mechanisms separate is deliberate: collapsing ordering into the key claim would either reject legitimate distinct events or admit stale ones, depending on which way you cut the corner.

4. Gate concurrent ledger writes

Tax and ledger math must run once per cycle. Use row locking or optimistic concurrency to serialize.

def apply_ledger_entry(account_id: str, amount_cents: int, tax_rate_bps: int, expected_version: int) -> None:
    row = db.query("SELECT balance, version FROM ledger WHERE id = %s FOR UPDATE", account_id)
    if row.version != expected_version:
        raise ValueError("Ledger version mismatch; retry with latest state")  # ⚠️ concurrent write
    # tax in basis points keeps money in integer minor units — never float
    gross = amount_cents + (amount_cents * tax_rate_bps) // 10_000
    db.execute(
        "UPDATE ledger SET balance = %s, version = %s WHERE id = %s AND version = %s",
        row.balance + gross, row.version + 1, account_id, row.version,
    )

The choice between the pessimistic FOR UPDATE lock shown here and pure optimistic concurrency (dropping the FOR UPDATE and relying solely on the WHERE ... AND version = %s guard on the UPDATE) comes down to contention. Under low contention — most billing accounts see one event at a time — optimistic concurrency is cheaper because it never blocks and only pays a cost on the rare conflict, which retries against fresh state. Under a renewal-day burst where many events target the same handful of high-volume accounts, optimistic retries can livelock, each attempt losing the version race to another, and the pessimistic row lock’s queueing behavior gives more predictable throughput. A practical rule: use the version-guarded UPDATE alone for per-subscription ledgers where write concurrency to any single row is naturally low, and add FOR UPDATE when the lock target is a shared aggregate such as a tenant-level balance or a monthly usage counter that many events legitimately touch at once.

Whichever you choose, the money arithmetic stays in integer minor units end to end. The (amount_cents * tax_rate_bps) // 10_000 expression keeps a 4990-cent charge at a 875-bps rate as exact integer cents (436), and the floor division makes the rounding rule explicit and reproducible rather than dependent on a float’s binary representation. A retry that re-derives the same gross from the same inputs lands on the same integer, which is what makes the tax figure safe to cache against the idempotency key in the first place.

Edge Cases & Failure Modes

The failure modes divide by which layer they attack: the network duplicates and reorders, the clock skews, and the cache evicts. The map sorts them so the defense — a DB constraint, a monotonic version, or treating Redis as advisory — is obvious.

The stale-processing sweeper and its grace period

The most operationally dangerous entry in that table is the swallowed event — a key left in processing after a crash. The sweeper that recovers it is deceptively easy to get wrong. A naive sweeper that reclaims any processing row older than, say, 30 seconds will happily reclaim a key whose worker is merely slow (a tax API timing out at 25 seconds, a Postgres failover pausing writes), and now two workers run the effect concurrently — the exact double-effect the whole system exists to prevent. The grace period must therefore exceed the maximum plausible legitimate processing time by a wide margin, and the reclaim itself must be atomic: UPDATE webhook_ingest_log SET status = 'pending', claimed_at = now() WHERE idempotency_key = :k AND status = 'processing' AND claimed_at < now() - interval '15 minutes' RETURNING ..., so that only one sweeper wins the reclaim and the original worker, if it later wakes, finds its row no longer in the state it expects and aborts. Pair the sweeper with a fencing token or the version guard so a resurrected zombie worker cannot commit against state that has moved on.

Amount or currency drift between duplicates

A rarer but instructive failure is a “duplicate” that is not actually identical. Some providers, after a partial refund or a currency correction, re-emit an event with the same logical identity but a changed amount_cents. If your key is the provider event ID, both deliveries share a key and the second is dropped — usually correct, since the provider event ID is meant to be immutable. But if your key is a content hash that includes amount_cents, the corrected amount produces a different key and sails through as a new event, double-applying. This is the deeper reason to prefer provider event IDs: they encode the provider’s own notion of event identity, which is the notion you actually want to deduplicate against. When you must hash, log the pre-image of every hash so that a support investigation can reconstruct why two deliveries did or did not collide.

Performance & Scale

Deduplication has a two-tier performance shape: a Redis SET NX pre-filter handles the common case in memory, and a Postgres unique constraint is the authority that never admits a duplicate even after a cache flush. The diagram shows the two tiers and why the database always has the final say.

Unique-constraint dedup is O(1) per event and scales to the renewal-day burst as long as the index stays hot — partition webhook_ingest_log by month and drop old partitions per your retention policy. Layer Redis SET NX as a fast pre-check in front of Postgres for high-throughput endpoints, but never as the authority: an eviction during a billing-cycle close must not admit a duplicate, so the database constraint always has the final say. Size key TTLs to exceed the provider’s maximum retry window (commonly 72 hours for Stripe) so a late retry still finds its claim. Keep the locked critical section minimal — claim the key, do the effect, commit — to avoid lock contention during renewal spikes.

The renewal-day burst is worth quantifying because it drives the sizing. A base of, say, 200,000 monthly subscriptions does not spread its invoice.created / invoice.paid / invoice.payment_succeeded events evenly across the month — a large fraction anchor to the first of the month, and each subscription emits several events per cycle. A conservative estimate is three to five webhook deliveries per subscription per renewal, concentrated into a few hours, which turns 200,000 subscriptions into a peak on the order of a few thousand claims per second against webhook_ingest_log. At that rate the cost that dominates is not the insert itself but index maintenance: an unpartitioned table whose index no longer fits in the buffer cache starts paying random-read latency on every claim, and p99 claim time climbs from sub-millisecond into tens of milliseconds precisely when you can least afford it. Monthly partitioning keeps the live partition’s index small enough to stay resident, and because dedup only ever queries recent keys (nothing older than the retry window can still arrive), the historical partitions can be detached and archived without affecting the hot path.

Sizing the retry window against the TTL

The TTL on the Redis pre-filter and the retention on the Postgres claim rows are governed by the same constraint from opposite ends: both must outlive the provider’s maximum retry window, or a late-but-legitimate retry will find no record of its earlier claim and be treated as a first delivery. Stripe retries a failing webhook endpoint with exponential backoff for up to three days; other providers stretch to a week. Set the Postgres retention to the provider’s window plus a safety margin — a common choice is the window plus one billing cycle, so that a reconciliation job can still see the claim while investigating a disputed charge. The Redis TTL can be shorter than the Postgres retention because Redis is only the pre-filter: if a retry arrives after the Redis key has expired but within the Postgres retention, the pre-filter misses, the request falls through to the INSERT ... ON CONFLICT, and the durable constraint catches it. That fall-through is by design and is why an aggressive Redis TTL trades a little extra database load for a lot less Redis memory without ever risking a duplicate.

The one number you must never undersize is the Postgres retention relative to the retry window. If a claim row is garbage-collected while the provider can still retry, you have reintroduced the double-effect risk through the back door. When in doubt, retain longer — claim rows are tiny (a key, a tenant, a status, two timestamps), and even at a few thousand per second the storage cost of holding months of them is trivial next to the cost of one wrongful double charge.

Testing Strategy

The tests attack the guarantee from four angles: concurrent replay, forged signature, out-of-order sequence, and a crash between claim and commit. The last is the subtlest — a claimed key with no committed effect must not permanently swallow the event. The panel lists them.

Write a replay test that submits the same provider_event_id N times concurrently and asserts exactly one ledger entry and one outbox row. Use a mock clock to make TTL-expiry and grace-period boundaries deterministic. Forge a webhook with an invalid signature and assert it is rejected before the idempotency gate is even reached. Inject a sequence of out-of-order events and assert the monotonic version guard discards the stale ones. Finally, simulate a crash between key claim and commit (kill the transaction) and assert the key is not retained, so a retry can still succeed.

The replay test only proves something if it exercises real concurrency, not a serial loop. A for loop that submits the same event ten times in sequence passes trivially against almost any implementation, including a broken check-then-insert, because there is never two workers in the critical section at once. The test that actually catches the race launches N deliveries against a real transactional database — a Testcontainers Postgres, not an in-memory stub — with a barrier that releases all N threads simultaneously, and then asserts SELECT count(*) FROM ledger WHERE ... is exactly one and SELECT count(*) FROM outbox WHERE ... is exactly one. Run it with N in the dozens and repeat the whole test a few hundred times in CI, because a one-in-a-thousand race will not show up in a single pass; a flaky assertion here is not flakiness to retry away but a genuine concurrency bug the harness happened to surface.

Property-based and reconciliation tests

Beyond the four scripted angles, a property-based test earns its keep here: generate a random interleaving of duplicates, out-of-order events, and distinct events for a single subscription_id, feed them through the handler in a shuffled order, and assert that the final ledger balance equals the balance you get from applying only the causally latest version of each distinct event exactly once. Because idempotency and ordering together claim that the outcome is independent of delivery order and multiplicity, that invariant is exactly what a property test can hammer with thousands of random schedules. Pair it with a periodic reconciliation check in production — sum the ledger deltas for each account and compare against the provider’s reported balance for the same period — so that any dedup defect that slips past the tests surfaces as a bounded, detectable drift rather than silent corruption discovered months later.

Frequently Asked Questions

How do I handle idempotency when the provider does not supply a unique event ID? Build a deterministic key by hashing stable fields — provider timestamp, customer ID, amount in minor units, and currency. Insert that hash into a unique index before processing. Two retries of the same logical event produce the same hash and collide on the constraint, so only the first takes effect.

Does idempotency guarantee exactly-once processing in distributed systems? Idempotency guarantees that repeated identical requests cause the same single state change; it does not stop the network from duplicating deliveries. You reach effective exactly-once by combining at-least-once delivery, a deduplication filter (the key claim), and an outbox so that the publish and the commit are atomic.

How should tax calculation engines interact with idempotent handlers? Make tax math deterministic, or cache the result keyed by the idempotency key. On a retry, return the cached tax figure rather than re-invoking the external tax API — that keeps the amount stable and avoids a second metered API call.

What database isolation level is recommended for ledger updates? Use REPEATABLE READ or SERIALIZABLE to prevent phantom reads during concurrent invoice generation and dunning transitions. Combine it with SELECT ... FOR UPDATE or a version check so conflicting writes fail fast and retry against fresh state.

Should the idempotency key live in the URL, a header, or the event body? For provider-driven webhooks you rarely get a choice — the key is whatever identity the provider embeds in the signed body (the event ID), and you extract it after verifying the signature. For your own internal APIs that mutate billing state, take an explicit Idempotency-Key request header, the convention Stripe popularized, and scope the stored key to the authenticated tenant plus the operation. Do not derive the key from the URL path alone, because two semantically different POSTs to the same path (two separate charges on one customer_id) would collide and the second would be wrongly deduplicated.

Can I just use INSERT ... ON CONFLICT DO NOTHING and skip Redis entirely? Yes, and for most workloads that is the correct default. Redis buys you a cheap in-memory rejection for the common repeat-delivery case, which spares the database a round trip, but it is strictly an optimization — the Postgres unique constraint is what makes the guarantee. Add Redis only when you have measured that duplicate deliveries are a meaningful fraction of your ingress and the database round trip to reject them is a real cost. Adding it speculatively just gives you a second store to keep consistent and a new eviction failure mode to reason about.

How does idempotency interact with the transactional outbox? They compose cleanly because both hang off the same transaction. The key claim, the state mutation, and the INSERT into the outbox all commit together, so a duplicate that loses the key claim never writes an outbox row, and a first delivery writes exactly one. The outbox relay downstream then needs its own idempotency on the consumer side — a delivered message can still be redelivered — but the producing side is already exactly-once by construction. This is why the outbox pattern and the key claim are described here as one stack rather than two independent features.

What should I return for a duplicate that arrives while the original is still processing? This is the in-flight case, distinct from a duplicate of an already-completed event. The second delivery loses the key claim (the row exists in processing), so it must not run the effect — but the effect is not done yet either. Return 200 and drop it: the provider is satisfied, and the original worker will complete the effect. Do not block the second request waiting for the first to finish, since that ties up a connection and risks the provider’s timeout; the whole point of claiming before processing is that a loser can bow out immediately.

Related

• Building Idempotent Webhook Handlers in Node.js
• Choosing Idempotency-Key TTLs for Billing Webhooks
• Database Sync & Consistency Patterns
• Outbox Pattern & Event Publishing
• Webhook Retry & Timeout Strategies
• Webhook Processing & Backend State Management