Choosing Idempotency-Key TTLs for Billing Webhooks

An idempotency key is only as good as how long you keep it: expire it too soon and a late provider retry sails through as a fresh event and double-charges; keep it forever and your key store bloats without bound. The TTL is a deliberate policy that must exceed the provider’s maximum retry horizon, and it interacts with whether the key lives in a fast evictable store or a durable one. This page is a focused companion to Idempotency & Event Deduplication and the Redis vs Postgres for webhook idempotency keys decision: how to size the TTL, and what “replay after expiry” should mean.

You reach for this the first time a webhook is redelivered days after the original — because a provider’s retry schedule can span 72 hours or more — and your 24-hour TTL has already let the key expire.

Trade-offs

The TTL trades duplicate-protection window against storage cost. Too short reopens the double-charge window; too long bloats the store. The right value is anchored to the provider’s retry horizon, not a round number. The map contrasts three sizings.

TTL sizing options A too-short TTL reopens the duplicate window, a horizon-matched TTL covers all retries, and an unbounded TTL bloats the store. Too short < retry horizon late retry double-charges never Horizon + margin e.g. 72h + buffer covers all retries the target Unbounded never expires store bloats durable audit only
Anchor the fast-store TTL to the provider retry horizon plus a margin — a round number like 24h is a coincidence, not a policy.

The correct fast-store TTL is the provider’s maximum retry window plus a safety margin — commonly 72+ hours for Stripe. The durable audit record can live far longer (or forever, partitioned), but the dedup key only needs to outlive retries.

Where the retry horizon actually comes from

The horizon is not a single number you can memorize once. Stripe documents retries with exponential backoff over roughly three days for live-mode endpoints, but the effective horizon shifts with your own behavior: if your endpoint returns non-2xx or times out, the provider keeps retrying, and if it disables the endpoint after repeated failures, delivery can resume hours later once you re-enable it. PayPal’s webhook retries stretch across three days with a different schedule, and Adyen will requeue notifications for far longer when your endpoint is unreachable, sometimes on the order of days rather than hours. Because each provider defines the horizon differently, the TTL is per-integration configuration, not a global constant. If you ingest events from three providers into one handler, size the fast-store TTL to the longest of the three horizons so a single expiry rule stays safe for all of them, and record which provider drove the number in a comment so the next engineer does not “optimize” it downward.

Sizing the margin, not guessing it

The margin exists to absorb three concrete sources of late delivery that sit on top of the documented horizon. The first is clock skew between the provider’s retry scheduler and your own store’s expiry clock — a few seconds in practice, but non-zero. The second is queue lag inside your own pipeline: if a webhook lands during an incident and sits in a backlog for six hours before the handler claims its key, the key’s clock has effectively started six hours late relative to the event’s created timestamp. The third is manual replay — an on-call engineer re-sending a batch of invoice.payment_succeeded events from the provider dashboard after a bug fix, which can arrive well after the automated schedule has stopped. A twelve-hour margin on a seventy-two-hour horizon is not padding for its own sake; it is the sum of these effects rounded up. If your ingestion pipeline can backlog for longer than the margin during an incident, key the TTL off the time you claim the key, not the event’s created field, so backlog never eats into the protection window.

The storage-cost side of the trade-off

The bloat worry is often overstated for the fast store and understated for the durable one. A dedup key is tiny — an event id like evt_1P9x... plus a one-byte value is well under a hundred bytes in Redis once you account for overhead. At ten thousand billing events per hour and an eighty-four-hour TTL, the fast store holds at most about 840,000 live keys, on the order of tens of megabytes — negligible next to the memory a Redis instance already reserves. The real cost lives in the durable tier, where every event_id you retain for audit is a row that also carries received_at, an index entry, and often the raw payload. That table grows linearly and forever unless you partition and drop, which is exactly why the two tiers get different lifetimes: the fast store is cheap and short, the durable store is expensive and must be actively pruned.

Step-by-Step Implementation

The five steps find the retry horizon, set the fast-store TTL above it, keep the durable key for audit, define replay-after-expiry semantics, and monitor. The two-tier diagram shows the fast key (TTL-bounded) versus the durable key (retention-bounded).

Two-tier key retention The Redis dedup key expires just past the retry horizon while the Postgres audit key is retained for the audit-retention horizon. Fast dedup key (Redis) TTL = retry horizon + margin e.g. 72h purpose: collapse retries Durable key (Postgres) retention = audit horizon months, partitioned purpose: audit + backstop
Two keys, two lifetimes — the fast key expires just past retries, the durable key lives for audit and never lets a cache flush admit a duplicate.

1. Set the fast-store TTL from the retry horizon

Read the provider’s documented maximum retry window and add a margin for clock skew and delayed delivery.

from datetime import timedelta

PROVIDER_RETRY_HORIZON = timedelta(hours=72)   # Stripe's documented max
MARGIN = timedelta(hours=12)                    # skew + delayed delivery

IDEMPOTENCY_TTL_SECONDS = int((PROVIDER_RETRY_HORIZON + MARGIN).total_seconds())  # 84h

def claim(redis, event_id: str) -> bool:
    return bool(redis.set(f"idem:{event_id}", "1", nx=True, ex=IDEMPOTENCY_TTL_SECONDS))

2. Keep the durable key for the audit horizon

The Postgres unique constraint on the raw event survives a cache flush and doubles as the audit trail. Partition by month and drop partitions past the retention horizon.

CREATE TABLE processed_events (
  event_id    TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (event_id, received_at)
) PARTITION BY RANGE (received_at);   -- drop old partitions per retention policy

3. Define replay-after-expiry semantics deliberately

Once both TTL and audit retention have elapsed, a redelivered event will reprocess. Decide whether that is correct — usually it is, because the provider stopped retrying long ago and any delivery now is a genuine new request or a manual replay.

def is_expected_reprocess(event_id: str, received_at, now) -> bool:
    # If the provider's retry horizon has fully elapsed, a fresh delivery is legitimate.
    return (now - received_at) > (PROVIDER_RETRY_HORIZON + MARGIN)

“Reprocess” here does not mean “re-run the side effect blindly.” For a charge.succeeded event that arrives four days after the original, the right behavior is usually to re-derive state, not to re-issue money movement: look up whether invoice_id is already marked paid, and if it is, record the delivery and return 200 without touching the ledger. The TTL governs how long the cheap dedup check protects you; past it, correctness falls back to the natural idempotency of the domain operation itself. Handlers that lean on the TTL as their only guard are the ones that double-apply when a manual replay lands after expiry. Treat the expired-key path as the moment your handler’s own conditional logic — “is this subscription_id already at this state?” — has to carry the load.

4. Monitor idempotency hit-rate and key-store size

A TTL that is wrong fails silently, so instrument it. Emit a counter every time claim() returns false (a deduplicated retry) versus true (a first delivery); the ratio is your idempotency hit-rate. A healthy billing endpoint sees a small but non-zero hit-rate — retries happen — and a sudden spike toward zero often means keys are expiring before retries arrive, which is the double-charge condition forming. Track the fast store’s live key count and its memory footprint as a time series so the eighty-four-hour steady state is visible; a slow upward drift means either event volume grew or a TTL was silently dropped from a set call somewhere. Alert on the durable table’s partition count too, because the failure mode there is not a crash but a table that has quietly reached hundreds of millions of rows because nobody wired up the partition-drop job.

def claim_with_metrics(redis, statsd, event_id: str) -> bool:
    fresh = bool(redis.set(f"idem:{event_id}", "1", nx=True, ex=IDEMPOTENCY_TTL_SECONDS))
    statsd.increment("webhook.idempotency.first" if fresh else "webhook.idempotency.dedup")
    return fresh

5. Keep the TTL and the horizon coupled in code

The most durable mistake is letting the TTL drift away from the horizon it was derived from. Store both as named values in one place — PROVIDER_RETRY_HORIZON and the MARGIN — and compute IDEMPOTENCY_TTL_SECONDS from them rather than pasting a literal like 302400 into a set call. A raw integer buried in application code loses its provenance the moment it is copied; six months later someone tuning cache memory sees “84 hours” with no explanation and trims it to 24. A config test that asserts IDEMPOTENCY_TTL_SECONDS > PROVIDER_RETRY_HORIZON.total_seconds() turns that silent regression into a red build.

Verification & Testing

The tests prove the TTL exceeds the retry horizon, a within-window retry is deduplicated, an after-horizon delivery reprocesses, and a cache flush does not admit a duplicate. Use a mock clock to advance to 71 hours and assert a redelivery is a no-op; advance to 85 hours and assert it reprocesses. Flush the fast store and assert the Postgres constraint still catches a duplicate. The panel lists them.

The boundary cases are where TTL bugs actually hide, so test the edges deliberately. Advance the clock to exactly the horizon plus margin minus one second and assert the retry still deduplicates; advance one second past and assert the key is gone. Off-by-one errors between seconds and milliseconds, or between > and >= in the reprocess predicate, only surface at that seam. Do not test with a real sleep — a suite that waits eighty-five real hours never runs, and one that shortens the TTL to two seconds “for testing” is validating a different code path than production. Inject the clock and the TTL as parameters so the test exercises the exact production constant. It is worth adding one explicit assertion that the deduplicated retry produced no ledger row and no outbound charge, because “the second call returned 200” is not the same guarantee as “the second call moved no money”; a handler can swallow the duplicate at the wrong layer and still emit a charge on the redelivery.

One more test earns its place: a concurrency race on first delivery. Fire two copies of the same event_id at the handler in parallel and assert exactly one wins the SET NX and one observes the key already present. The nx=True flag makes the claim atomic in Redis, but the surrounding handler must actually branch on the return value — a handler that claims the key and then unconditionally processes has no protection against the two simultaneous deliveries a provider will occasionally send when its own retry fires before the original’s 2xx is recorded.

TTL tests A within-window retry deduplicates, an after-horizon delivery reprocesses, a cache flush is caught by Postgres, and the TTL exceeds the retry horizon. Within window 71h retry deduplicated After horizon 85h delivery reprocesses Cache flush Redis cleared PG catches TTL > horizon assertion config test
The after-horizon reprocess test encodes the deliberate policy — reprocessing an ancient delivery is correct, not a bug.

Gotchas & Production Pitfalls

The pitfalls are round-number TTLs, TTL shorter than the retry horizon, forgetting the durable backstop, and unbounded audit growth. The map groups them.

TTL pitfalls A round-number TTL, a TTL shorter than the retry horizon, no durable backstop, and unbounded audit growth are the recurring pitfalls. Round number 24h "feels right" → anchor to horizon Too short retry after expiry → horizon + margin No backstop Redis-only → durable constraint Unbounded audit grows forever → partition + drop
Four pitfalls — a TTL shorter than the retry horizon is the one that silently double-charges.
  • Round-number TTLs. 24 hours “feels safe” but is shorter than many providers’ retry windows. Derive the TTL from the documented horizon, not intuition.
  • TTL shorter than the retry horizon. This is the double-charge bug: the key expires, a late retry looks new, and the effect runs twice. Always exceed the horizon with margin.
  • No durable backstop. A Redis-only TTL means a cache flush reopens the whole window. Keep a durable unique constraint that survives eviction.
  • Unbounded audit growth. The durable key is also an audit record and grows forever if unmanaged. Partition by month and drop partitions past your retention horizon.

The pitfall that hides longest is a TTL that was correct when written and silently wrong after a provider changed its retry policy. Providers publish their retry horizons, and they do adjust them — a window that was 72 hours can become three days of exponential backoff, or a new event type can carry a different retry schedule than the one you sized against. A TTL hard-coded to a number that matched the old policy will not error when the policy changes; it will simply start expiring keys before the last retry, reopening the double-charge window for exactly the events that retried longest. The defense is to treat the retry horizon as a named, documented constant with a comment citing the provider’s published figure and a periodic check that reconciles it against the provider’s current documentation, so a policy change surfaces as a review task rather than as a reconciliation discrepancy weeks later.

A second slow failure is TTL drift between environments. If staging runs a shorter TTL to keep its key store small and production runs the real horizon, a load test or a replay in staging can exhibit deduplication behavior that production will not — a test that “passes” against a 1-hour TTL tells you nothing about a 72-hour production window. Derive the TTL from a single source of truth shared across environments, and let the only difference be data volume, never the retention logic itself. The whole point of coupling the TTL to the retry horizon in code is defeated the moment two deployments disagree about what that horizon is.

Frequently Asked Questions

Should the TTL differ per provider? Yes, if you ingest from several. Size the fast store to the longest retry horizon among them so one expiry rule stays safe for all.

Does a longer TTL cost anything real? Very little in the fast store, where keys are tiny. The cost concentrates in the durable audit tier, which is why the two have different lifetimes.

What should the durable retention be? Long enough to answer a customer dispute, which is usually months rather than days, with partitioning so old data can be dropped cheaply.

How is a wrong TTL detected? By monitoring the deduplication hit rate. A rate trending toward zero means keys are expiring before retries arrive, which is the double-charge condition forming.