Dead-Letter Queues for Unprocessable Billing Webhooks

A billing webhook consumer that retries forever will eventually meet an event it can never process — a payload referencing a subscription that was deleted, a schema the code does not understand, a bug that throws on one specific customer. Without somewhere for that event to go, it either blocks the partition behind it or spins in a retry loop indefinitely, and in both cases every event behind it stops being applied. This guide sits under Webhook Retry & Timeout Strategies and covers building that escape hatch.

The stakes in billing are higher than in most pipelines. A blocked consumer means payment outcomes are not being applied: customers who paid stay locked out, customers who churned keep access, and dunning runs against stale state. The failure is silent from the outside — the provider is delivering successfully, your endpoint is returning 200 — while the effects accumulate.

The design goal is that one bad event costs one event, not a pipeline. Everything below serves that.

Trade-offs

Failure handling Blast radius Recovery Observability Notes
Retry forever The whole partition Manual intervention Poor — looks healthy The default and the worst
Drop after N retries One event, lost None Poor Silent data loss
Dead-letter after N retries One event, parked Replay after a fix Good The right shape
Fail the whole batch Everything Redeliver all Noisy Only for genuine outages
Blocked partition versus dead-letter routing Without a dead-letter path a single unprocessable event blocks every event behind it, while routing it aside lets the pipeline continue. No dead-letter path poison event everything behind it waits, indefinitely With a dead-letter path poison event dead-letter store pipeline continues, one event parked
The difference is one bounded loss versus an unbounded one — and the unbounded case reports itself as healthy.

Dropping after N retries deserves explicit rejection. It converts a visible problem into an invisible one, and in billing the dropped event is frequently the payment outcome that would have unlocked a customer’s account.

Step-by-Step Implementation

1. Classify the failure

class Retryable(Exception): ...      # database unavailable, lock timeout, provider 5xx
class Permanent(Exception): ...      # unknown event kind, schema mismatch, missing entity

def handle(event, db):
    try:
        return apply(event, db)
    except (ConnectionError, TimeoutError) as e:
        raise Retryable(str(e)) from e
    except (KeyError, ValidationError) as e:
        raise Permanent(str(e)) from e

Permanent failures should go straight to the dead-letter store without consuming the retry budget. Retrying a schema mismatch twelve times delays the alert by an hour and changes nothing.

2. Give each event a bounded budget

MAX_ATTEMPTS = 6
BACKOFF_SECONDS = [5, 30, 120, 600, 1800, 3600]

def next_attempt_delay(attempt: int) -> int | None:
    if attempt >= MAX_ATTEMPTS:
        return None                    # exhausted: dead-letter it
    return BACKOFF_SECONDS[attempt]

Six attempts across roughly an hour covers essentially every transient failure. Longer budgets mainly delay the discovery of real bugs.

3. Store enough to diagnose and replay

Dead-letter record contents The raw payload, the signature and headers, the failure class and message, and the attempt history are all required to diagnose and replay an event. Raw payload exactly as received byte-for-byte Headers signature, timestamps re-verifiable Failure class and message plus stack trace Attempts when and why each failed
Storing the raw bytes and headers is what makes a replay a genuine reprocessing rather than a reconstruction.
CREATE TABLE webhook_dead_letters (
  dead_letter_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  provider       TEXT NOT NULL,
  event_id       TEXT NOT NULL,
  raw_body       BYTEA NOT NULL,
  headers        JSONB NOT NULL,
  failure_class  TEXT NOT NULL,        -- 'permanent' | 'exhausted'
  failure_detail TEXT NOT NULL,
  attempts       INT NOT NULL,
  first_seen_at  TIMESTAMPTZ NOT NULL,
  parked_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  replayed_at    TIMESTAMPTZ,
  UNIQUE (provider, event_id)
);

Namespacing on (provider, event_id) matters for the same reason it does in the live path, particularly during a provider migration when two providers are emitting simultaneously.

4. Replay through the same handler

def replay(dead_letter_id: str, db, handler) -> str:
    dl = db.get_dead_letter(dead_letter_id)
    event = handler.parse_webhook(dl.raw_body, dl.headers["signature"])
    try:
        handler.handle(event, db)          # the SAME idempotent path as live traffic
    except Exception as e:
        db.record_replay_failure(dead_letter_id, str(e))
        return "still_failing"
    db.mark_replayed(dead_letter_id)
    return "replayed"

Replaying through a separate code path is the mistake that turns a recovery into a second incident. The live handler is idempotent — that property is exactly what makes replay safe — so use it.

5. Alert on arrivals, not attempts

A retry is normal. An arrival in the dead-letter store is not, and it should page during business hours and at minimum create a ticket outside them. Alert on the arrival rate and on the age of the oldest unreplayed record, since a dead-letter store nobody drains is the same as dropping events with extra steps.

Draining the Queue Without Making It Worse

A dead-letter store accumulates two very different populations, and treating them the same is what causes replay incidents.

The first population is events that failed because of a bug now fixed. These are safe to replay in bulk once the fix is deployed, and the correct order is by original receipt time so that state transitions apply in sequence. Replaying newest-first can apply a cancellation before the payment that preceded it, and while an idempotent, ordered handler should resist that, relying on it unnecessarily is poor practice.

The second population is events that will never be processable: references to entities deleted long ago, test-mode events delivered to a production endpoint, event kinds the system deliberately does not handle. These should be resolved rather than replayed, with a recorded reason. A store where half the records are permanently unprocessable trains everyone to ignore the alert.

Add a triage step rather than a single replay button. For each record, the choices are replay now, replay after a fix (with a link to the issue), or resolve as not applicable with a reason. That vocabulary keeps the store meaningful and turns draining it into a bounded task rather than an open-ended one.

Rate-limit bulk replay. A thousand parked events replayed at once can overwhelm the same downstream that was failing, and it can trigger a burst of customer-facing side effects — a month of dunning emails arriving in a minute. Replay through the normal consumer with normal concurrency, and suppress notifications for replayed events where the notification’s moment has passed.

Finally, keep a retention policy. Resolved and replayed records can be archived after a period; unresolved ones should never be deleted silently, since each one is a payment outcome that was never applied.

Dead-letter triage Each parked record is replayed now, held for a fix, or resolved as not applicable with a recorded reason. Parked record Replay now transient cause is gone Hold for a fix linked to an issue Resolve not applicable, with a reason
Three outcomes, all recorded — a store with only a replay button fills with records nobody can close.

What Dead-Letter Volume Tells You

Treated as a signal rather than a chore, the dead-letter store is one of the more informative dashboards a billing system has.

A sudden burst from one event kind means a schema or contract change, usually on the provider’s side. Providers version their APIs and occasionally add fields or change shapes within a version, and a parser that was strict about unknown fields will park everything of that kind at once. The volume and the uniformity of the failure message make the cause obvious within minutes.

A steady trickle from many kinds, all referencing missing entities, usually means a data-lifecycle problem rather than a parsing one: something is being deleted or archived while events about it are still arriving. The fix is upstream, in whatever removes the entity, not in the consumer.

Records concentrated on a handful of accounts point at data specific to those accounts — a null where the code assumes a value, an unusual currency, a subscription in a state the handler does not expect. These are the most valuable parked events, because each one is a real bug that only a small population triggers and that no synthetic test would have found.

A store that is always empty is worth a moment’s suspicion. Either the pipeline is genuinely healthy, or failures are being swallowed somewhere before they reach it — a broad exception handler that logs and returns success is the usual culprit, and it produces exactly the invisible loss the dead-letter path exists to prevent.

Track the age of the oldest unresolved record as a first-class metric alongside the arrival rate. Arrivals tell you about incoming problems; age tells you whether anyone is dealing with them, and in billing an unapplied payment outcome sitting for a week is a customer-affecting condition regardless of how few there are.

Verification & Testing

Test the classification boundary directly: a database timeout must be retryable and consume the budget, while an unknown event kind must be parked immediately with zero retries. Assert the attempt counts, since the difference between the two paths is exactly what keeps alerting timely.

Test that replay uses the live handler and is idempotent: park an event, replay it twice, and assert one effect. Then park an event whose effect was already applied before the failure — a common case when the failure happened after a partial write — and assert replay produces no duplicate.

Test the ordering property with a pair of events for the same subscription parked out of order, and assert that replaying them by receipt time produces the correct terminal state. Also assert that the pipeline continued processing other subscriptions while both were parked, which is the whole reason the mechanism exists.

Gotchas & Production Pitfalls

  • Returning 200 and swallowing the error. The provider stops retrying, the event is lost, and the endpoint looks perfectly healthy.
  • Retrying permanent failures. Delays the alert without any chance of success, and fills logs that hide the real signal.
  • A separate replay code path. Diverges from the live handler and reintroduces the bug the replay was meant to resolve.
  • No triage vocabulary. The store fills with unprocessable records, the alert becomes noise, and real events sit unreplayed.
  • Bulk replay at full speed. Overwhelms the downstream and can emit a burst of stale customer notifications.

Frequently Asked Questions

Should the endpoint return 200 or 500 when parking an event? 200, once the event is durably stored in the dead-letter table. The provider’s job is delivery, and you have taken responsibility for the event; returning 500 invites redelivery you no longer need and can get the endpoint disabled.

How does this interact with provider-side retries? They cover transport failures. The dead-letter store covers processing failures after successful delivery. Both are needed, and neither substitutes for the other — see webhook retry and timeout strategies for the delivery side.

Can events be replayed from the provider instead? Sometimes, within their retention window, and it is a reasonable fallback. It is not a substitute, because the window is finite and because the provider cannot tell you which events your handler failed on.

What if replay changes state that has since moved on? The handler’s own idempotency and ordering guards should make a stale event a no-op. If they do not, the parked event has surfaced a genuine gap in the consumer rather than a replay problem.