Webhook Processing & Backend State Management

A subscription billing system is, at its core, a distributed state machine driven by events it does not control. Stripe, Paddle, Adyen and your bank all emit asynchronous webhooks — invoice.paid, customer.subscription.updated, charge.dispute.created — and your backend must turn that unreliable, out-of-order, frequently-duplicated stream into a ledger accurate to the cent. The hard part is not parsing the JSON. The hard part is that webhooks are delivered at least once, arrive out of order, can be replayed by an attacker, and may land while another node is mutating the same subscription. Get any of these wrong and you get phantom renewals, double charges, suspended-but-paying customers, and an audit trail that does not reconcile. This guide covers the four subsystems that make webhook processing survivable at scale: keeping state consistent across services through Database Sync & Consistency Patterns, guaranteeing exactly-once effects with Idempotency & Event Deduplication, surviving provider outages with sound Webhook Retry & Timeout Strategies, and publishing your own internal events atomically with the outbox pattern for billing events.

Architecture Overview

Every inbound event traverses the same pipeline before it is allowed to touch money. Signature verification rejects forgeries and replays at the edge. An idempotency gate collapses duplicate deliveries to a single effect. A finite state machine validates that the transition is legal. Only then does the handler write domain state and an outbox row in one transaction; a poller drains the outbox to downstream consumers and the double-entry ledger.

The components and their responsibilities:

The single most important structural decision is to split the pipeline into a thin ingress and a thick worker. The ingress endpoint does the minimum that must happen synchronously — verify the signature, persist the raw event, and return 200 OK — and nothing else. Everything expensive (state transitions, ledger postings, downstream fan-out) happens asynchronously in a worker that reads from the durable event log. This matters because providers treat a slow response as a failure and retry: if your handler does real work inline and takes three seconds because the database is busy, the provider times out at, say, two seconds, marks the delivery failed, and sends it again — so slowness directly manufactures duplicates. A handler that acknowledges in single-digit milliseconds and defers the work converts an unreliable synchronous contract into a reliable asynchronous one, and it means a downstream outage can never back-pressure into the provider’s retry machinery.

The thin-ingress rule has a corollary about the raw body. HMAC verification must run over the exact bytes the provider signed, so the ingress must read the raw request body before any JSON middleware parses and re-serializes it — a re-encoded body will not match the signature even when the content is semantically identical. Capture the raw bytes, verify, hash them for the dedup record, and only then parse. Frameworks that eagerly parse JSON are a common cause of “signatures that mysteriously fail in production but pass in tests,” because the test harness happens to preserve byte order and production does not.

There is one more edge worth designing for up front: what status code to return when your system is unhealthy. If the ingress cannot even persist the raw event — the database is down — you must return a 5xx so the provider retries later, not a 200 that silently drops the event forever. But if the event is persisted and merely awaits async processing, return 200 immediately; the provider’s job is done the moment the bytes are durable on your side. Getting this boundary right is what makes the whole system tolerant of your own outages: a database blip becomes a brief spike in provider retries that drain automatically once you recover, rather than a permanent hole in your event history. The provider’s retry schedule is, in effect, a free durable queue in front of your ingress — but only if you signal failure honestly with the status code.

Core Data Model

The schema separates three concerns that must never be conflated: the raw event log (audit and dedup), the versioned subscription state (concurrency control), and the outbox (decoupling internal events from inbound webhooks). Keeping them in distinct tables is what lets deduplication, state transitions, and publishing each fail and recover independently. The diagram shows the three tables and the guarantee each provides.

Why three tables instead of one status column on the subscription? Because each concern has a different lifecycle and a different failure mode. The event log is write-once and grows forever — it is your audit trail and your dedup index, and it should never be updated after insert. Subscription state is mutable and hotly contended, so it needs versioning and a narrow row. The outbox is transient — rows appear, get published, and are pruned — so it wants its own partial index and retention policy. Fusing them creates lock contention (the audit insert blocks the state update), muddies retention (you cannot prune published events without touching audit history), and destroys the independent-recovery property that makes the pipeline debuggable. Keep the raw payload on the event-log row as well, not just its hash: when a consumer has a bug, the ability to replay the original bytes through the fixed code is the difference between a five-minute recovery and a data-loss incident.

-- Inbound event log: deduplication AND audit trail in one table.
CREATE TABLE webhook_events (
  event_id            UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  provider_event_id   VARCHAR(255) NOT NULL,
  tenant_id           UUID         NOT NULL,
  event_type          VARCHAR(64)  NOT NULL,
  payload_hash        BYTEA        NOT NULL,            -- sha256 of raw body
  status              VARCHAR(16)  NOT NULL DEFAULT 'received',
  received_at         TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at        TIMESTAMPTZ,
  -- the dedup guarantee: one effect per provider event per tenant
  CONSTRAINT uq_provider_event UNIQUE (tenant_id, provider_event_id)
);

-- Versioned subscription state. version drives optimistic concurrency.
CREATE TABLE subscriptions (
  subscription_id     UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  customer_id         UUID         NOT NULL,
  price_id            VARCHAR(64)  NOT NULL,
  state               VARCHAR(24)  NOT NULL,            -- trialing|active|past_due|canceled|unpaid
  current_period_end  TIMESTAMPTZ  NOT NULL,
  version             BIGINT       NOT NULL DEFAULT 0,  -- monotonic, set from event sequence
  updated_at          TIMESTAMPTZ  NOT NULL DEFAULT now()
);

-- Transactional outbox: written in the SAME tx as the subscription mutation.
CREATE TABLE billing_outbox (
  id                  UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id        UUID         NOT NULL,            -- subscription_id
  event_type          VARCHAR(64)  NOT NULL,
  payload             JSONB        NOT NULL,
  status              VARCHAR(16)  NOT NULL DEFAULT 'pending',
  retry_count         INT          NOT NULL DEFAULT 0,
  created_at          TIMESTAMPTZ  NOT NULL DEFAULT now(),
  published_at        TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending ON billing_outbox (created_at)
  WHERE status = 'pending';

The version column is the linchpin of correctness: it is set from the provider’s event sequence (or a Lamport-style counter) so that a late-arriving older event can be detected and discarded rather than clobbering newer state. Money is never stored as a float — ledger amounts live as BIGINT minor units (cents) elsewhere in the model.

Note that the dedup constraint is scoped by tenant_id, not global. Providers guarantee event-ID uniqueness only within an account, and in a multi-tenant system where different tenants connect their own provider accounts, two tenants can legitimately present the same provider_event_id. A global unique index would then reject a second tenant’s genuine event as a “duplicate” — a data-loss bug that only appears once you have enough tenants for the birthday paradox to bite. Scoping the constraint to (tenant_id, provider_event_id) is the correct grain. The payload_hash column earns its place for a different reason: it lets a reconciliation job detect the rare case where a provider reuses an event ID with a different body (a provider bug, but they happen), by flagging any redelivery whose hash disagrees with the stored one instead of silently acking it.

Key Design Patterns

Four patterns carry the webhook pipeline, each guarding a distinct failure at a distinct stage. Idempotency keys stop duplicate effects at ingress, the transactional outbox stops lost or double-published events at the boundary, the subscription FSM stops illegal transitions during apply, and dead-letter routing stops poison messages from stalling the queue. The matrix maps each to its failure.

Idempotency keys

Every effect is gated on a unique key derived from the provider event. The insert is the lock: if two concurrent deliveries race, exactly one wins the unique constraint and the other short-circuits to 200 OK. Use this whenever a provider may deliver the same logical event more than once — which is always.

INSERT INTO webhook_events (provider_event_id, tenant_id, event_type, payload_hash)
VALUES ($1, $2, $3, $4)
ON CONFLICT (tenant_id, provider_event_id) DO NOTHING
RETURNING event_id;   -- zero rows returned ⇒ duplicate, ack and stop

The subtlety most teams miss is what “processed” means for the idempotency record. Inserting the row at receipt dedupes redeliveries, but if the worker then crashes mid-processing, the event is marked seen yet its effects never landed — a silent drop. The robust pattern is a small state machine on the event row itself: received → processing → done, where the transition to done commits in the same transaction as the effect it guards. A redelivery that finds the row in processing past a lease timeout re-claims it; one that finds done acks immediately. This way the idempotency key means “this effect happened exactly once,” not merely “we saw this ID.” The key’s retention must also outlive the provider’s entire retry horizon — if you expire keys after an hour but the provider retries a failed delivery for three days, a day-old redelivery sails past a now-empty dedup table and double-applies.

Transactional outbox

Writing domain state and also calling the message broker over the network in one operation is not atomic — a crash between them loses or duplicates events. The outbox sidesteps this: both writes hit Postgres in one transaction, and a separate poller publishes. Use it whenever a state change must reliably emit an event. See Outbox Pattern & Event Publishing for the full treatment.

with db.transaction():                       # ✅ both writes commit or neither does
    db.execute("UPDATE subscriptions SET state=%s, version=%s WHERE subscription_id=%s",
               new_state, event_version, sub_id)
    db.execute("INSERT INTO billing_outbox (aggregate_id, event_type, payload) "
               "VALUES (%s, %s, %s)", sub_id, "subscription.activated", payload)
# a poller drains billing_outbox → broker, at-least-once, idempotent consumers

The outbox also gives you ordering for free within an aggregate: because rows are inserted in transaction-commit order and the poller drains them in created_at order per aggregate_id, downstream consumers see a subscription’s events in the sequence they actually occurred, even though the inbound webhooks that caused them may have arrived scrambled. Set the poller’s publish to record published_at only after the broker acknowledges, so a crash mid-publish leaves the row pending and it is simply re-sent — at-least-once, which the idempotent consumer collapses to exactly-once. Prune published_at IS NOT NULL rows on a retention schedule (say, keep 30 days for debugging, then archive), because an unbounded outbox eventually makes the partial-index scan expensive. The one rule that cannot bend: never publish inside the request transaction as an “optimization.” The moment the broker call sits in the same transaction as the state write, you are back to the dual-write problem the outbox exists to solve.

Subscription FSM

A finite state machine rejects illegal jumps and enforces monotonic versioning so out-of-order delivery cannot corrupt state. Use it for any lifecycle with forbidden transitions.

ALLOWED = {
    "trialing": {"active", "canceled"},
    "active":   {"past_due", "canceled"},
    "past_due": {"active", "unpaid", "canceled"},
}

def transition(record, target_state, event_version):
    if event_version <= record.version:          # ⚠️ stale/out-of-order event
        return "IGNORED_OUT_OF_ORDER"
    if target_state not in ALLOWED.get(record.state, set()):
        raise ValueError(f"{record.state} -> {target_state} blocked")   # ✗ illegal jump
    record.state, record.version = target_state, event_version
    return "TRANSITION_APPLIED"

Dead-letter routing

Events that fail validation repeatedly must not poison the queue. Route them to a dead-letter queue (DLQ) after N attempts for manual inspection, keeping the main pipeline flowing. Use it for malformed payloads and unrecognised event types.

The DLQ is only useful if it is actionable, which means each dead-lettered message must carry enough context to diagnose it without re-running production: the original raw payload, the exception and stack, the attempt count, and the handler version that failed. A DLQ that stores only “event 12345 failed” forces an engineer to reconstruct the failure from logs during an incident. Distinguish two failure classes when routing: poison messages (a malformed payload or a bug that will fail identically forever) belong in the DLQ immediately, while transient failures (a downstream 503, a lock timeout) belong back on the main queue with backoff. Misclassifying transient failures as poison drains your recovery budget; misclassifying poison as transient blocks the queue behind a message that will never succeed. The safe default is to retry a bounded number of times with backoff and dead-letter only after the budget is exhausted, then alert on any non-empty DLQ so a human decides the fix — replay after a code deploy, or discard as a known-bad duplicate.

Compliance & Regulatory Boundaries

The webhook ingress sits inside PCI-DSS and GDPR scope because payloads carry PII and transaction metadata, so the compliance boundaries must be enforced at the edge — before anything is logged or stored. The map shows the three obligations and where each is discharged in the pipeline.

Webhook payloads routinely carry PII and transaction metadata, so the ingestion layer is squarely in scope for PCI-DSS and GDPR. Never persist raw PANs or CVVs — strip them at the edge before anything is logged or stored. Apply field-level encryption (AES-256-GCM) to customer email and address fields before insertion, with keys managed by a KMS. For GDPR erasure requests that collide with financial retention obligations, pseudonymise the PII while preserving the immutable ledger and event log under a legal-hold flag; the audit trail must remain reconstructable for PCI-DSS and SOC 2. Implement erasure as a forward-only operation — overwrite the PII columns with a tombstone and record that the erasure happened as its own audit event, rather than deleting rows — so the immutability guarantee the ledger depends on is never violated even in service of a privacy request. Because the raw event log may contain PII inside stored payloads, the erasure job must reach into those payloads too, redacting the personal fields while leaving the financial amounts and identifiers that reconciliation needs. This is why storing the payload with PII already field-encrypted pays off: erasure can be as cheap as destroying the per-subject key, cryptographically shredding the data without rewriting history. Where webhooks carry tax-relevant amounts, the VAT/GST figures you persist become part of your regulatory reporting record, so they must be reconcilable against the double-entry ledger posting and never silently mutated.

The signing secret itself is a compliance-relevant asset, and it needs a rotation story before you need it in an incident. Providers let you run two active signing secrets during a rotation window; the verifier should try each configured secret and accept if any matches, so you can roll the secret without a synchronized deploy that risks dropping live events. Store the secret in a secrets manager, not an environment variable baked into an image, and scope read access to the ingress service alone. Log that verification happened and its outcome, but never log the secret or the raw signature — those belong in the same untouchable category as card data. When a secret is suspected compromised, the rotation procedure is: add the new secret as secondary, deploy, promote it to primary, then retire the old one, verifying at each step that the accept-rate on live traffic stays at 100%.

Scalability & Failure Modes

The failure modes shift as you scale: at 10k subscriptions a single primary absorbs the renewal burst, but at 100k the dedup table and outbox poller become bottlenecks, and a slow broker can trigger a retry-storm cascade. The map shows the danger at scale and the mechanism that contains it.

At 10k subscriptions a single Postgres primary with a unique-constrained event log comfortably absorbs the renewal-day burst. At 100k, the failure modes shift. Renewal cycles cluster events at period boundaries, so the dedup table and outbox poller become the bottleneck — partition webhook_events by month and keep the partial index on status = 'pending' so the poller’s working set stays small. Cascade failures are the real danger: if the broker is down and you publish synchronously inside the webhook transaction, every webhook blocks, connection pools exhaust, and the provider’s retries amplify the load (a retry storm). The outbox breaks that cascade by making the broker an asynchronous concern. Wrap every provider API call (used during reconciliation) in a circuit breaker so a provider 5xx does not propagate; trip open after a threshold, serve a pending_verification fallback state, and probe half-open on a timer. Honour Retry-After headers and apply exponential backoff so you never become the cause of the provider’s rate limiting.

Scaling the worker tier horizontally introduces its own hazard: if two poller instances read the same pending outbox rows, you double-publish. The clean fix is SELECT ... FOR UPDATE SKIP LOCKED, which lets each poller claim a disjoint batch of rows without blocking its peers — one poller locks and processes a batch while another skips those locked rows and grabs the next. This turns the outbox into a competing-consumers queue backed by Postgres, scaling to as many pollers as your write throughput allows, with no external broker required for the internal event stream. The same pattern applies to the async worker draining the raw event log. What you must never do is scale by simply running the whole handler on more nodes without row-level claiming, because then concurrency becomes correctness-breaking rather than throughput-enhancing. Measure the poller’s batch size against publish latency: too small and you pay per-round-trip overhead, too large and a single slow publish stalls a big claimed batch behind it.

Operational Runbook

Instrument the pipeline so drift is caught before it reaches a financial statement. Four signals matter most — outbox lag, dead-letter depth, idempotency hit-rate, and reconciliation variance — each with a threshold that turns it from a dashboard number into a page. The panel is the on-call summary.

Instrument the pipeline so drift is caught before it reaches a financial statement. Thread a single correlation ID from the inbound event through every downstream effect — the state transition, the ledger posting, the published internal event — so a trace answers “what did this one webhook cause?” in one query. Without that thread, debugging a mis-posted charge means correlating timestamps across four services by hand. Emit a structured log line at each pipeline stage keyed on that ID and the provider_event_id, and you can reconstruct the full causal chain of any financial movement from the logs alone, which is exactly what an auditor or an incident reviewer asks for.

One metric deserves promotion to a first-class SLO: end-to-end processing lag, measured from the provider’s event timestamp to the moment your ledger reflects it. This single number captures the health of the entire asynchronous pipeline — a rising lag means the worker tier, the outbox poller, or a downstream consumer is falling behind, and it rises before any individual component alarms. Because your correctness argument rests on eventual consistency, the bound on “eventual” is a business decision you should state explicitly: if a customer’s successful payment must be reflected in their account within, say, sixty seconds, then processing lag crossing sixty seconds is an SLO breach worth paging on, not a curiosity. Track it at p50 and p99, because a healthy median hiding a long p99 tail means a subset of subscriptions — often the busiest tenants, whose events queue behind each other — are silently far staler than the average suggests.

A closing word on how to introduce this architecture into an existing system, because most teams meet these patterns while already running a leaky version. Do not attempt a big-bang rewrite of the webhook path; retrofit the pieces in the order that buys the most safety per unit of risk. First add the dedup gate — a unique constraint on (tenant_id, provider_event_id) — because it is a small change that immediately stops the most expensive failure, double charges. Second, move real work off the ingress into an async worker reading the raw event log, which decouples your processing latency from the provider’s timeout and stops slowness from manufacturing duplicates. Third, introduce the outbox for your internal event publication, replacing any dual write. Fourth, add the FSM guard and monotonic versioning so out-of-order delivery stops corrupting state. Each step is independently valuable and independently shippable, and each reduces a distinct class of incident, so the sequence delivers safety continuously rather than all at the end. The team that tries to land all four in one release usually lands none of them; the team that ships them as four boring, well-tested changes ends up with the same architecture and none of the drama.

The deeper lesson underneath every pattern here is that a webhook pipeline is an exercise in making an unreliable, adversarial, out-of-order input stream produce an exact financial record — and exactness under unreliability is only achievable by pushing the guarantees down into the storage layer where they can be enforced atomically, rather than up into application logic where they race. The unique constraint dedupes, the transaction makes the outbox atomic, the version column orders, the ledger balances: every one of these is a database-enforced fact rather than an application-checked hope. That is the through-line worth carrying to every new feature on this surface — when you find yourself about to enforce a money-affecting invariant in application code, ask first whether the database can enforce it for you, because the database does not have races and your application code does.

• Monitoring signals: outbox lag (count of pending rows older than 60s), webhook p99 processing latency, idempotency hit-rate (sudden spikes signal a provider replay), DLQ depth, and circuit-breaker state transitions.
• Alert thresholds: page when outbox lag exceeds 500 rows or 5 minutes; warn when DLQ depth is non-zero; page when reconciliation variance exceeds one cent per 10k subscriptions.
• Reconciliation jobs: a nightly job diffs provider settlement reports against ledger_lines, and a 15-minute job re-syncs any subscription whose last_synced_at is older than two hours. Both write provider-truth overrides to an immutable audit table.
• Stuck-state sweep: a periodic job clears webhook_events left in processing past the maximum retry window, which otherwise leak idempotency keys.

Because the raw event log is durable and complete, your most powerful operational tool is deterministic replay. Build a small, access-controlled command that takes an event ID or a time range and re-drives those events through the current handler code; because every effect is idempotent, replaying already-applied events is a safe no-op, so the tool can be pointed at “everything since the bad deploy” without fear of double-charging. This single capability collapses a whole category of incidents — a consumer bug that mis-processed a day of events becomes “fix the code, replay the day” rather than a manual data-repair project. Pair it with a provider-backfill command that pulls the authoritative event list from the provider’s API for a window and diffs it against your event log, so a webhook the provider says it sent but you have no record of surfaces as a concrete, fetchable gap rather than a silent hole. Rehearse both in staging on a copy of production traffic, because the worst time to discover your replay tool has a bug is mid-incident.

Frequently Asked Questions

How do we prevent duplicate charges when a provider sends the same webhook multiple times? Gate every effect on a unique index over (tenant_id, provider_event_id) and perform the insert with ON CONFLICT DO NOTHING before any business logic. The first delivery wins the constraint and proceeds; later deliveries return zero rows, so the handler acknowledges with 200 OK and stops. This pushes deduplication into the storage layer where it is atomic.

When should I return a non-2xx status to the provider? Only when you genuinely failed to durably accept the event — for example, the database is unreachable and you could not persist the raw payload. In that case a 5xx tells the provider to retry, and its retry schedule acts as a durable queue in front of you. Once the raw event is stored, return 200 even though async processing is still pending; the delivery contract is satisfied by durable receipt, not by completed work. Never return 5xx because your business logic rejected the event — that just triggers pointless retries of a message that will fail identically.

Should I publish internal events directly from the webhook handler or use an outbox? Use an outbox. Publishing to a broker over the network inside the same operation as your database write is not atomic — a crash between the two loses or duplicates events. Writing the event to an outbox table in the same transaction, then draining it with a poller, gives you at-least-once delivery that pairs cleanly with idempotent consumers for exactly-once effects.

How should the subscription state machine handle out-of-order event delivery? Make transitions monotonic and idempotent. Stamp each subscription with a version derived from the provider event sequence, and discard any event whose version is less than or equal to the current one. Combine that with explicit transition guards so an early canceled cannot be overwritten by a late active.

What is the right architecture for surviving a provider outage? A durable queue (SQS, Kafka, or RabbitMQ) with dead-letter routing, exponential backoff, and a circuit breaker around the provider’s API. The outbox keeps your internal state consistent while the provider recovers, and the reconciliation job repairs any drift once the provider is healthy again.

How do you scale the outbox poller without double-publishing? Claim rows with SELECT ... FOR UPDATE SKIP LOCKED. Each poller instance locks and processes a disjoint batch while others skip the locked rows and grab the next, turning the outbox into a competing-consumers queue backed by Postgres with no external broker required. Record published_at only after the broker acks, so a crash leaves the row pending for safe re-send. This scales to as many pollers as your write throughput supports; what breaks correctness is scaling the handler across nodes without row-level claiming, which lets two workers publish the same row.

What compliance considerations apply to storing webhook payloads? Never store raw PANs or CVVs; strip them at ingestion. Encrypt PII fields at rest, keep an immutable audit log of every state mutation for PCI-DSS and SOC 2, and resolve GDPR erasure-versus-retention conflicts by pseudonymising PII while preserving the financial record under legal hold.

Should the webhook handler do the work synchronously or just enqueue it? Just enqueue it. Verify the signature, persist the raw event, return 200 OK, and let an asynchronous worker do the state transitions and ledger postings. Doing real work inline couples your processing latency to the provider’s timeout: if you are slow, the provider marks the delivery failed and retries, so slowness directly manufactures the duplicates you then have to deduplicate. A thin ingress that acknowledges in milliseconds is both faster and more correct.

How do you test a webhook pipeline without hitting the real provider? Drive it from captured real payloads. Record a corpus of genuine provider events (with PII scrubbed) and replay them through the handler in tests, including deliberately duplicated, out-of-order, and malformed variants. Verify the signature check rejects a tampered body, the idempotency gate collapses duplicates, the FSM ignores a stale-version event, and a poison payload lands in the DLQ. Because the pipeline is event-driven and idempotent, this corpus-replay approach gives deterministic, provider-independent coverage of exactly the failure modes production will throw at it.

Related

• Database Sync & Consistency Patterns
• Payment Provider Migration & Portability
• Idempotency & Event Deduplication
• Webhook Retry & Timeout Strategies
• Outbox Pattern & Event Publishing
• Subscription Billing Architecture & Pricing Models
• Tax, Compliance & Revenue Recognition