Outbox Pattern & Event Publishing
The hardest reliability bug in billing is the dual write: your service updates a subscription in Postgres and then publishes an invoice.paid event to a broker, and the two operations are not atomic. A crash between them either loses the event (state advanced, nobody downstream knows) or, with naive retries, publishes it twice against a committed-or-not state. The transactional outbox closes that gap by making the event publish part of the same transaction as the state change. This page sits under Webhook Processing & Backend State Management and shows the table, the relay worker, and the delivery semantics that turn at-least-once into effectively-once. It pairs naturally with Idempotency & Event Deduplication on the consumer side.
Why does this matter so much in billing specifically? The events a subscription engine emits are the triggers for money movement and irreversible side effects: a dropped invoice.paid means revenue recognition never fires and a customer keeps a service they have already paid for is never provisioned; a duplicated payment.failed can dunning-email a customer twice or, worse, cancel a subscription that actually succeeded on the retry. Unlike a search-index update, you cannot shrug off a lost billing event as eventually-consistent noise — each one maps to a ledger entry, a tax calculation, or a state machine transition that auditors will later reconcile against the payment processor’s own record. The outbox is attractive precisely because it gives you a per-event audit trail: every row that ever existed in event_outbox corresponds to a committed domain fact, so “did we emit this?” is answerable from the same database that holds the truth, without cross-referencing broker logs that may have already rolled off retention.
It helps to be precise about what the outbox does and does not buy you. It does not make delivery exactly-once — that is provably impossible over an unreliable network without a coordinated commit protocol the broker rarely supports. What it buys is the elimination of the lost-event failure mode entirely, converting an unbounded correctness problem (some committed changes silently never publish) into a bounded duplication problem (every committed change publishes one or more times) that a downstream dedup key resolves deterministically. That trade is almost always the right one, because duplicates are cheap to absorb and lost financial events are expensive to detect and reconstruct.
Prerequisites
The outbox pattern is a small set of collaborating parts: a transactional database, an outbox table written in the same transaction as domain state, a broker to publish to, a relay to drain it, and idempotent consumers to collapse duplicates. The stack shows the pieces before the checklist.
One constraint is worth calling out before you commit to the pattern: the outbox requires that your domain writes and your event writes live in the same transactional database. If your subscription state is in Postgres but you were hoping to publish events sourced from a change in Redis or an external processor’s API, there is no shared transaction to enroll the outbox row in, and the pattern degrades back into a dual write. In practice this is rarely a problem for a billing engine because the invoices, subscriptions, and payment_attempts tables that generate interesting events already sit in one relational store. Where teams get into trouble is sharding: once invoice_id and subscription_id live on different physical Postgres instances, a single business transaction can no longer write both the domain row and one outbox row atomically, and you need either an outbox per shard (with a relay per shard) or a redesign so the aggregate that owns the event also owns the outbox.
A second prerequisite that is easy to underweight is a broker whose publish call returns a durable acknowledgment you can actually trust. The whole mark-sent step hinges on broker.publish() returning success only after the message is replicated and fsynced, not merely accepted into a producer buffer. With Kafka that means acks=all and a sane min.insync.replicas; with SQS it means treating the SendMessage 200 as authoritative; with a fire-and-forget UDP-style path it means the outbox guarantee is a lie, because the relay will mark rows sent that the broker later drops. Verify this property before you rely on it — it is the seam where at-least-once quietly becomes at-most-once.
Architecture & Data Flow
The pattern has four moving parts. A business transaction writes the domain row and an outbox row atomically. A relay worker polls pending outbox rows, publishes each to the broker, and marks it sent only after acknowledgment. Because the write is atomic and the publish is retried until acknowledged, every committed change is delivered at least once; an idempotent consumer collapses any duplicate into effectively-once.
Inputs are domain commands (charge a card, close an invoice). Processing is the atomic write plus the relay loop. Outputs are durable broker messages that downstream services and a database sync layer consume to keep their own state aligned.
Where the outbox row is written matters
There is a subtle design choice hidden in “write the domain row and an outbox row atomically”: whether the application code constructs the event payload, or a database trigger does. Building the payload in application code keeps the event schema in the same place as the business logic that knows what changed — the handler that flips invoices.status to paid already holds the amount_cents, the subscription_id, and the idempotency_key, so it serializes them into the payload directly. The trigger approach instead fires on the UPDATE and assembles the payload from the changed row, which guarantees no code path can forget to emit but couples your event schema to your table schema and hides emission from anyone reading the service code. For billing, application-side construction usually wins, because events like invoice.paid carry derived fields (proration deltas, tax breakdowns) that are not columns on any single row and would be awkward to reconstruct inside PL/pgSQL.
Ordering is a per-aggregate property, not a global one
The data flow diagram shows a single stream, but the honest mental model is one ordered stream per aggregate_id. Within one subscription_id, the sequence subscription.created → invoice.paid → subscription.canceled must reach the consumer in that order, because applying a cancellation before the creation exists is nonsense. Across different subscriptions there is no such requirement, and pretending there is would force a global serialization that caps your throughput at a single partition. The relay preserves per-aggregate order by publishing with the aggregate as the partition key and by draining a given key’s pending rows in created_at order before moving on; the broker then keeps same-key messages on one partition, and a single consumer instance processes that partition sequentially. Get this wrong and you will see a class of bug where a consumer’s state briefly reflects a later event than the one it is currently applying.
Implementation Walkthrough
The four steps define the table, write state and event atomically, run the relay loop, and deduplicate at the consumer. The crux is the atomic write in step two — after that, the relay’s job is simply to keep retrying the publish until the broker acknowledges. The state lifecycle of an outbox row shows the whole journey.
1. Define the outbox table
Keep the row self-describing: a stable event_id for dedup, the aggregate it belongs to (for partition/ordering), the serialized payload, and a status lifecycle. Index the relay’s scan path.
A few column choices repay the thought. The event_id is declared UNIQUE deliberately — if a caller ever retries the business transaction with a deterministic event_id derived from, say, the invoice_id plus a version, the unique constraint turns a would-be second outbox row into a clean conflict instead of a duplicate emission at the source. The attempts counter is what lets the relay distinguish a transient broker hiccup from a genuinely poison payload; without it, a single un-serializable row would be retried forever and starve every row behind it. Storing payload as JSONB rather than TEXT is a small but real win: it lets an operator query the backlog by field (payload->>'subscription_id') during an incident without deserializing every row in application code, which matters when you are trying to answer “which customers are affected by this stuck relay?” at 3am. Resist the temptation to add a wide set of denormalized columns to the outbox — every extra column is one more thing the atomic write must populate correctly, and the payload already carries the full event.
The partial index deserves emphasis because it is the single most important performance decision on this table. WHERE status = 'pending' means the index only contains rows the relay still cares about; once a row flips to sent, it drops out of the index entirely. On a healthy system where the backlog is a few hundred rows, that index stays tiny even as the table itself accumulates millions of sent rows awaiting the retention job. Without the partial predicate, the index would grow with the whole table and every relay scan would pay to skip past long-delivered rows.
CREATE TABLE event_outbox (
outbox_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
event_id UUID NOT NULL UNIQUE, -- stable id consumers dedup on
aggregate_id UUID NOT NULL, -- e.g. subscription_id, for ordering
event_type TEXT NOT NULL, -- 'invoice.paid', 'payment.failed'
payload JSONB NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending | sent | failed
attempts INT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
sent_at TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending
ON event_outbox (created_at)
WHERE status = 'pending';2. Write domain state and the outbox row atomically
This single transaction is the whole point: if it commits, the event will be delivered; if it rolls back, neither the state change nor the event exists.
The failure you are engineering against is not the happy path but the crash between the two statements. Because both the UPDATE invoices and the INSERT INTO event_outbox are enrolled in one transaction, there is no interleaving where the invoice is marked paid but the outbox row is missing — the database’s atomicity guarantee is doing the work that a manual “update then publish” sequence could never do. Note that this means the outbox INSERT must be part of the same connection and transaction as the domain write; if your ORM opens a fresh session for the event, or an after_commit hook fires the insert on a separate connection, you have quietly reintroduced the dual write you were trying to eliminate. When reviewing outbox code the first thing to check is that a single BEGIN/COMMIT brackets both writes.
There is also a latency cost to be honest about. Adding the outbox INSERT to every mutating transaction lengthens it slightly and puts more write pressure on the table and its index. For a billing engine this is almost always negligible against the value delivered — a single JSONB insert is cheap next to the invoice math that preceded it — but under very high write concurrency the pending partial index can become a hot spot for index-page contention. If you measure that, the usual remedy is not to abandon the outbox but to reduce contention with more granular locking or to move the outbox onto its own tablespace; it is rarely the constraint before the relay is.
BEGIN;
UPDATE invoices
SET status = 'paid', paid_at = now()
WHERE invoice_id = $1;
INSERT INTO event_outbox (event_id, aggregate_id, event_type, payload)
VALUES (
$2, -- event_id (deterministic where possible)
$3, -- subscription_id as aggregate_id
'invoice.paid',
jsonb_build_object('invoice_id', $1, 'amount_cents', $4)
);
COMMIT; -- ✅ state and event are now inseparable3. Run the relay loop
Poll pending rows with FOR UPDATE SKIP LOCKED so multiple relay instances never grab the same row. Publish, then mark sent only on broker acknowledgment. Order by created_at and, where ordering matters, drain per aggregate_id.
SKIP LOCKED is the quiet hero of this loop. Without it, running two relay instances for availability would mean the second instance blocks on the rows the first has locked, serializing your throughput and defeating the point of scaling out. With it, each instance’s SELECT ... FOR UPDATE SKIP LOCKED grabs a disjoint batch of currently-unlocked rows and simply steps over anything a sibling already holds, so N relays partition the backlog among themselves without any external coordination, leader election, or advisory lock. The rows stay locked only for the duration of the publish-and-mark transaction, so a relay that dies mid-batch releases its locks on connection teardown and its rows become eligible again on the next tick.
The batch size and poll interval are the two knobs that trade latency against database load. A LIMIT 100 with a 200 ms tick publishes up to 500 events per second per relay while touching the pending index only five times a second; if your latency budget wants events on the broker within tens of milliseconds, shrink the interval and accept more empty polls against the (cheap) partial index. Watch out for one interaction: if the publish call is slow and your batch is large, the whole batch is held in a single database transaction for the duration, extending lock hold time and delaying the commit that frees those rows. Publishing within the transaction is simplest to reason about but couples broker latency to lock duration; a common refinement is to publish outside the row lock and mark sent in a short follow-up transaction, at the cost of a slightly wider duplicate window if the relay crashes between the two.
One more subtlety hides in the except BrokerError branch. Incrementing attempts on failure is correct, but if the failure is a timeout where the broker actually did accept the message, you will republish on the next tick — which is fine, because the consumer dedups, but it means attempts > 0 does not imply the event was never delivered. Treat the counter as “how many times we tried”, not “how many duplicates exist downstream”. The only place a duplicate is authoritatively resolved is the consumer’s idempotency store.
def relay_tick(db, broker) -> None:
with db.transaction() as txn:
rows = txn.fetch("""
SELECT outbox_id, event_id, aggregate_id, event_type, payload
FROM event_outbox
WHERE status = 'pending'
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED
""")
for row in rows:
try:
broker.publish(
topic=row["event_type"],
key=row["aggregate_id"], # partition by aggregate -> per-key order
value=row["payload"],
headers={"event_id": row["event_id"]},
)
txn.execute(
"UPDATE event_outbox SET status='sent', sent_at=now() "
"WHERE outbox_id=$1", row["outbox_id"]) # ✅ delivered
except BrokerError:
txn.execute(
"UPDATE event_outbox SET attempts = attempts + 1 "
"WHERE outbox_id=$1", row["outbox_id"]) # ⚠️ retry next tick4. Deduplicate on the consumer
At-least-once means duplicates are guaranteed, not hypothetical. The consumer records each event_id in an idempotency store inside the same transaction as its effect, achieving effectively-once.
The structural trick here mirrors the producer side: the dedup insert and the effect commit together, so there is no window where the effect is applied but the event_id is not yet recorded. If you recorded the event_id in a separate transaction — say, a Redis SETNX before applying the Postgres effect — a crash between the two would leave the event marked consumed while its effect never landed, and the redelivery that should have fixed it would be skipped as a duplicate. Keeping both in one transaction against the same store closes that gap. The ON CONFLICT DO NOTHING RETURNING idiom is the whole mechanism: on first delivery it inserts and returns the id, so the effect runs; on any redelivery the insert conflicts, returns nothing, and the handler exits before touching state.
Choosing what goes in the consumed_events table is a retention question in disguise. The row only needs to live as long as duplicates can plausibly arrive, which is bounded by the broker’s retention plus the outbox’s max-attempts window — typically hours to a few days, not forever. A consumed_at timestamp plus a periodic delete of rows older than your redelivery horizon keeps this table from becoming its own unbounded growth problem, the same discipline the outbox itself requires. For high-volume events you can key the dedup table by (event_type, aggregate_id, event_id) and partition it, but for most billing workloads a single event_id primary key with a nightly prune is plenty.
It is worth naming the alternative that does not need a dedup table: a naturally idempotent effect. If applying invoice.paid is written as UPDATE invoices SET status='paid' WHERE invoice_id=$1 AND status<>'paid', replaying it is a no-op regardless of how many times it arrives, and you may not need consumed_events at all for that event. The dedup store earns its keep when the effect is not idempotent on its own — incrementing a usage counter, appending a ledger line, or sending an email — where a second application would double-count or double-send.
def consume(event_id: str, payload: dict, db) -> None:
with db.transaction() as txn:
if not txn.execute(
"INSERT INTO consumed_events (event_id) VALUES ($1) "
"ON CONFLICT DO NOTHING RETURNING event_id", event_id):
return # ✅ already processed, skip
apply_effect(payload, txn) # state change + dedup commit togetherEdge Cases & Failure Modes
The outbox failures divide into delivery duplication, backlog growth, and ordering. Duplication is expected and absorbed by idempotent consumers; backlog and poison rows need retention and a max-attempts escape; ordering needs per-aggregate partitioning. The map groups them.
| Failure scenario | Symptom | Mitigation |
|---|---|---|
| Relay crashes after publish, before mark-sent | Event republished next tick | Idempotent consumer collapses the duplicate |
| Two relay instances race | Same row published twice | FOR UPDATE SKIP LOCKED serializes row ownership |
| Outbox grows unbounded | Table bloat, slow scans | Archive/delete sent rows on a retention job; partition by month |
| Broker down for minutes | Pending rows pile up | Bounded retry with attempts; alert on pending-age, not row count alone |
| Poison event (always fails) | One row blocks throughput | After max attempts, set status='failed' and route to operators |
| Per-key ordering broken | Consumer sees events out of order | Partition broker by aggregate_id; drain that key serially |
Performance & Scale
The scaling story is a progression: polling with a partial index carries you to thousands of events per second, and only then does change data capture become worth its operational weight. The diagram contrasts the two relay strategies and where the crossover sits.
Polling is the default and it scales further than people expect: a partial index on WHERE status='pending' keeps the relay scan proportional to the backlog, not the table. Batch 100–500 rows per tick and tune the interval (50–500 ms) against latency budget. Delete or archive sent rows aggressively — an outbox is a transient queue, not a log of record. When polling latency or DB load becomes the bottleneck, switch the relay to change data capture (CDC) reading the Postgres WAL via Debezium: it eliminates the poll, captures inserts in near-real-time, and removes the mark-sent write. CDC trades operational complexity (connector, WAL slots, schema registry) for lower latency and load; polling wins on simplicity until you are publishing thousands of events per second.
Testing Strategy
The tests target the failure paths that make or break the pattern: a crash between publish and mark-sent, two racing relays, a broker outage, and a consumer replay. Each proves one property of effectively-once delivery. The panel lists them.
Drive the failure paths deterministically. Kill the relay between publish and the mark-sent update, restart it, and assert the consumer applied the effect exactly once. Run two relay instances against a seeded outbox and assert no event publishes twice. Inject a broker outage and assert pending rows accumulate, then drain cleanly on recovery with attempts reflecting the retries. Replay the same event_id at the consumer twice and assert one effect. Use a mock clock so retention and retry-age assertions need no real waiting.
def test_crash_after_publish_is_effectively_once(relay, broker, consumer):
seed_outbox(event_id="evt_invoice_paid_42")
relay.publish_then_crash() # broker has it, row still 'pending'
relay.restart().run_tick() # republishes the same event_id
consumer.drain(broker)
assert consumer.effects("evt_invoice_paid_42") == 1 # ✅ exactly onceFrequently Asked Questions
How do I guarantee exactly-once processing across retries?
You cannot guarantee exactly-once delivery, but you can get effectively-once. The outbox gives at-least-once (the atomic write plus retried publish), and an idempotent consumer keyed on event_id collapses every duplicate. The combination is indistinguishable from exactly-once at the effect level.
Should I poll the outbox or use CDC?
Start with polling — a partial index plus SKIP LOCKED carries most workloads to thousands of events per second with trivial ops. Move to CDC (Debezium reading the WAL) when poll latency or database load becomes the constraint, accepting the added operational surface.
Does the outbox preserve event order?
Globally, no — created_at ordering is best-effort. Per aggregate, yes: partition the broker by aggregate_id and drain each key serially so a subscription’s events stay in order even when the global stream interleaves.
How big can the outbox grow before it hurts?
Throughput is bounded by the relay, not the table, as long as you keep a partial index on pending rows and archive sent rows. Treat the outbox as a queue: a retention job that deletes or partitions out delivered rows keeps scans flat.
Should the outbox payload carry the full event or just an identifier?
Prefer a lean row — the aggregate_id, the event type, and enough context to publish — and let consumers fetch the full current state on demand, rather than serializing a fat snapshot into every outbox row. A lean row keeps the publish fast and the partial-index scan small under a renewal burst, and it sidesteps a subtle staleness bug: a fat payload captures the state at write time, but a consumer usually wants the state at read time, and the two can differ if several events for one aggregate queue together. Carry the identity, not the snapshot, unless you specifically need the historical point-in-time value preserved in the event itself.
What breaks if the relay publishes an event whose transaction rolled back? Nothing, because it cannot — that is the whole point of writing the outbox row inside the same transaction as the state change. The relay only ever reads rows that committed, so an event is published if and only if the state change that produced it is durable. This is precisely the guarantee a naive “update the row, then call the broker” dual write cannot make, and it is why the outbox row must never be written in a separate transaction from the mutation it describes.