Designing a High-Throughput Metering Event Pipeline

You design this pipeline the day a single Postgres writer stops keeping up with your meter events — usually somewhere past a few thousand events per second, or when one noisy tenant can stall everyone else’s billing. At that point ingestion, aggregation, and rating can no longer share a thread, and “just insert the row” stops being a correctness strategy. This page builds the scaled version of the system described in Usage-Based Billing Implementation: how to ingest events idempotently at volume, buffer them in a log, count them exactly once under replay, and absorb bursts without dropping a single billable event.

Trade-offs

The one architectural choice that dominates throughput is what buffer sits between ingestion and aggregation. The options form a ladder from a direct Postgres insert to a partitioned log, trading operational weight for orders of magnitude more sustained throughput. The map places them on that ladder.

The core architectural decision is what sits between ingestion and aggregation. Each option trades operational weight against the guarantees and throughput you get.

Kafka (or any partitioned log) is the default for genuinely high throughput because partitioning by customer_id gives you per-customer ordering and lets each consumer own counters with zero cross-shard locking. The rest of this page assumes a partitioned log, but the patterns transfer to Redis Streams and SQS.

The number in the throughput column is per writer or per partition, not per cluster, and that distinction is where sizing goes wrong. A direct Postgres insert caps near 1–3k events/s because every event pays for a WAL fsync, an index update on the unique constraint, and connection-pool contention; you do not scale past it by adding read replicas, only by moving the write off the hot path. A partitioned log reaches its numbers by amortizing fsyncs across a batch — a producer configured with linger.ms=20 and a batch size of a few hundred KB will emit far fewer, far larger writes than one flushing per event, and that batching is the single biggest lever on sustained throughput. Measure your real event size first: a 400-byte metering event and a 4 KB event with a fat attribute bag behave completely differently at 200k events/s, because the second saturates network and disk bandwidth long before it saturates message count.

Read the throughput ladder as a decision about blast radius, not just speed. Direct Postgres couples ingestion latency to your billing database’s health, so a slow rating query stalls event intake; a log decouples them so the two failure domains move independently. That decoupling is the actual reason to adopt a log even when your raw event rate would technically fit in Postgres — one noisy tenant replaying a month of backfill no longer competes with live traffic for the same writer, because the backfill lands as consumer lag on that tenant’s partition instead of lock contention on the shared table.

Step-by-Step Implementation

The pipeline is six stages, each cheap and each idempotent: dedupe, append to a partitioned log, fold with an applied-events guard, advance a watermark, publish through an outbox, and apply backpressure by lagging. The diagram shows the stages and the guarantee each adds.

1. Deduplicate at ingestion

The ingestion endpoint must be cheap — a deduplication check and an append, nothing more. Heavy work here caps your throughput. Key on the event’s natural identity; this mirrors the idempotent webhook consumer pattern applied to the metering boundary.

import hashlib

def ingest(event: dict, redis, producer) -> dict:
    idem_key = hashlib.sha256(
        f"{event['customer_id']}:{event['meter']}:{event['event_id']}".encode()
    ).hexdigest()

    # Claim the key; 7-day TTL covers any producer retry window
    if not redis.set(f"idem:{idem_key}", "1", nx=True, ex=604800):
        return {"status": "duplicate"}                       # ✅ collapse retry

    producer.send(
        topic="usage.raw",
        key=event["customer_id"].encode(),                   # ✅ per-customer ordering
        value=event,
    )
    return {"status": "accepted"}                            # 202, fast path

Redis is the fast first line; it is not the source of truth. A flushed cache must not let duplicates through, which is what the next stage’s durable guard handles.

Compose the idempotency key from fields the producer can regenerate identically on retry, and nothing else. Hashing customer_id, meter, and a client-supplied event_id works because a retrying client resends the same event_id; folding in a server receipt timestamp or an auto-generated row id would make every retry look unique and defeat the whole check. The 7-day TTL is not arbitrary — it must exceed the longest window over which any producer, queue, or mobile client with intermittent connectivity might resend the same event. If a device buffers offline usage for a week and flushes on reconnect, a 24-hour TTL would have already evicted the key and you would double-count the reconnect. Size the TTL to your worst-case retry horizon plus a margin, and remember the memory cost: at 200k events/s a 7-day window holds on the order of 10^11 keys, so this Redis instance needs real capacity planning or a shorter TTL backed by the durable guard for anything older.

2. Append to a partitioned durable log

Partitioning by customer_id is the load-bearing decision. All of one customer’s events land on one partition, so a single consumer sees them in order and can hold that customer’s counters locally. Add partitions to scale; rebalancing moves whole customers, never splitting a counter across consumers.

Choose the partition count generously up front, because it is expensive to change later: growing the partition count rehashes customer_id to new partitions and breaks the ordering guarantee for in-flight windows during the transition. A common rule is to provision for the throughput you expect in a year, not today — a few hundred partitions costs little idle and saves a painful migration. The counterweight is the hot-partition problem. Because a single customer_id maps to exactly one partition, one enormous tenant emitting millions of events per hour can pin a partition at 100% while its siblings idle. Sharding key of customer_id gives correctness for free but not load balance. When you have a whale, sub-shard its key — customer_id:bucket where bucket is derived from the meter or a sub-account — and reconcile the counters for that customer at the fold stage, accepting that you have traded a little aggregation complexity for the ability to spread one tenant across several consumers.

3. Fold into windows with an exactly-once guard

Each consumer reads its partitions and folds events into per-window counters. Because the log is at-least-once and crashes cause replay, the increment itself must be idempotent. Gate every fold on a durable per-window record of applied event ids — the increment runs only when the guard row is freshly inserted.

-- Atomic: claim the event for this window, and only then increment.
WITH claim AS (
  INSERT INTO applied_usage_events (customer_id, meter, period_start, event_id)
  VALUES (:customer_id, :meter, :period_start, :event_id)
  ON CONFLICT DO NOTHING
  RETURNING event_id
)
INSERT INTO usage_counters (customer_id, meter, period_start, period_end, total_quantity)
SELECT :customer_id, :meter, :period_start, :period_end, :quantity
FROM claim                                   -- ✗ no row if already applied → no double count
ON CONFLICT (customer_id, meter, period_start)
DO UPDATE SET total_quantity = usage_counters.total_quantity + EXCLUDED.total_quantity;

Counters are commutative sums, so out-of-order arrival within a window is harmless — a + b == b + a. Ordering only matters for the watermark, not the math. The consumer commits its log offset only after the fold transaction commits, so a crash replays from the last durable point and the guard makes the replay a no-op.

The one guard row per event is the cost of this design, and at high throughput it becomes the bottleneck before the counter update does. Two writes per event — the applied_usage_events insert and the usage_counters upsert — double your write amplification, so batch the fold rather than committing per event. A consumer that accumulates a few hundred events in memory, deduplicates them against a single multi-row INSERT ... ON CONFLICT DO NOTHING, sums the survivors per window in application code, and applies one counter upsert per (customer_id, meter, period_start) per batch turns thousands of transactions into tens. The commutativity of the counter is what makes the in-memory pre-aggregation safe: you are only reordering additions that were always going to commute. Keep the batch small enough that a crash re-does a bounded amount of work — a few hundred events, not a million — and remember the batch is only ever a performance optimization, never a correctness one, because the guard still catches any event the batch double-submits after a partial-failure replay.

4. Advance the watermark and route late events

A window closes when the watermark — min(event_time) still in flight, minus a grace allowance — passes its period_end. Events arriving after their window’s watermark are late. Do not drop them and do not retroactively mutate a rated window; route them to a carry-forward adjustment on the next open invoice.

The grace allowance is the tuning knob that trades billing latency against the late-event rate, and there is no universal right value. Set it too tight and every clock-skewed producer or briefly-partitioned consumer generates carry-forward noise you then have to explain to customers on their next invoice; set it too loose and you delay closing every window for everyone to accommodate a handful of stragglers, holding revenue recognition open longer than finance wants. Instrument the actual distribution of arrival lateness — the delta between occurred_at and the time the event was folded — and pick a grace that captures the 99th percentile, not the maximum. A useful default is to watermark on the minimum in-flight event time across all partitions rather than per partition, because a single stalled partition otherwise holds the global watermark back and no window ever closes. Track per-partition progress and treat a partition whose event time stops advancing as an alert, not as a reason to freeze billing for the whole account base.

def handle_event(event, window_for, watermark, carry_forward):
    win = window_for(event["customer_id"], event["occurred_at"])
    if event["occurred_at"] < watermark(win):
        carry_forward(event)          # ⚠️ late: adjust next invoice, never the closed one
    else:
        fold(event, win)              # ✅ on-time: count into the open window

5. Publish the rated result through an outbox

When a window is rated, write the ledger entry and the outbound event in one transaction so a consumer crash can never publish twice or lose the event — the outbox pattern keeps the commit and the publish atomic.

BEGIN;
INSERT INTO ledger_entries (ledger_entry_id, customer_id, invoice_id, amount_cents, kind, window_key)
VALUES (gen_random_uuid(), :customer_id, :invoice_id, :amount_cents, 'usage', :window_key)
ON CONFLICT (window_key) DO NOTHING;
INSERT INTO outbox (event_type, payload, status)
VALUES ('usage.rated', :payload, 'PENDING');
COMMIT;

6. Apply backpressure instead of dropping

When ingestion outruns aggregation, the correct response is to lag, not to shed. The durable log is the shock absorber: ingestion keeps appending while consumers fall behind, and consumer lag becomes your alerting signal. If producers outrun the log itself, return 429 from ingestion so clients back off — never drop a billable event silently.

Verification & Testing

The headline guarantee — full-replay idempotency — anchors the test suite: replay the whole log and the counters must be byte-identical. Around it sit a mid-batch crash test, a mixed-chaos test (duplicates, reorder, late), and a load test that proves backpressure. The panel summarizes before the code.

Drive the pipeline with a deterministic event generator and a controllable clock. Assert that replaying the entire log from offset zero yields byte-identical counters — full-replay idempotency is the headline guarantee. Kill a consumer mid-batch and assert no double counting after it resumes from its last committed offset. Inject duplicates, out-of-order events, and post-watermark late events in one run and assert: duplicates collapse, out-of-order events land in the right window, late events become carry-forward rows. Load-test ingestion to your target rate and assert p99 latency stays flat while consumer lag (not error rate) rises — proving backpressure works.

def test_full_replay_is_deterministic(pipeline):
    events = generate_events(n=100_000, dup_rate=0.1, out_of_order_rate=0.2)
    for e in events:
        pipeline.ingest(e)
    pipeline.consume_all()
    snapshot = pipeline.counter_snapshot()

    pipeline.reset_consumer_offsets()      # simulate full replay
    pipeline.consume_all()
    assert pipeline.counter_snapshot() == snapshot   # ✅ exactly-once under replay

Gotchas & Production Pitfalls

The pitfalls at scale cluster around three decisions: how you partition, when you commit the offset, and what you do under load. Each has a wrong default that works in testing and fails in production. The map groups them.

• Partitioning by event id instead of customer. You lose per-customer ordering and scatter a customer’s counter across consumers, forcing distributed locks. Partition by customer_id.
• Committing the log offset before the fold transaction. A crash in that gap loses the event entirely. Always commit the offset after the DB transaction succeeds, and lean on the applied-events guard to make the inevitable replay safe.
• Treating Redis as the dedup source of truth. A cache flush then admits duplicates. Redis is the fast path; the durable per-window guard (or a unique constraint on raw events) is the real exactly-once boundary.
• Mutating a rated window when a late event arrives. It corrupts an already-invoiced number and breaks the audit trail. Route late events to carry-forward only.
• Dropping events under load. Shedding billable events is silent revenue loss. Absorb bursts in the log and signal with 429 and consumer-lag alerts instead.
• Unbounded applied_usage_events growth. It grows with every event forever. Partition it by period_start and drop partitions once their windows are rated and beyond any dispute horizon.

Frequently Asked Questions

How large should a metering event be? As small as the rating logic allows. Event size drives network and storage cost linearly at high volume, and a fat attribute bag that nobody rates is pure overhead.

Should events be deduplicated at ingestion or at rating? At ingestion, so downstream stages can assume uniqueness. Deduplicating later means every consumer has to reimplement it, and one of them eventually will not.

What throughput actually needs a partitioned log? Beyond a few thousand events per second sustained, or wherever ingestion must not be coupled to the billing database’s health. Below that, a well-indexed table with a unique constraint is simpler and sufficient.

How long should raw events be retained? Long enough to recompute any invoice a customer might dispute, which usually means at least a full billing year. Partition by time so old data can be dropped without a mass delete.

Related

• Usage-Based Billing Implementation
• Implementing Metered Billing with Stripe vs Custom
• Building Idempotent Webhook Handlers in Node.js
• Using the Outbox Pattern for Reliable Billing Events
• Resolving Out-of-Order Webhook Delivery Issues
• Subscription Billing Architecture & Pricing Models