Usage-Based Billing Implementation

Usage-based billing turns a stream of raw telemetry — API calls, gigabytes transferred, seats active, tokens generated — into a single correct number on an invoice. The hard part is not the pricing formula; it is the pipeline underneath it. Events arrive duplicated, out of order, and sometimes hours late, yet the invoice you charge must be exactly right and reproducible during an audit. This guide is part of Subscription Billing Architecture & Pricing Models, and it focuses on the three subsystems that decide whether your metering is trustworthy: idempotent ingestion, windowed aggregation, and late-event reconciliation. If you are weighing a managed meter against your own, start with implementing metered billing with Stripe vs custom; if you need the pipeline to survive real traffic, see designing a high-throughput metering event pipeline.

Prerequisites

A metering pipeline earns its trust one guarantee at a time, and each guarantee has a prerequisite. Exactly-once counting needs an idempotency store and a database unique constraint; correct windowing needs a per-subscription billing calendar; late-event handling needs an explicit watermark policy. The stack shows the dependencies.

• An append-only usage_events table or topic with a per-event natural key
• An idempotency-key store (Redis with SET NX or a Postgres unique constraint)
• A durable queue or log (Kafka, SQS, Redis Streams) buffering raw events before aggregation
• A billing-cycle calendar per subscription (current_period_start / current_period_end in UTC)
• An outbox table written in the same transaction as ledger postings
• A watermark/grace-window policy for events that arrive after their window opens

The order of these prerequisites is not cosmetic. If you build aggregation before you have a stable idempotency store, you will spend the first month chasing phantom drift: counters that read 3 percent high on Mondays because a weekend retry storm replayed a batch, and no way to prove after the fact which events were double-folded. The append-only usage_events table is what lets you answer that question retroactively — it is the source of truth you reconcile against, so it must exist before any counter does. Treat usage_counters as a derived cache that you can always rebuild from the raw log, never as an authority in its own right.

One prerequisite people skip is a written definition of the natural key. “The client sends an event_id” is not a specification until you decide what happens when two clients, or the same client after a reinstall, generate the same UUID. Prefer a composite natural identity — customer_id plus meter plus a client-side monotonic counter, or a content hash of the metered action — so that identity survives a client that resets its RNG seed. If you cannot control the collector, at minimum record the raw client event_id and the received_at separately, so a collision shows up as two rows with the same event_id but different arrival times rather than a silently swallowed charge.

Finally, size the idempotency store against your worst retry budget, not your average. A seven-day Redis TTL sounds generous until a downstream queue backs up over a long weekend and replays events nine days later; the key has expired, the fold happens twice, and only the usage_events unique constraint saves you. That is precisely why the constraint is a separate prerequisite from the cache: the two protect different time horizons, and a mature pipeline needs both.

Architecture & Data Flow

The pipeline separates four concerns: collection (untrusted, high-volume, may duplicate), ingestion (deduplicate and persist the raw event), aggregation (fold events into per-window counters), and rating (turn a closed counter into money on the ledger). Each stage hands the next a stronger guarantee. Collection promises nothing. Ingestion promises every distinct event is stored exactly once. Aggregation promises a counter that only counts each event once, even across replays. Rating promises one ledger entry per window, written atomically with its outbound event.

Inputs are raw events carrying customer_id, a meter name, a quantity, an event-time occurred_at, and a client-generated event_id. Processing deduplicates on event_id, buffers, then aggregates per (customer_id, meter, window). Outputs are one ledger entry per closed window plus a usage.rated event on the outbox for downstream consumers.

Why the four stages stay physically separate

The strongest reason to keep collection, ingestion, aggregation, and rating as distinct deployments is that they fail independently and recover at different speeds. Collection is the layer you least control — it runs inside customer SDKs, edge Workers, or third-party gateways, and it will occasionally emit garbage. Ingestion is the boundary where you refuse to trust that garbage: it validates the shape, clamps absurd timestamps, and dedupes, but it must stay fast enough to answer in single-digit milliseconds or the collector’s own retry logic will amplify your load. Aggregation is allowed to be slow and can lag minutes behind without any customer noticing, because nothing about a mid-cycle counter is user-visible. Rating is the layer that touches money, so it runs least often and under the strictest transactional discipline. Fusing any two of these — say, rating inline with ingestion — couples the money path to the hot path and turns a collector spike into a billing incident.

The data contract between stages is deliberately narrow. Ingestion hands aggregation nothing but validated raw events keyed by customer_id; aggregation hands rating nothing but sealed, immutable window totals keyed by window_key. Neither downstream stage can reach back and mutate an upstream artifact. That one-directional flow is what makes the whole pipeline replayable: you can throw away every usage_counter and every un-finalized ledger entry, replay usage_events from offset zero, and land on byte-identical totals. If any stage wrote back into its predecessor’s store, that guarantee would evaporate and audits would become archaeology.

Choosing the meter’s unit of measure

Before the first event flows, pin down what a single quantity means and store it in minor, indivisible units. A meter that counts “API calls” is trivial, but “gigabytes transferred” is a trap: decide whether you bill on base-10 GB or base-2 GiB, whether you accumulate bytes and divide at rating time, or accumulate pre-rounded gigabytes per event. The former is correct — round once, at the window boundary, on the summed total — because rounding each event independently leaks fractions of a cent thousands of times per cycle and produces an invoice that no customer can reconstruct from their own logs. Store quantity as an integer count of the smallest meaningful unit (bytes, tokens, whole seconds) and defer every division and rounding decision to the rating stage, where it happens exactly once against the sealed window total.

Implementation Walkthrough

The five steps hand progressively stronger guarantees down the pipeline: ingestion dedupes, buffering decouples, aggregation folds exactly once, rating posts atomically, and reconciliation proves the invoice before it finalizes. Each stage assumes the guarantee the previous one established.

1. Ingest each event behind an idempotency key

The collector cannot be trusted to send each event once. Retries, at-least-once queues, and client bugs all duplicate. Derive a deterministic key from the natural identity of the event — never from a server timestamp, which changes on replay.

import hashlib
from datetime import timedelta

def ingest_usage_event(event: dict, redis, db) -> dict:
    # Identity is the client's event_id, not when we received it
    idem_key = hashlib.sha256(
        f"usage:{event['customer_id']}:{event['meter']}:{event['event_id']}".encode()
    ).hexdigest()

    # SET NX: claim the key, 7-day window covers any retry storm
    claimed = redis.set(f"idem:{idem_key}", "1", nx=True, ex=int(timedelta(days=7).total_seconds()))
    if not claimed:
        return {"status": "duplicate", "event_id": event["event_id"]}  # ✅ safe replay

    db.execute(
        """INSERT INTO usage_events
             (event_id, customer_id, meter, quantity, occurred_at, received_at)
           VALUES (%(event_id)s, %(customer_id)s, %(meter)s, %(quantity)s,
                   %(occurred_at)s, now())
           ON CONFLICT (event_id) DO NOTHING""",
        event,
    )
    return {"status": "accepted"}

The ON CONFLICT (event_id) DO NOTHING is the durable backstop: even if Redis is flushed, the unique constraint guarantees the raw event lands once. This is the same guarantee an idempotent webhook consumer pattern gives inbound webhooks — applied to the metering boundary instead.

There is a subtle ordering hazard in the two-store approach worth calling out. The Redis SET NX claims the key before the Postgres insert commits, so if the process crashes between the two, the key is claimed but no row exists — the event is now silently lost, because a retry will see the claimed key and return duplicate. The fix is to make Postgres the authority and Redis the fast-path optimization: on a Redis claim, still attempt the insert, and treat only a successful ON CONFLICT no-op as proof of a true duplicate. Alternatively, set the Redis key with a short TTL and only extend it after the row commits, so a crash leaves the key expiring on its own within seconds rather than blocking the event for the full seven-day window. For low-volume meters you can drop Redis entirely and let the unique constraint carry the whole load; the cache exists only to keep a duplicate from touching the disk at high fan-in.

Note also that the idempotency key deliberately folds in meter alongside event_id. A single client action sometimes emits several distinct metered facts — one request might increment api_calls, bytes_egress, and compute_ms at once — and if those share an event_id, keying on event_id alone would collapse three legitimate counts into one. Scoping the key to (customer_id, meter, event_id) keeps each meter’s stream independently idempotent while still deduplicating retries within a meter.

2. Buffer raw events in a durable log

Synchronous aggregation couples ingestion latency to write contention on counter rows. Put a durable queue between them. The ingestion handler returns 202 Accepted the moment the event is persisted; an aggregation consumer drains the log at its own pace and survives a downstream outage by simply lagging.

def publish_for_aggregation(event: dict, producer) -> None:
    # Partition by customer so all of a customer's events land on one consumer,
    # which lets each consumer keep counters in memory with no cross-shard locking
    producer.send(
        topic="usage.raw",
        key=event["customer_id"].encode(),  # ✅ ordering per customer
        value=event,
    )

3. Aggregate into windows with a watermark

A window is a (customer_id, meter, period_start, period_end) bucket aligned to the subscription’s billing cycle. The aggregator upserts a counter per window. The watermark is the policy that decides when a window is final: an event whose occurred_at falls in window W but arrives after W’s watermark has passed is late.

The alignment to the subscription’s own calendar, rather than a global wall clock, is what makes multi-tenant metering tractable. Two customers on monthly plans that started on the 3rd and the 19th have windows that open and close on different days, so a single global “close all windows at midnight UTC on the 1st” job would rate half your base against the wrong period boundaries. Derive period_start and period_end from each subscription’s current_period_start, stored in UTC, and compute the bucket for an event by asking which of that subscription’s periods contains its occurred_at. Keep the boundaries half-open — period_start <= occurred_at < period_end — so an event landing exactly on the boundary belongs to precisely one window and can never be counted in both the closing and the opening period.

-- Counter is keyed by the window; UPSERT folds each event in once.
-- The aggregator only writes events whose event_id has not been folded yet,
-- enforced by the per-window applied-events guard below.
INSERT INTO usage_counters (customer_id, meter, period_start, period_end, total_quantity)
VALUES (:customer_id, :meter, :period_start, :period_end, :quantity)
ON CONFLICT (customer_id, meter, period_start)
DO UPDATE SET total_quantity = usage_counters.total_quantity + EXCLUDED.total_quantity,
              updated_at = now();

To keep counting exactly once even if the consumer reprocesses the log after a crash, gate each fold on a per-window record of applied event ids:

-- Returns 0 rows if this event was already folded into this window → skip the increment.
INSERT INTO applied_usage_events (customer_id, meter, period_start, event_id)
VALUES (:customer_id, :meter, :period_start, :event_id)
ON CONFLICT DO NOTHING
RETURNING event_id;

4. Rate the closed window into the ledger atomically

When a window’s watermark passes, rate it. The counter, the pricing tier, and the resulting amount become one ledger entry. Write the ledger row and an outbox row in the same transaction so the “usage was rated” event can never be lost or double-published — the outbox pattern makes the commit and the publish atomic.

Rating is also where tiered and graduated pricing get resolved, and the arithmetic must run against the sealed total, never incrementally as events arrive. Consider a graduated meter where the first 1,000,000 API calls cost 50 minor units per thousand and the next tier costs 30. If you priced each event as it landed, an out-of-order or late event could push the running total across a tier boundary retroactively, and every already-rated event would now be mispriced. Because rating waits for the window to seal, it sees the final total_quantity once, walks the tier schedule top to bottom, and emits a single amount_cents that any auditor can reproduce by hand from the counter and the published rate card. The window_key — a deterministic hash of (customer_id, meter, period_start) — is the idempotency handle for the money path: the ON CONFLICT (window_key) DO NOTHING guarantees that even if the rating job runs twice, the customer is charged for the window exactly once.

Keep the ledger entry immutable once written. If a correction is needed — a late event that must still be billed, a pricing bug discovered after finalization — post a compensating entry with its own ledger_entry_id rather than mutating the original. An append-only ledger means the sum of entries is always the truth, restatements are visible in the history, and you never have to explain to a finance team why an amount changed with no trace.

BEGIN;
INSERT INTO ledger_entries (ledger_entry_id, customer_id, invoice_id, amount_cents, kind, window_key)
VALUES (gen_random_uuid(), :customer_id, :invoice_id, :amount_cents, 'usage', :window_key)
ON CONFLICT (window_key) DO NOTHING;            -- ✅ one entry per window

INSERT INTO outbox (event_type, payload, status)
VALUES ('usage.rated', :payload, 'PENDING');
COMMIT;

5. Reconcile before the invoice finalizes

Before finalizing, re-derive the window total directly from usage_events and compare it against the usage_counters value the ledger used. A mismatch means the aggregator dropped or double-counted; block finalization and alert rather than charge a wrong number.

SELECT c.total_quantity AS counter_total,
       COALESCE(SUM(e.quantity), 0) AS raw_total
FROM usage_counters c
LEFT JOIN usage_events e
  ON e.customer_id = c.customer_id
 AND e.meter = c.meter
 AND e.occurred_at >= c.period_start
 AND e.occurred_at <  c.period_end
WHERE c.customer_id = :customer_id AND c.period_start = :period_start
GROUP BY c.total_quantity;

Edge Cases & Failure Modes

Metering failures cluster around three properties: when an event happened versus arrived, how many times it was counted, and how much burst the pipeline can absorb. The map sorts the failure modes onto those axes so each mitigation is obvious.

When a subscription changes tier mid-window, the counter must be split at the change instant so each segment is priced against the correct rate — coordinate this with Proration Logic & Calculations so usage caps and baseline fees stay consistent with the audit trail.

The negative-quantity and correction problem

Metering is not always additive. Refunds, cancelled operations, and dispute credits arrive as events that must subtract from a counter, and they expose failure modes that pure increments hide. If a customer’s compute job is billed at start and cancelled seconds later, the cancellation event carries a negative quantity that may reference a window already sealed and rated. Never let a negative event silently drive a counter below zero — that usually signals a bug or a duplicate correction, not a real credit, and a negative total_quantity will crash rating or produce a negative amount_cents. Clamp the floor at zero, route the anomaly to a review queue, and require corrections to name the original event_id they reverse so the reversal is itself idempotent. A reversal that lands after its window is rated becomes exactly the same carry-forward adjustment a late positive event does, just with the opposite sign.

Zero-usage and the missing-heartbeat case

A window that receives no events at all is a legitimate and easily mishandled state. If your aggregator only creates a counter row on the first event, a customer with zero usage in a period has no usage_counters row, and a rating job that iterates over counter rows will simply skip them — fine for pure usage pricing, but wrong the moment a plan carries a minimum commitment or a base fee that applies regardless of usage. Drive rating from the set of active subscriptions for the period, not from the set of counter rows, and treat a missing counter as a genuine zero. The distinction between “zero usage” and “no data arrived” also matters for alerting: a meter that normally sees thousands of events per hour and suddenly sees none is far more likely a broken collector than a quiet customer, and only a heartbeat expectation per meter lets you tell those two apart before the invoice goes out wrong.

Performance & Scale

The pipeline has two distinct scaling profiles: ingestion is O(1) and embarrassingly parallel, while aggregation is contended and must be partitioned by customer so no two consumers touch the same counter. Keeping the two on separate fleets means a reconciliation backlog never slows collection. The diagram shows the split.

Ingestion is the hot path: keep it to a Redis SET NX plus an append. Both are O(1) and horizontally scalable. The aggregator is the contended path — partition the durable log by customer_id so each consumer owns a disjoint key range and never contends on another consumer’s counter rows. Index usage_events on (customer_id, meter, occurred_at) so window reconciliation is a range scan, not a full table scan. Batch counter upserts (e.g. fold 500 events, then one UPDATE) to amortize write amplification. At 100k+ subscriptions, run aggregation as a separate fleet from ingestion so a reconciliation backlog never slows event collection. For the deep version of this — exactly-once counting, backpressure, and dedup at sustained throughput — see designing a high-throughput metering event pipeline.

The raw usage_events table is the fastest-growing object in the system, and its growth is the scaling constraint most teams hit first. At a million events per day, a year of history is over 350 million rows, and the reconciliation range scan that was instant in month one becomes a multi-second query that blocks invoice finalization. Partition usage_events by time — monthly partitions keyed on occurred_at work well — so that both the reconciliation scan and the eventual cold-storage archival touch only the relevant partition. Once a billing period is closed, rated, and reconciled, its partition is immutable and can be moved to cheaper storage or rolled up into a per-window summary; you only need the raw events online for as long as a dispute or audit might reach back, typically the current period plus a rolling retention window your finance team specifies.

Hot-partition skew and the whale customer

Partitioning by customer_id distributes load evenly only when customers are evenly sized, and in usage-based billing they never are. One enterprise account can emit more events than the rest of your base combined, and hashing it to a single consumer creates a hot partition that lags while its neighbors idle. Detect this with per-partition lag metrics rather than an aggregate, and give outsized customers a compound partition key such as (customer_id, meter) or (customer_id, shard) where the shard is derived from the event id. Splitting a whale across shards is safe precisely because counters are commutative sums: each shard maintains a partial total, and the rating stage adds the shard partials for the window before pricing. The cost is that reconciliation must now sum across shards, which is why you only shard the accounts that need it rather than the whole base.

Testing Strategy

The metering pipeline is testable precisely because its correctness is a set of invariants under replay and reordering. The four tests below — replay idempotency, event-time bucketing, late-event carry-forward, and full-log replay — each pin one invariant. The panel summarizes before the code.

Test the pipeline deterministically by controlling the clock and replaying events. Assert that replaying the same event id twice yields one increment. Assert that an out-of-order event lands in the window matching its occurred_at, not its arrival. Inject a late event after the watermark and assert it produces a carry-forward adjustment, not a silent drop or a duplicate. Replay the entire durable log from offset zero and assert the final counters are identical — the system must be idempotent under full replay.

def test_replay_is_idempotent(pipeline, clock):
    event = {"event_id": "evt_1", "customer_id": "cus_42",
             "meter": "api_calls", "quantity": 10, "occurred_at": clock.now()}
    pipeline.ingest(event)
    pipeline.ingest(event)              # duplicate delivery
    pipeline.aggregate_all()
    pipeline.aggregate_all()            # consumer crash + replay
    assert pipeline.counter("cus_42", "api_calls") == 10   # ✅ counted once

The most valuable test in a metering suite is not any single-case assertion but a property-based one: generate a random multiset of events with random occurred_at values, shuffle their arrival order, duplicate a random subset, and assert that the final counters equal the deterministic sum of distinct quantities per window regardless of the permutation. This one test subsumes replay idempotency, out-of-order handling, and commutativity at once, and it routinely surfaces boundary bugs — an event exactly on period_end, a duplicate that straddles a watermark — that hand-written cases miss. Pin the seed on failure so the shrunk counterexample is reproducible, then promote it to a named regression test.

Reconciliation deserves its own adversarial test. Deliberately corrupt a counter — increment total_quantity by one behind the aggregator’s back — and assert that the pre-finalization reconciliation query catches the drift and blocks the invoice rather than charging the inflated number. A reconciliation step that is never tested against a real mismatch tends to rot into a query that always passes, and you discover it was silently returning zero rows only when a genuine discrepancy reaches a customer. Treat the guard as production code that must fail loudly on injected faults.

Frequently Asked Questions

How long should the late-event grace window be? Set it to the largest realistic clock skew plus collector retry budget — commonly 24 to 72 hours. Events inside the window adjust the open counter; events past it become a carry-forward adjustment on the next invoice. The window must be a deliberate policy, not an accident of when your cron happens to run.

How do you guarantee exactly-once counting when the queue is at-least-once? Counting once does not require delivery once. Deduplicate on a natural event_id at ingestion, and gate each aggregation fold on a per-window applied_usage_events record. Re-delivery then either hits the Redis key, the usage_events unique constraint, or the applied-events guard — three layers that all collapse a duplicate to a no-op.

Should I bucket events by when they happened or when they arrived? By event time (occurred_at). Arrival time is a property of your network, not the customer’s usage, and bucketing by it produces wrong totals the moment delivery is delayed. Reserve received_at for observability and late-event detection only.

Do I need Kafka, or is Postgres enough? For tens of thousands of events per day, a Postgres usage_events table polled by a worker is plenty and far simpler to operate. Reach for a partitioned log like Kafka when sustained ingestion exceeds what a single Postgres writer comfortably handles, or when multiple independent consumers need the same event stream.

Where should rounding happen, per event or per window? Per window, once, against the sealed total. Rounding each event independently accumulates a systematic bias — thousands of fractional-cent truncations per cycle that always round the same direction — and produces an amount_cents the customer cannot reproduce from their own logs. Keep quantity as an integer count of the smallest meaningful unit through ingestion and aggregation, and defer every division and rounding decision to the rating stage where it runs exactly once.

How do I bill a customer whose usage event references a subscription that changed plans mid-cycle? Bucket the event by its occurred_at into whichever plan segment was active at that instant. Close the counter at the change boundary and open a fresh one, so the events before the change rate against the old tier and the events after rate against the new one. The event’s arrival time is irrelevant; only when the usage physically happened decides which price applies, which keeps the split consistent with the proration entries on the same invoice.

Can I let the aggregator update the invoice in real time as usage accrues? You can surface a running estimate for dashboards, but never treat a mid-cycle counter as a finalized charge. The counter is still mutable — late events, corrections, and tier splits can all move it before the watermark passes — so anything derived from it before rating is an estimate, not a bill. Draw a hard line at window sealing: below it, everything is advisory; above it, the ledger entry is immutable and authoritative.

What retention do I need for the raw usage_events? Long enough to reconcile and defend any invoice a customer might dispute, which in practice means the current open period plus a rolling window your finance and legal teams specify — often twelve to twenty-four months. Keep recent partitions online for fast reconciliation, roll sealed periods into per-window summaries, and archive the raw events to cold storage rather than deleting them, so an audit can always rebuild a counter from first principles.

Related

• Implementing Metered Billing with Stripe vs Custom
• Designing a High-Throughput Metering Event Pipeline
• Handling Usage Spikes and Bill-Shock Caps
• Proration Logic & Calculations
• Building Idempotent Webhook Handlers in Node.js
• Using the Outbox Pattern for Reliable Billing Events
• Subscription Billing Architecture & Pricing Models
• Handling Late-Arriving Usage Events After Cycle Close