Syncing Subscription Status Across Microservices

You hit this problem the moment subscription status lives in more than one place. The billing service marks a subscription past_due from an invoice.payment_failed webhook, but the entitlements service still shows active, so a delinquent customer keeps premium access — or worse, a paying customer is locked out because a canceled event was processed before the invoice.paid that should have preceded it. Syncing subscription status across microservices is fundamentally about turning an unreliable, out-of-order provider event stream into convergent local state without double-applying effects. This is the applied form of the Database Sync & Consistency Patterns that govern the wider pipeline, and it leans on the outbox pattern for billing events plus an idempotent webhook consumer pattern to stay correct under retries.

Trade-offs

The propagation mechanisms trade latency, ops burden, and blast radius. Synchronous fan-out is simplest but couples the webhook transaction to every consumer’s health; outbox-plus-poller decouples them; CDC scales to many consumers at high operational cost. The map ranks the common options.

The propagation mechanism you pick determines latency, operational burden, and how cleanly you can reason about ordering. Real values below assume a Postgres-backed billing service at ~50k active subscriptions.

The pragmatic default is outbox + poller as the primary path with periodic reconciliation as the safety net. CDC is worth its operational weight only once you have many independent consumers.

Why ordering is the hidden cost

The latency column hides the decision that actually bites you in production: ordering. Provider webhooks are ordered per resource on the sending side, but nothing preserves that order across a network of retries, load-balanced webhook receivers, and a broker that shards by partition key. A customer.subscription.updated carrying status=active and the invoice.payment_failed that should precede it can arrive seconds apart in either sequence. Synchronous fan-out inherits whatever order the webhook receiver saw, which is already scrambled. Outbox plus poller lets you stamp a monotonic sequence at capture time and drain in created_at order, so per-subscription ordering survives even when the broker reshuffles. That single property — a durable local sequence number keyed on subscription_id — is the difference between “we replay and converge” and “we page someone at 3am because a canceled customer still has access.” When you evaluate CDC, remember that Debezium preserves order only within a Kafka partition, so you must partition on subscription_id (never on event_type or customer_id) or you reintroduce the exact reordering you paid Kafka to avoid.

Coupling and the cost of a slow consumer

The blast-radius column is really a statement about coupling. With synchronous HTTP fan-out, the webhook handler holds its database transaction open while it waits for the entitlements, tax, and portal services to acknowledge. If the tax engine takes 8 seconds under load, the billing transaction takes 8 seconds too, connection-pool slots drain, and the provider starts seeing timeouts and retrying — which multiplies the load you are already failing to handle. The outbox breaks that chain: the webhook handler commits one local row and returns 200 in single-digit milliseconds, and a slow consumer only grows its own lag without touching the ingest path. This is why “medium ops complexity” for the outbox is worth paying: you trade a background poller and a billing_outbox table for the guarantee that no downstream service can ever slow down or fail your webhook acknowledgement.

Step-by-Step Implementation

The four steps move a status change from the billing service to every consumer without loss or double-apply: capture change plus outbox atomically, drain to the bus, consume idempotently behind a state-machine guard, and reconcile as a backstop. The flow shows the fan-out.

1. Capture the change and an outbox row atomically

The provider event mutates local state and enqueues an internal event in the same transaction, so a downstream crash can never lose the event.

CREATE TABLE billing_outbox (
  id                UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id      UUID         NOT NULL,           -- subscription_id
  provider_event_id VARCHAR(255) UNIQUE NOT NULL,
  event_type        VARCHAR(64)  NOT NULL,
  payload           JSONB        NOT NULL,
  status            VARCHAR(16)  NOT NULL DEFAULT 'pending',
  retry_count       INT          NOT NULL DEFAULT 0,
  created_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at      TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending ON billing_outbox (created_at) WHERE status = 'pending';

2. Drain the outbox to the bus

A poller claims a batch with FOR UPDATE SKIP LOCKED so multiple workers never publish the same row, dispatches to the broker, then marks rows published.

async function drainOutbox(db, bus) {
  await db.transaction(async (trx) => {
    const { rows } = await trx.query(
      `SELECT id, aggregate_id, event_type, payload
         FROM billing_outbox
        WHERE status = 'pending'
        ORDER BY created_at
        LIMIT 100
        FOR UPDATE SKIP LOCKED`            // ✅ no two workers grab the same row
    );
    for (const row of rows) {
      await bus.publish(row.event_type, { subscriptionId: row.aggregate_id, ...row.payload });
      await trx.query(
        `UPDATE billing_outbox SET status = 'published', processed_at = now() WHERE id = $1`,
        [row.id]
      );
    }
  });
}

3. Consume idempotently with a state-machine guard

Each consuming service deduplicates on provider_event_id and rejects illegal transitions, so a replayed or out-of-order event is a no-op rather than corruption.

const ALLOWED = {
  trialing: ['active', 'canceled'],
  active:   ['past_due', 'canceled'],
  past_due: ['active', 'unpaid', 'canceled'],
};

async function consume(evt, db) {
  const dup = await db.query('SELECT 1 FROM processed_events WHERE event_id = $1', [evt.providerEventId]);
  if (dup.rowCount > 0) return { status: 'duplicate' };          // ⚠️ already applied

  const sub = await db.query('SELECT state FROM local_subscriptions WHERE subscription_id = $1', [evt.subscriptionId]);
  if (!(ALLOWED[sub.rows[0].state] || []).includes(evt.targetState)) {
    return { status: 'illegal_transition' };                     // ✗ blocked
  }
  await db.transaction(async (trx) => {
    await trx.query('INSERT INTO processed_events (event_id) VALUES ($1)', [evt.providerEventId]);
    await trx.query('UPDATE local_subscriptions SET state = $1 WHERE subscription_id = $2',
      [evt.targetState, evt.subscriptionId]);
  });
  return { status: 'applied' };
}

The UNIQUE constraint on provider_event_id in the outbox is doing quiet but critical work. Providers occasionally deliver the same webhook twice with the same event id, and your receiver may itself retry the insert after a network blip. Because the local state mutation and the outbox insert share one transaction, the unique violation aborts the whole thing, so you cannot end up with a state change that has no corresponding outbox row or an outbox row with no state change. Catch the constraint error, treat it as a benign duplicate, and return 200 to the provider so it stops retrying. The partial index idx_outbox_pending keeps the poller’s hot query cheap: it only ever scans rows still in pending, so once a subscription has been quiet for a while its processed history never slows the drain, and the index stays small even as billing_outbox grows into the millions of rows.

3a. Carry a sequence number, not just a timestamp

Wall-clock created_at is fine for draining but too coarse for conflict resolution, because two events for one subscription_id can land in the same millisecond. Add a per-aggregate sequence — either the provider’s own event sequence if it exposes one, or a monotonic counter you assign at capture — and store the highest sequence you have applied on each local_subscriptions row. The consumer’s guard then becomes two checks: reject the event if its sequence is lower than the stored high-water mark (a stale replay), and reject it if the state-machine transition is illegal. The first check is what makes an out-of-order active arriving after a later canceled a safe no-op instead of a resurrection. This is more robust than relying on ALLOWED transitions alone, because the transition table cannot distinguish “a legitimate new active” from “a delayed old active” — only the sequence can.

4. Reconcile against the provider

A scheduled job re-fetches provider subscriptions and overrides local billing fields, catching anything the event stream dropped. Defer to provider truth for status and cancel_at_period_end; keep local metadata as-is.

def reconcile(tenant_shard: int) -> None:
    stale = db.query(
        "SELECT subscription_id FROM local_subscriptions "
        "WHERE last_synced_at < now() - INTERVAL '2 hours' AND tenant_shard = %s",
        tenant_shard,
    )
    for sub_id in stale:
        remote = provider.get_subscription(sub_id)        # honours Retry-After + circuit breaker
        if remote.status != db.local_status(sub_id):
            db.apply_override(sub_id, remote.status, reconciliation_job_id=JOB_ID)  # ✅ audited

Verification & Testing

The three tests prove convergence under scrambled order, idempotency under replay, and exactly-once publishing when the broker dies mid-drain. The reconciliation query is the zero-drift proof. The panel lists them.

Assert convergence under chaos: feed the consumer a scrambled sequence (canceled before past_due before active) and assert the final state equals the highest-sequence event, regardless of arrival order. Replay one provider_event_id twice and assert exactly one row in processed_events and one state change. Run this reconciliation query in staging to prove zero drift after a sync:

SELECT l.subscription_id, l.state AS local, p.status AS provider
FROM   local_subscriptions l
JOIN   provider_snapshot   p USING (subscription_id)
WHERE  l.state <> p.status;   -- expect zero rows post-reconciliation

Integration-test the poller by killing the broker mid-drain and asserting that pending rows are re-claimed and published exactly once on recovery (no duplicates downstream because consumers are idempotent).

Testing the crash window explicitly

The broker-death test hides a subtle ordering of failures worth pinning down with its own case. There are two crash windows in drainOutbox: after bus.publish succeeds but before the UPDATE ... SET status = 'published' commits, and after the commit but before the next batch. The first window is the dangerous one — the row is still pending, so on restart the poller republishes it, and the event reaches every consumer a second time. Your test must inject a fault precisely between publish and update (a mock bus that succeeds, then a transaction that throws) and then assert that each consumer’s processed_events table still holds exactly one row for that provider_event_id. If that assertion holds, you have empirically demonstrated the at-least-once-plus-idempotent equals effectively-once property that the whole design rests on; if it fails, your dedup key is wrong or your consumer applies effects before recording the id. Also assert the inverse for reconciliation: run it against a local_subscriptions row deliberately set to a stale status, then check that apply_override fired with a non-null reconciliation_job_id so the override is auditable and distinguishable from an event-driven change in your history table.

What to measure in production, not just in tests

Convergence tests prove the logic; production tells you whether the logic is fast enough. Track consumer lag as the age of the oldest pending outbox row and as the delta between a subscription’s provider-side updated timestamp and its local last_synced_at. A p99 of a few seconds is healthy; a p99 climbing into minutes means the poller batch size or broker throughput is the bottleneck, and customers are living in split-brain reads while it drains. Emit the count of illegal_transition and stale-sequence rejections as a metric too — a sudden spike is usually not an attack but a sign that one consumer’s ALLOWED table has drifted out of sync with a new provider status the billing service already understands.

Gotchas & Production Pitfalls

The pitfalls here are about timing and duplication at the consumer edge: reacting too fast to past_due, double-suspending on duplicate failures, clock skew on renewal windows, and reconciliation tripping rate limits. The map groups them.

• Grace-period drift: delayed webhook delivery can leave a subscription past_due during a legitimate grace window. Enforce a configurable grace period (e.g. 72 hours) before any service suspends access, rather than reacting to the first past_due.
• Double suspension from duplicate failures: repeated invoice.payment_failed events can suspend twice and fire two dunning emails. Deduplicate on provider_event_id and verify the invoice_id before mutating entitlements.
• Timezone skew on current_period_end: services compute renewal windows differently if any one of them works in local time. Store and compare everything in UTC; apply offsets only at the presentation layer.
• Reconciliation rate-limit exhaustion: a naive full-table reconciliation will trip the provider’s API limits and partially sync. Shard by tenant, honour Retry-After, and cache successful reads with a short TTL.
• Outbox poller falling behind: if the poller lags by hours you get split-brain reads. Add a health check that halts downstream mutations when consumer lag exceeds a threshold, and alert on pending rows older than five minutes.

The reconciliation-versus-event race

The pitfall that survives every other fix is reconciliation and the live event stream fighting over the same subscription. Picture a reconciliation job that reads the provider at T0 and sees active, then a real invoice.payment_failed webhook flips the subscription to past_due at T0+200ms and commits, and finally the reconciliation write lands at T0+400ms and stamps the stale active back over it. Now your slow backstop has actively corrupted state that the fast path had correct. The fix is to make reconciliation writes conditional on the sequence high-water mark, exactly like event consumers: apply_override must refuse to lower or reorder a subscription whose local sequence is newer than the snapshot the job read. Treat the reconciliation read as an event carrying the provider’s current sequence, not as an unconditional source of truth, and the race disappears. This is also why reconciliation should override only billing-authoritative fields — status, cancel_at_period_end, current_period_end — and never local-only fields like feature flags or seat counts that no provider snapshot knows about.

Fan-out that is not idempotent at the effect layer

Deduplicating the state change is necessary but not sufficient when a consumer produces external side effects. The entitlements service can correctly record one state transition to canceled yet still call a downstream provisioning API twice if the dedup check and the side effect are not in the same transactional boundary. Keep the effect inside the guard: record the provider_event_id in processed_events and enqueue the provisioning call in the same commit, then let a separate worker drain that queue idempotently, rather than firing the API call inline where a retry after a mid-handler crash re-invokes it. The rule generalizes — dedup protects your database, but any effect that leaves the database (an email, a webhook to a customer, a call to a fulfilment system) needs its own idempotency key derived from the provider_event_id so replays collapse there too.

Frequently Asked Questions

Should services read subscription state directly from the billing database? No. A shared table becomes an implicit contract that nobody can change, and the coupling shows up as a migration that requires six deploys.

Is polling ever acceptable instead of events? As a backstop, yes. A periodic reconciliation that corrects drift is worth having even with reliable events, because it turns a missed event from a permanent inconsistency into a bounded one.

How should services handle a state they do not recognise? Conservatively, and loudly. Treating an unknown status as active grants access that may not be paid for; treating it as inactive can lock out a paying customer. Log it and pick the safer default for that service.

What ordering guarantees are needed? Per subscription, at minimum. A cancellation applied before the renewal that preceded it produces exactly the state corruption these patterns exist to prevent.

Related

• Database Sync & Consistency Patterns
• Idempotency & Event Deduplication
• Outbox Pattern & Event Publishing
• Resolving Out-of-Order Webhook Delivery Issues