Running Dual-Provider Writes During a Cutover

The riskiest period of a provider migration is not the cutover itself but the weeks around it, when some subscriptions bill at the old provider, some at the new one, and both are sending events about customers they may or may not still own. Every consistency bug in that window produces a customer-visible symptom: a double charge, a missed renewal, or a subscription that cancels itself because an event from the wrong provider was applied. This guide sits under Payment Provider Migration & Portability and covers the mechanics of the overlap.

The organising rule is that exactly one provider owns each subscription at any instant, that ownership is recorded in your database rather than inferred, and that every event is checked against it before being applied. Everything else follows from that.

The second rule is that cohorts, not percentages, define the rollout. A bounded, listable set of subscriptions gives you a blast radius you can enumerate and a pause point you can actually use.

Trade-offs

Ordering cohorts by risk compounds the benefit. Small monthly card subscriptions first, larger monthly next, annual renewals after that, and anything on a mandate-backed rail last, since mandates frequently cannot be transferred and need re-collection rather than migration.

Step-by-Step Implementation

1. Make ownership explicit and monotonic

ALTER TABLE subscriptions
  ADD COLUMN provider          TEXT NOT NULL DEFAULT 'incumbent',
  ADD COLUMN provider_pinned   BOOLEAN NOT NULL DEFAULT FALSE,
  ADD COLUMN provider_moved_at TIMESTAMPTZ,
  ADD COLUMN provider_epoch    INT NOT NULL DEFAULT 0;   -- increments on each move

provider_epoch is what makes ownership monotonic. A late event carrying an older epoch is ignored, which prevents a delayed message from the previous provider from resurrecting the previous ownership.

2. Namespace every event key

def dedup_key(event) -> str:
    return f"{event.provider}:{event.id}"      # never bare event.id

Event identifiers collide across providers eventually, and a collision silently drops a real event as a duplicate. Making the change before the second provider exists is trivial; making it afterwards means rewriting an existing key space.

3. Route by cohort, and pause

def provider_for(sub) -> str:
    if sub.provider_pinned:
        return sub.provider                      # explicit override wins, e.g. after rollback
    if sub.subscription_id in current_cohort():
        return "incoming"
    return sub.provider

Pause after each cohort for at least one full renewal cycle of that cohort. The failures worth catching — declines at a different rate, an authentication challenge you did not expect, a webhook shape that does not map — appear at renewal, not at migration.

4. Reject events from the non-owner

Record ignored events rather than discarding them. During an investigation, “we received this and deliberately ignored it” is a far better answer than silence, and the record is what tells you when the old provider is still emitting for a large cohort you thought was fully moved.

5. Alert on ownership conflicts

-- Any subscription active at more than one provider is an incident, not a report.
SELECT subscription_id, array_agg(DISTINCT provider) AS providers
  FROM provider_subscription_mirror
 WHERE status = 'active'
 GROUP BY subscription_id
HAVING count(DISTINCT provider) > 1;

Page on a non-empty result immediately. Double billing is discovered by customers faster than by daily reports, and the recovery involves refunds and an apology rather than a quiet fix.

The Reconciliation That Runs Every Night

Three checks, all comparing your state against each provider independently — never the providers against each other, since they legitimately hold different subsets.

Ownership. No subscription active at both. This is the incident check above and it should return zero rows every night from the first cohort until decommissioning.

Existence. Every subscription your system believes is billing at a provider exists there and is active, and every active subscription at each provider is known to you. The first direction catches failed creations; the second catches orphans left behind by a partial cutover, which are the ones that quietly renew and charge a customer you thought had moved.

Amount. The next charge amount your system expects matches what the provider will charge. This catches drift introduced by an amendment applied on one side only — a quantity change made during the window between creating the new subscription and cancelling the old one.

Maintain a local mirror table for these checks rather than calling provider APIs for every subscription. The mirror is updated from webhooks and refreshed periodically, and it turns a reconciliation that would take thousands of API calls into a local query. It also keeps the check working during a provider outage, which is exactly when an ownership conflict is most likely to be created.

Retire the checks after decommissioning, with one exception: if you end up running two acquirers permanently for redundancy, keep the ownership check forever. It is cheap, and the condition it detects is one of the few billing defects that customers escalate publicly.

What Support and Finance Need During the Overlap

The overlap is not only an engineering condition; it changes what two other teams need to be able to see, and neglecting that turns ordinary questions into escalations.

Support needs to know, for any customer, which provider currently owns their subscription and which one holds each historical charge. Without it, a refund request becomes a guessing exercise and a “why was I charged twice?” question cannot be answered at all. Surface both on the customer view — current provider, and provider per charge — before the first cohort moves rather than after the first ticket.

Support also needs to know what changes for the customer. The statement descriptor may differ with a new acquirer, which produces a small wave of “I don’t recognise this charge” contacts that look like fraud reports. Brief the team on the new descriptor and, where the provider allows, keep it as close to the old one as possible.

Finance needs the payout shapes normalised before the first cohort, not after the first month-end. Two providers report fees, reserves, and payout timing differently, and a close performed by manually merging two report formats is a close that will be late. Normalising both into your own ledger, as covered in reconciling payouts against your ledger, makes the overlap invisible to the accounting process.

Finance also needs the revenue reporting to remain continuous across the boundary. Because your own subscription state — not the provider’s — is the source for revenue metrics, this usually costs nothing, but it is worth verifying explicitly: run the movement job across a migrated cohort’s boundary month and confirm no phantom churn or new-sale movements appear where a subscription merely changed provider.

Verification & Testing

Rehearse the rollback before the first cohort moves. Pin a migrated test subscription back to the legacy provider, run a renewal, and assert the legacy token and legacy path were used. A rollback discovered to be broken during an incident is worse than having no rollback, because the plan assumed one existed.

Test the ownership guard directly: deliver an event from the non-owning provider for a migrated subscription and assert nothing changed except the ignored-event record. Then deliver a dispute event from the non-owning provider and assert it was processed, since disputes route by charge. Both assertions in one test file document the rule better than any comment.

Test the epoch logic with an out-of-order event carrying a stale epoch, and assert it is ignored rather than reverting ownership. Multi-day delivery windows make this scenario real rather than theoretical, and it is the failure that is hardest to diagnose after the fact.

Gotchas & Production Pitfalls

• Source subscription not cancelled. Creating the new subscription without cancelling the old one double-bills at the next renewal. Do both in one transaction and reconcile nightly regardless.
• Bare event identifiers. Cross-provider collisions drop real events as duplicates. Namespace before the second provider exists.
• Disputes routed by subscription. Sends the dispute to a provider that never took the charge, and the evidence deadline passes unnoticed.
• Cohort defined at charge time only. If the cohort is recomputed rather than recorded, a change to the cohort function silently moves subscriptions back and forth. Record membership.
• Pausing for a week instead of a cycle. The interesting failures happen at renewal; a pause shorter than the cohort’s billing period proves nothing.

Frequently Asked Questions

Should we write to both providers simultaneously? No. Dual writes mean two systems of record and reconciliation between them; what you want is dual availability with a single owner per subscription. Write to one, keep the other reachable.

How long should the overlap last? Until the last charge at the old provider passes its dispute horizon, which for cards is at least 120 days after the final renewal. Budget for the incumbent’s minimum fees across that period.

What if a cohort shows worse authorisation rates? Pause, and compare by issuer and country before continuing. A new acquirer with weaker issuer relationships in one market is a real risk, and it is far cheaper to discover on one cohort than on the whole base.

Can the mirror table be skipped? Only at small scale. Above a few thousand subscriptions the reconciliation becomes an API-rate-limit problem, and it stops working during exactly the provider incidents when you most need it.

Related

• Payment Provider Migration & Portability
• Building a Provider-Neutral Payment Interface
• Migrating Stored Cards Between Payment Processors
• Resolving Out-of-Order Webhook Delivery Issues
• Database Sync & Consistency Patterns