Grace Period & Retry Logic

A grace period turns an involuntary payment failure into a recoverable event instead of an instant cancellation, and the retry logic behind it is where most SaaS revenue recovery lives or dies. Built as part of Frontend Checkout UX & Dunning Recovery Flows, this subsystem must balance three forces that pull against each other: recovering revenue, respecting card-network velocity limits, and not annoying a customer whose card simply had a temporary hold. The wrong retry cadence trips issuer fraud heuristics; the wrong grace window either churns recoverable accounts or extends free service to dead cards.

The recovery engine depends on a clean vault. Token lifecycle management and Secure Card Vaulting & Tokenization are prerequisites for off-session retries that do not re-prompt the customer or trigger an avoidable SCA challenge. Everything here assumes charges are merchant-initiated against stored credentials.

Grace and retry are two separate mechanisms that people conflate. The grace period is a business decision about entitlement: how long a subscription_id keeps its feature flags and seat access after a renewal fails to settle. Retry is a technical decision about authorization: when and how often you re-present a stored credential to the network. They interact but they are not the same clock. You can run a fourteen-day grace window with only three retry attempts, or a three-day grace window with a retry every twelve hours; the grace window bounds how long you are willing to serve unpaid usage, while the retry schedule bounds how aggressively you chase the money. Keeping them as distinct fields — grace_expires_at and next_retry_at on the subscription row — is what lets product tune churn tolerance without an engineer touching the scheduler, and lets the payments team tune authorization rates without changing what the customer can see.

The economics are concrete enough to reason about directly. Involuntary churn — failures caused by expired cards, insufficient funds, or issuer risk holds rather than a deliberate cancel — typically accounts for 20% to 40% of gross churn in a card-billed SaaS book. A recovery engine that lifts the eventual settlement rate on failed renewals from 55% to 70% is not a rounding error; on a book doing 2,000,000,00 (twenty million dollars in minor units of ARR) that fifteen-point swing is worth low seven figures a year. That is the reason this subsystem earns real engineering time rather than a naive “retry tomorrow” cron: every percentage point of recovered authorizations flows straight to net revenue retention with no acquisition cost attached.

Prerequisites

Off-session recovery assumes a clean vault, an explicit FSM, a delayed-delivery queue, and decline-code mapping. Each prerequisite is what makes a retry safe rather than a churn accelerator. The stack lists them.

The reason these are hard prerequisites rather than nice-to-haves is that a retry engine built on a missing foundation does not fail loudly — it fails as slow revenue leakage that nobody notices for a quarter. Skip the tokenized vault and your retries either re-prompt the customer (defeating the point of off-session recovery) or fall back to a raw PAN you should not be holding. Skip the explicit FSM and two concurrent workers can both observe past_due, both fire a charge, and both succeed, double-billing a customer_id who then disputes and costs you a chargeback fee on top of the refund. Skip the delayed job queue and you are back to a nightly cron that retries every failed account at 02:00 UTC, producing exactly the synchronized authorization spike that issuer fraud models are tuned to flag. Skip decline-code mapping and you retry a stolen_card decline, which not only never succeeds but actively pushes your merchant authorization rate down, because networks weight repeated attempts against a flagged credential.

The idempotency key deserves special attention because it is the one prerequisite that must be right on the very first attempt, not retrofitted later. Derive it deterministically from the tuple that identifies a unique charge intent — for example hash(subscription_id + invoice_id + attempt_number) — so that a network timeout followed by a re-send presents the same idempotency_key and the gateway collapses them to one authorization. A key derived from a timestamp or a random UUID generated per HTTP call defeats the entire mechanism: the retry looks like a new charge, and an ambiguous timeout becomes a genuine double charge. Store the key alongside the attempt row before the network call is made, never after, so a process crash between call and persistence still leaves a recoverable record.

Retry prerequisites A tokenized vault, a subscription FSM, a delayed job queue, decline-code mapping, and idempotency keys underpin safe retries. Recovery engine Tokenized vault off-session Subscription FSM explicit states Job queue delayed + jitter Decline map soft vs hard Idempotency per attempt
Five foundations — the vault and the decline map are what let a retry recover revenue without re-prompting the customer.

Architecture & Data Flow

A failed renewal moves the subscription into a grace period, the decline code determines whether a retry is even scheduled, and each retry either recovers the account or advances the FSM toward suspension. The state machine, not a cron job, owns the lifecycle; the scheduler only enqueues the next attempt.

The distinction between “the FSM owns the lifecycle” and “the scheduler enqueues attempts” is the single most important architectural boundary on this page, so it is worth being precise about the data flow. When a renewal invoice for invoice_id fails, the billing engine emits a charge.failed event carrying the raw gateway response. A consumer maps the decline code, and only if it is soft does it write a row to the retry_schedule table with a computed next_retry_at. The scheduler is a dumb loop: it polls for rows whose next_retry_at has passed and whose owning subscription is still past_due, dispatches a charge, and writes the outcome back. It never decides whether the subscription should be suspended — that decision belongs to a separate reconciliation pass that reads grace_expires_at against the current time. This separation means a bug in the scheduler can at worst over- or under-retry; it can never leave a subscription in a state the FSM forbids, because every write goes through the guarded transition.

Idempotency and event ordering across the flow

The flow crosses at least three trust boundaries — your billing engine, the payment gateway, and the gateway’s asynchronous webhook channel — and each boundary can duplicate, drop, or reorder a message. The design assumption must be that every event will be delivered at-least-once and occasionally out of order. Concretely, a single retry can produce a synchronous API response (the charge call returns succeeded) and, seconds to minutes later, an asynchronous charge.succeeded webhook describing the same authorization. Both paths must funnel through the same idempotency check keyed on the gateway’s authorization id so the ledger for that customer_id moves exactly once. Treat the synchronous response as an optimization and the webhook as the source of truth: if they disagree, the webhook wins after signature verification, because the synchronous path can be lost to a socket reset while the charge still settled on the network.

Where the grace clock actually lives

A recurring design mistake is scattering the grace deadline across the codebase — a check in the API gateway, another in the feature-flag service, a third in the scheduler — each recomputing “is this subscription still in grace” from slightly different inputs. Store one authoritative grace_expires_at timestamp on the subscription row, set it once when the subscription enters past_due, and have every other component read it rather than recompute it. Entitlement checks then reduce to now < grace_expires_at, a single comparison that cannot drift between services. When product wants to extend grace for a specific high-value account, they update one field; when a customer pays and recovers, the field is cleared as part of the past_due → active transition. Recomputing the deadline from the original failure date plus a config constant in three places guarantees that a config change mid-flight will suspend some accounts early and others late.

Grace period retry state machine A failed charge enters a grace period; soft declines schedule retries, hard declines suspend immediately, and exhausted retries lead to cancellation. active charge fails past_due grace window retry soft decline suspended hard / expired recovered or canceled
Soft declines loop through retries within the grace window; hard declines and exhausted retries drive the subscription to suspension and cancellation.

Implementation Walkthrough

The five steps model the FSM, classify the decline, schedule soft-decline retries with jittered backoff, pause for SCA, and commit only after verification. The backoff timeline is the heart of it — retries spread across the grace window with per-issuer jitter.

Retry backoff timeline Soft-decline retries fire at day 0, day 3, day 7, and day 14 within the grace window, each jittered to avoid hammering one issuer. day 0 day 14 (grace end) R0 R3 R7 R14 each retry ± jitter · hard declines skip this entirely · exhausted → suspended
Tiered backoff at day 0, 3, 7, and 14 with jitter — hard declines skip retries, exhausted schedules suspend.

1. Model the lifecycle as a deterministic FSM

Transitions are explicit and version-guarded; concurrent mutations are rejected via optimistic locking.

The ALLOWED map is doing more than documentation — it is the enforcement point that makes every other guarantee on this page hold. Notice that CANCELED maps to an empty array: once a subscription is canceled it is terminal, and any code path that tries to reactivate it must go through an explicit new-subscription flow rather than a transition, because a “resurrected” cancellation carries stale entitlement and billing-anchor assumptions. Notice too that SUSPENDED can return to ACTIVE but past_due is not reachable from suspended: once you have stopped serving the product, a later successful charge recovers directly to active rather than re-entering the grace window, so a customer cannot ride an endless loop of suspend-recover-suspend to get free service across many grace periods.

Optimistic locking is what makes the guard safe under concurrency. Carry a version column on the subscription row and include it in the WHERE clause of every transition update: UPDATE subscriptions SET state = $new, version = version + 1 WHERE subscription_id = $id AND version = $observed. If the update affects zero rows, another worker moved the row first and you re-read and re-decide rather than blindly overwriting. Without this, the classic race is a scheduled retry succeeding at the same millisecond a customer updates their card in the portal — both observe past_due, both try to write active, and one silently clobbers the other’s version history, which later makes the audit log unreadable. The optimistic-lock retry loop is cheap because contention on a single subscription row is rare; the correctness it buys under the rare collision is not.

enum SubState { ACTIVE='active', PAST_DUE='past_due', SUSPENDED='suspended', CANCELED='canceled' }

const ALLOWED: Record<SubState, SubState[]> = {
  [SubState.ACTIVE]:    [SubState.PAST_DUE],
  [SubState.PAST_DUE]:  [SubState.ACTIVE, SubState.SUSPENDED, SubState.CANCELED],
  [SubState.SUSPENDED]: [SubState.ACTIVE, SubState.CANCELED],
  [SubState.CANCELED]:  [],
};
function assertTransition(from: SubState, to: SubState) {
  if (!ALLOWED[from].includes(to)) throw new Error(`invalid ${from}->${to}`); // ✗ reject
}

2. Classify the decline

A hard decline must never be retried; a soft decline enters the schedule. The detailed code taxonomy lives in Smart Retry Timing With Card Issuer Decline Codes.

The classification is not merely soft-versus-hard; the granularity matters because the right response differs within each bucket. insufficient_funds is soft and genuinely worth several retries, because balances refill on payday cycles — a retry timed for the 1st or 15th of the month lands far better than one two hours after the failure. expired_card is technically a decline but is not a transient one: retrying the same token will never succeed, yet it is not fraud either, so the correct action is to fire a network account updater lookup and, in parallel, prompt the customer to update the card in the portal, rather than either retrying blindly or hard-suspending. do_not_honor is the treacherous middle case: it is a soft-looking generic decline that often masks a permanent issuer block, so cap it at one or two attempts rather than the full schedule you would grant insufficient_funds.

A subtle but expensive mistake is treating the gateway’s normalized decline code as ground truth without checking the issuer’s raw response where the gateway exposes it. Two issuers can both return a code your gateway normalizes to generic_decline, but one carries an advice code meaning “do not retry, account closed” while the other means “temporary, try again.” Where that advice code is available — Visa and Mastercard both carry retry-eligibility hints in the authorization response — feed it into the classifier so a permanently closed account for some customer_id is never scheduled, saving both the wasted authorization attempt and the drag on your merchant standing.

HARD_DECLINES = {"stolen_card", "lost_card", "fraudulent", "pickup_card", "do_not_honor"}

def is_hard_decline(decline_code: str) -> bool:
    return decline_code in HARD_DECLINES  # ✗ no retry path

3. Schedule soft-decline retries with backoff and jitter

Spread retries across the grace window and add per-issuer jitter so a renewal cohort does not hammer one issuer simultaneously.

The choice of [0, 3, 7, 14] days is deliberate rather than a geometric backoff, and the reasoning is worth making explicit because copying a generic exponential schedule here loses money. An immediate retry at day 0 catches genuinely transient failures — a network hiccup at the issuer, a momentary hold that cleared. The day-3 and day-7 attempts straddle the most common paycheck cadences without clustering on any single date. The day-14 attempt is the last chance before the grace window closes, deliberately placed at the boundary so the final authorization has the freshest possible balance behind it. Pure exponential backoff, by contrast, would bunch attempts early (day 0, 1, 2, 4) and then go quiet exactly when a paycheck would have refilled the account, which is the opposite of what recovery data supports.

The ±6h jitter window is sized against a specific failure mode, not chosen arbitrarily. Consider a large cohort of subscriptions that all renew on the 1st and all fail; without jitter they would all schedule their day-3 retry for the same minute on the 4th, presenting a synchronized burst to whichever issuer holds the largest share of that BIN. Issuer fraud systems watch for exactly this signature — many authorizations against one BIN in a tight window — and can start declining otherwise-good cards. Six hours of uniform jitter spreads a cohort of tens of thousands of retries across a wide enough band that the per-minute rate to any one issuer stays under its velocity radar, while still landing every attempt within the intended day. Widen the jitter for very large cohorts and narrow it for small ones; the invariant to preserve is that peak per-issuer authorizations-per-second stays below the threshold you have negotiated or observed.

One implementation caveat: compute next_retry_at from a stable anchor, not from “now” at the moment of each failure. If attempt N failed at an odd hour because a previous retry was itself delayed, anchoring the next offset on that odd hour drifts the whole schedule. Anchor the offsets on the original renewal-failure timestamp for invoice_id so [0, 3, 7, 14] always means “days since the renewal failed,” keeping the schedule aligned to the grace window regardless of when individual attempts actually executed.

import random
from datetime import datetime, timedelta, timezone

BACKOFF_DAYS = [0, 3, 7, 14]  # within a 14-day grace window

def next_retry_at(attempt: int) -> datetime | None:
    if attempt >= len(BACKOFF_DAYS):
        return None  # ⚠️ exhausted — advance FSM to suspended
    jitter_h = random.uniform(-6, 6)  # ±6h to break up cohorts
    return datetime.now(timezone.utc) + timedelta(days=BACKOFF_DAYS[attempt], hours=jitter_h)

4. Pause and authenticate on SCA

When the issuer demands step-up, off-session retries fail. Pause the queue and send the customer a secure authentication link rather than retrying blindly.

The reason a requires_action result must halt rather than back off is that it is not a transient failure — it is the issuer telling you that this particular credential now needs a cardholder present to authenticate. Retrying it off-session will return requires_action again every time, so a naive backoff loop burns the entire retry budget producing identical failures while the fourteen-day clock runs out. The only path forward is to move the customer on-session: send a link that opens a hosted authentication page where the issuer can present its 3-D Secure challenge with the cardholder actually there to complete it. Once they authenticate, the resulting successful payment method can carry a fresh exemption for subsequent merchant-initiated charges.

There is a jurisdictional wrinkle worth encoding in the pause logic. Under PSD2 in the European Economic Area, off-session merchant-initiated transactions rely on an exemption that the issuer can decline at will; in card markets without a mandate the same off-session charge usually clears without any step-up. So the requires_action branch is common for EEA-issued cards and rare elsewhere, which means the same retry schedule produces very different pause rates by region. Track pause rate as a per-region metric so a sudden spike — say EEA requires_action jumping from 8% to 25% of retries — surfaces an exemption-configuration regression at the gateway rather than being buried in an aggregate that looks fine.

def on_retry_result(result: dict) -> None:
    if result.get("requires_action"):
        pause_retry_queue(result["subscription_id"])         # ⚠️ stop retrying
        send_authentication_link(result["customer_id"])      # MIT/COF exemption failed
    elif result["status"] == "succeeded":
        transition(result["subscription_id"], SubState.ACTIVE)  # ✅ recovered

5. Commit only after signature verification

Retry-result webhooks must pass HMAC verification and idempotency before the ledger moves.

The ordering of the three checks in the handler is not cosmetic — it is a security and correctness ordering. Signature verification comes first because everything downstream trusts the payload’s contents; if you parse and act on JSON before verifying the HMAC, an attacker who guesses your webhook URL can forge a charge.succeeded and mark a delinquent subscription_id as paid. Use hmac.compare_digest rather than == so the comparison runs in constant time and does not leak the secret one byte at a time through timing. The idempotency check comes second, after verification but before the transaction, so a duplicate delivery of a legitimate event returns early without taking a database lock. Only then does the ledger move, and it moves inside a single transaction that also records the event id, so a crash after ledger.apply but before store.mark cannot commit the money without the dedupe record — they either both land or neither does.

The ttl=604800 on the idempotency store — seven days in seconds — is a deliberate trade between memory and safety. It must comfortably exceed the gateway’s own webhook retry horizon, because a webhook the gateway gives up redelivering after, say, three days should still be recognized as a duplicate if it arrives on day four due to a queue backlog on your side. Setting the TTL shorter than the gateway’s redelivery window reopens the double-apply hole precisely for the slow, retried deliveries that idempotency exists to catch. Seven days is a safe margin for the common gateways; if yours documents a longer redelivery tail, match it.

def handle_retry_webhook(payload: bytes, signature: str, store, ledger) -> None:
    expected = hmac.new(WEBHOOK_SECRET.encode(), payload, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        raise ValueError("invalid signature")  # ✗ reject spoofed payload
    event = json.loads(payload)
    if store.exists(event["id"]):
        return  # ✅ duplicate
    with db.transaction():
        ledger.apply(event)
        store.mark(event["id"], ttl=604800)

Edge Cases & Failure Modes

The retry edge cases split into ambiguous outcomes (timeouts, out-of-order webhooks), timing collisions (retry waves, portal-update races), and SCA. The map groups them so the defense — idempotency, jitter/cancellation, or a pause — is obvious.

Retry edge cases Ambiguous outcomes need idempotency reconciliation, timing collisions need jitter and job cancellation, and SCA needs a paused queue with an authentication link. Ambiguous gateway timeout out-of-order webhook → idempotency reconcile Timing retry wave portal update race → jitter + cancel job SCA step-up on off-session card expires mid-grace → pause + auth link
Three categories — the SCA pause and the portal-race cancellation are the two that most often double-charge if missed.
Scenario Mitigation
Gateway timeout leaves attempt in ambiguous state Reconcile via idempotency key; never blind-retry
Card expires mid-grace Trigger network account updater; request portal update
Synchronized retry wave trips issuer velocity Per-issuer jitter; cap concurrent retries per BIN
Portal payment update races a scheduled retry Cancel pending retry job on successful update
Out-of-order retry webhooks Sequence validation; buffer until predecessor commits
SCA step-up on off-session retry Pause queue; send authentication link; resume on success

Performance & Scale

The retry queue is the pressure point: a fixed schedule spikes; per-issuer jitter flattens it; a circuit breaker bounds an outage; and a per-BIN cap keeps you under network velocity. The diagram shows the four levers.

Retry scale levers Per-issuer jitter flattens spikes, an indexed scheduler pulls due jobs, a circuit breaker halts on a 15 percent failure rate, and a per-BIN cap respects velocity limits. Jitter flattens spike per-issuer Indexed pull (next_at, status) range scan Circuit breaker >15% / 5 min fail open Per-BIN cap hourly limit under velocity
Four levers keep the retry queue under gateway and issuer limits — jitter and the per-BIN cap protect authorization rates.

The retry queue is the scaling pressure point. At 100k subscriptions, a fixed-interval schedule (everyone retried at exactly day 3) produces a spike that exceeds gateway and issuer limits — per-issuer jitter flattens it. Index the scheduler table on (next_retry_at, status) so the dispatcher pulls due jobs with a single range scan. Enforce a circuit breaker per gateway: if the failure rate exceeds 15% in a rolling 5-minute window, halt retries and fail open to a secondary processor rather than amplifying an outage. Cap retries per BIN per hour to stay under network velocity thresholds, and defer tax recalculation until a charge settles rather than re-running it on every failed attempt.

Testing Strategy

The tests drive a mock clock for grace-window timing, replay a webhook for idempotency, assert hard declines enqueue zero retries, and prove the SCA pause. The panel lists them before the detail.

Retry tests A mock clock times the suspension, a replayed webhook yields one ledger entry, a hard decline enqueues zero retries, and an SCA result pauses the queue. Mock clock grace expiry suspends on time Replay webhook twice one entry Hard decline do_not_honor zero retries SCA result requires_action pause + link
Four tests — the hard-decline-zero-retries and SCA-pause tests protect authorization rates and avoid blind retries.

Drive the FSM with a mock clock to assert that a subscription advances to suspended exactly when the grace window expires, not a second early. Replay the same retry webhook twice and assert one ledger entry. Feed a do_not_honor decline and assert zero retry jobs are enqueued; feed insufficient_funds and assert the backoff schedule materializes with jitter inside expected bounds. Forge an invalid HMAC and assert rejection before any state change. Simulate an SCA requires_action result and assert the queue pauses and an authentication link is sent rather than a blind retry.

Frequently Asked Questions

How should retries handle an SCA challenge during the grace period? Off-session retries rely on merchant-initiated-transaction and credential-on-file exemptions. When the issuer still demands step-up, pause the retry queue, send the customer a secure authentication link via the portal, and resume scheduling only after successful verification or timeout — never retry blindly into a requires_action response.

What retry cadence balances recovery against issuer penalties? A tiered backoff with jitter works well: an immediate retry for soft declines, then day 3, day 7, and day 14 within a two-week grace window. Hard declines bypass retries entirely. The decline code should drive timing — see Smart Retry Timing With Card Issuer Decline Codes.

How do I stop double billing when retry webhooks arrive out of order? Key idempotency on the gateway event id and validate a monotonic sequence per subscription. Buffer out-of-order events until the preceding transition is committed, and wrap every ledger move in a single transaction so duplicates and reorders converge to one outcome.

Should tax be recalculated on every failed retry? No. Recalculate only on successful settlement or when the grace period crosses a fiscal boundary or jurisdiction change. Re-running tax on each failure adds API load and risks rounding drift in the ledger.