Caching Entitlement Checks Without Stale Access

An entitlement check runs on every authenticated request, so it must be fast; it decides whether a customer may use something they may have stopped paying for, so it must be fresh. Those two requirements pull against each other, and the usual compromise — a five-minute cache with manual purges — quietly produces the worst of both: still slow enough to matter on a cold miss, still stale enough that a downgraded customer keeps enterprise features for minutes. This guide sits under Entitlements & Feature Gating and covers the caching design that makes staleness bounded and observable rather than accidental.

The moment this becomes urgent is a real one: a customer’s payment recovers at 03:00, their subscription lifecycle state returns to active, and they still cannot log in because six application instances each hold a cached snapshot that says otherwise. Nothing is broken and nothing will fix itself until a TTL expires. Version-keyed caching removes that failure mode entirely.

Trade-offs

Strategy	Staleness bound	Invalidation cost	Cold-miss cost	Notes
No cache	Zero	None	Every request	Correct and too slow above modest traffic
TTL-only cache	The TTL	None	One store read	Simple; staleness is the TTL, always
Purge on change	Near zero, if delivered	Fan-out to every node	One store read	Fails silently when a purge is missed
Version-keyed	Version propagation time	One small read	One store read	Stale entries become unreachable, not wrong

The asymmetry that matters: a failed purge serves wrong data, while a failed version read only costs a cache miss.

Version-keying converts a correctness problem into a performance problem, which is a trade every operator should take. The cost is one extra read to learn the current version — a few hundred bytes, cacheable for a second or two — and the benefit is that no code path can serve an entry from before a change.

Step-by-Step Implementation

1. Give every snapshot a monotonic version

The version must increase on every change and must come from an ordered source: the subscription’s own sequence number, the event ordinal, or a per-account counter incremented in the same transaction that writes the snapshot. Wall-clock time is not acceptable, because clock skew between publishers reorders versions.

-- The account's current entitlement version, bumped in the same transaction as the snapshot.
CREATE TABLE entitlement_versions (
  account_id UUID PRIMARY KEY,
  version    BIGINT NOT NULL DEFAULT 1,
  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

UPDATE entitlement_versions
   SET version = version + 1, updated_at = now()
 WHERE account_id = $1
RETURNING version;

2. Build the cache key from account and version

def snapshot_key(account_id: str, version: int) -> str:
    return f"ent:{account_id}:v{version}"

Every layer — the in-process map, the shared cache, the CDN if you have one — uses this key. When the version moves, every layer misses simultaneously without any of them being told, because the key they hold no longer matches the key being requested.

3. Read the version cheaply, then the snapshot

VERSION_TTL_SECONDS = 2       # short: this is the staleness bound
SNAPSHOT_TTL_SECONDS = 900    # long: safe, because the key contains the version

def get_entitlements(local, redis, account_id: str) -> dict:
    vkey = f"entver:{account_id}"
    version = local.get(vkey)
    if version is None:
        version = int(redis.get(vkey) or load_version_from_db(account_id))
        local.set(vkey, version, ttl=VERSION_TTL_SECONDS)

    skey = snapshot_key(account_id, version)
    snap = local.get(skey)
    if snap is None:
        snap = redis.get(skey)
        if snap is None:
            snap = build_and_store_snapshot(account_id, version)
        local.set(skey, snap, ttl=SNAPSHOT_TTL_SECONDS)
    return snap

The asymmetric TTLs are the design. The version is tiny and read constantly, so a two-second local TTL costs almost nothing and caps staleness at two seconds. The snapshot is larger and expensive to rebuild, so it can live for fifteen minutes safely — it can never be served after a change, because a change produces a different key.

4. Choose a failure policy per feature

A blanket policy either takes the product down during a cache incident or gives away paid capacity — the choice belongs per feature.

Implement it as a property of the feature definition rather than a try/except at the call site, so the behaviour is visible in one place and testable.

FAIL_OPEN = {"dark_mode", "beta_ui", "csv_export"}   # low-risk, read-only

def check(feature: str, snapshot: dict | None) -> bool:
    if snapshot is not None:
        return bool(snapshot.get(feature))
    return feature in FAIL_OPEN        # store unavailable

5. Serve the last known good snapshot during an outage

Keeping a longer-lived copy — an hour, in a separate key or on local disk — gives the fail-open path something better than a guess. Mark responses served from it so dashboards can show how much traffic is running on degraded entitlements, which is the number that tells you whether an incident is affecting revenue.

Measuring Staleness as a Service Level

Staleness is usually assumed rather than measured, which is why it surprises people. Instrument it directly: when a snapshot is published, record the publish timestamp inside it; when a request reads a snapshot, emit the difference between now and that timestamp as a histogram. That single metric answers “how far behind is enforcement?” with a distribution rather than an anecdote, and it makes the effect of a TTL change visible immediately.

Set an explicit objective — for example, 99% of entitlement reads reflect changes within five seconds — and alert on the 99th percentile rather than the mean. The mean will look excellent even while a subset of instances hold minutes-old versions because a version read is failing on one node.

The second metric worth having is the version-miss rate: how often a request finds a version newer than the one it had cached. A sudden rise means a fan-out change touched many accounts and the snapshot store is about to see a rebuild storm. Rate-limit snapshot rebuilds per account with a short lock so a thundering herd on a popular account rebuilds once rather than a thousand times.

def build_and_store_snapshot(account_id: str, version: int) -> dict:
    lock = redis.set(f"entlock:{account_id}:{version}", "1", nx=True, ex=10)
    if not lock:
        # Someone else is building it; a brief wait beats a duplicate rebuild.
        return wait_for_key(snapshot_key(account_id, version), timeout=2.0)
    snap = resolve_from_grants(account_id)
    redis.set(snapshot_key(account_id, version), json.dumps(snap), ex=SNAPSHOT_TTL_SECONDS)
    return snap

Only the version entry can serve stale data, so the entire staleness budget is one small, cheap, short-lived value.

That concentration is the property to protect during future optimisation. Every proposal to reduce load by lengthening a TTL should be evaluated against which box it lands in: lengthening the snapshot TTL is free, and lengthening the version TTL spends directly from the staleness budget. Teams that lose track of this distinction end up with a fifteen-minute version cache and a support process for manually restarting instances after a plan change.

Verification & Testing

The test that matters most asserts that a change is visible within the staleness bound. Write a snapshot, bump the version, and assert that a reader with a warm cache observes the new value within the version TTL — not eventually, but within the number you have committed to. Run it with several simulated processes so the assertion covers cross-node behaviour rather than one in-process map.

Then assert the negative: an old key is unreachable. Fetch a snapshot at version 4, bump to version 5, and confirm that no code path can return the version-4 body. This is the property version-keying buys, and it is worth a test that would fail if someone “optimised” the key back to a plain account ID.

Test the failure policy explicitly by making the store unavailable and asserting each feature class behaves as documented — countable features deny, listed fail-open features allow — and that the response is not a 500. Finally, test the rebuild lock by firing many concurrent misses for the same account and asserting exactly one rebuild occurred.

Rolling the Cache Out Without an Incident

Introducing a cache in front of a correctness-critical check is one of the few changes where a gradual rollout genuinely reduces risk rather than just spreading it out. The sequence that works is shadow, then read-through for a subset, then full.

In the shadow phase, the middleware reads from the cache and from the authoritative store on every request, uses the authoritative answer, and logs any disagreement with the account, feature, and both values. Disagreements in this phase are almost never cache bugs — they are usually discoveries that two code paths disagreed about entitlements all along, and the cache simply made the disagreement visible. Resolve those before proceeding, because they will otherwise be attributed to the cache after cutover.

In the read-through phase, route a fraction of accounts to the cached answer. Choose the fraction by account rather than by request, so a single account’s experience is consistent and a report of “my feature keeps flickering” is not a possible outcome. Internal accounts first, then low-risk customers, then general availability.

Keep the disagreement logging permanently, sampled. It costs one extra read on a small percentage of requests and it is the only thing that will tell you when a new code path starts writing snapshots without bumping the version — the defect class that version-keying cannot protect against, because a version that never moves looks exactly like a system with no changes.

One last operational note: make the cache bypassable per request by an internal header, and expose it in support tooling. When a customer reports that a plan change has not taken effect, the first diagnostic question is whether the authoritative store already reflects it, and being able to answer that in one request rather than by reasoning about TTLs turns a twenty-minute investigation into a ten-second one.

Gotchas & Production Pitfalls

Version derived from wall-clock time. Two publishers with skewed clocks produce versions that go backwards, and a stale snapshot becomes reachable again. Use a sequence.
Local cache without an absolute TTL. A version-keyed entry is safe from staleness but not from unbounded memory growth. Bound every local entry regardless.
Purging as well as versioning. Adding purges on top of version keys reintroduces the failure mode you removed, because a purge that partially succeeds leaves inconsistent nodes. Pick one mechanism.
The version read becoming the bottleneck. If every request reads the version from a shared store, that store now sees your full request rate. The short local TTL is what prevents this; do not remove it to reduce staleness to zero.
Snapshots that embed derived data. Including anything that changes independently of grants — usage counters, for example — means the snapshot is stale for reasons the version cannot capture. Keep counters out of the snapshot.

Frequently Asked Questions

Is a two-second version TTL too aggressive? It is a small read against a shared cache, typically well under a millisecond, and it is what converts staleness from “however long the TTL is” into a bounded, measurable number. If the load is genuinely a problem, raise it to five seconds and state that as your bound.

Can I skip the shared cache and use only in-process memory? Only if you accept that a rebuild happens per process. With a handful of instances that is fine; with autoscaling it produces a rebuild every time an instance starts, which is exactly when it is least welcome.

What about long-lived connections such as WebSockets? They never re-enter the middleware, so entitlements must be re-checked on a timer or on an explicit invalidation message. A connection opened while entitled and never re-checked is an unbounded staleness window that no cache TTL covers.

How does this interact with the customer portal showing plan features? Serve the portal from the same resolver and the same snapshot. Showing a customer a different answer than the API enforces produces the most confusing support tickets in the entire billing surface.