Building Idempotent Webhook Handlers in Node.js

You reach for this the first time a payment provider double-fires a webhook in production and you discover a customer was charged twice — or a subscription was suspended and reinstated in a flickering loop. Duplicate deliveries are not an edge case; they are the contract. A 200 that takes too long, a deploy that drops a connection, or the provider’s own retry policy all produce repeats of the same invoice.payment_succeeded. Building idempotent webhook handlers in Node.js is about making the second, third, and fiftieth delivery into safe no-ops. This is the concrete Express implementation of the idempotent webhook consumer pattern; for the broader pipeline it plugs into, see Webhook Processing & Backend State Management.

Trade-offs

The decision is where the idempotency key lives, and the four stores fall on a durability-versus-latency line. An in-process map is fastest and useless for billing; Postgres is the durable authority; Redis is a fast pre-filter; the layered combination gets both. The map places them.

The core decision is where the idempotency key lives. The options differ sharply on durability, latency, and auditability.

For financial events the answer is Postgres as the authority, optionally fronted by Redis. The database survives restarts and cache flushes and gives you a queryable audit trail — both non-negotiable for PCI-DSS and SOC 2. The detailed reasoning lives in Redis vs Postgres for Webhook Idempotency Keys.

Choosing the key: provider event ID versus a derived hash

The store is only half the decision; the other half is what you write into it. The default and correct choice for Stripe-style providers is the event ID they mint — evt_1Mq... — because the provider guarantees it is stable across every retry of the same logical event. That stability is the whole game: two deliveries that carry the same evt_ string are, by contract, the same event, so a unique constraint on that column gives you deduplication for free. Do not derive your own key from the payload body unless you have to. A hash of the JSON is fragile because providers reserialize payloads between retries — a re-ordered key, an added field, or a whitespace change flips the hash and admits a duplicate straight through your check. The one time a derived key is justified is when a provider genuinely does not supply a stable ID, in which case compose it from the immutable business coordinates plus the event type, for example sha256(subscription_id + ':' + invoice_id + ':' + event.type), and never from anything that varies per delivery attempt such as a timestamp or a delivery UUID.

TTL sizing and the retention window

If you front Postgres with Redis, the TTL on the Redis key is not a throwaway constant — it has to exceed the provider’s maximum retry horizon or the pre-filter stops filtering. Stripe, for instance, retries with exponential backoff for up to roughly three days, so a Redis key that expires after one hour leaves a two-and-a-half-day gap in which a retried invoice.payment_succeeded finds no cached claim and falls through to Postgres. That is not a correctness bug, because Postgres still catches it, but it defeats the point of the cache during exactly the window when retries are most likely. Size the Redis TTL to the provider’s documented retry ceiling plus a margin, and keep the Postgres rows far longer — quarters, not days — because they double as the audit log a dispute or a reconciliation job will read months after the fact.

Step-by-Step Implementation

The handler is three steps that must run in order: verify the signature on the raw bytes, claim the key atomically, then process state and outbox in one transaction. Verifying first keeps forgeries out of the store; claiming before processing makes the insert the lock. The flow shows the ordering and the two exit points — reject and duplicate-ack.

1. Verify the signature on the raw body

Use express.raw() so the bytes you HMAC are exactly what the provider signed — parsing first and re-serializing changes the bytes and breaks verification. Compare with a timing-safe equality check.

const express = require('express');
const crypto = require('crypto');
const { Pool } = require('pg');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

app.use('/webhooks/billing', express.raw({ type: 'application/json' }));

function verifySignature(rawBody, signature) {
  const expected = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET)
    .update(rawBody)
    .digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(signature || '');
  return a.length === b.length && crypto.timingSafeEqual(a, b);   // ✅ constant-time
}

2. Define the idempotency key and the store

The provider event ID is the key. Back it with a unique-constrained table that doubles as the audit log.

CREATE TABLE webhook_events (
  id            VARCHAR(255) PRIMARY KEY,           -- provider event id = idempotency key
  type          VARCHAR(64)  NOT NULL,
  status        VARCHAR(16)  NOT NULL DEFAULT 'processing',
  received_at   TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at  TIMESTAMPTZ
);

3. Claim the key atomically and process in one transaction

INSERT ... ON CONFLICT DO NOTHING makes the claim the lock. If the insert affects zero rows the event is a duplicate, so acknowledge 200 and stop. Otherwise process state and write the outbox row in the same transaction.

async function webhookMiddleware(req, res, next) {
  const signature = req.headers['x-webhook-signature'];
  if (!verifySignature(req.body, signature)) {
    return res.status(401).json({ error: 'Invalid signature' });   // ✗ reject forgery
  }

  const event = JSON.parse(req.body);
  const idempotencyKey = event.id;
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const claim = await client.query(
      `INSERT INTO webhook_events (id, type, status)
       VALUES ($1, $2, 'processing')
       ON CONFLICT (id) DO NOTHING`,
      [idempotencyKey, event.type]
    );

    if (claim.rowCount === 0) {                                     // ⚠️ duplicate delivery
      await client.query('ROLLBACK');
      return res.set('X-Webhook-Status', 'duplicate').status(200).json({ received: true });
    }

    req.event = event;
    req.dbClient = client;          // hand the open tx to the route handler
    req.commit = async () => {
      await client.query(
        `UPDATE webhook_events SET status = 'done', processed_at = now() WHERE id = $1`,
        [idempotencyKey]
      );
      await client.query('COMMIT');
    };
    next();
  } catch (err) {
    await client.query('ROLLBACK');
    client.release();
    res.status(500).json({ error: 'Idempotency check failed' });
  }
}

app.post('/webhooks/billing', webhookMiddleware, async (req, res) => {
  const { event, dbClient } = req;
  try {
    // domain mutation + outbox row, all inside the SAME transaction
    await dbClient.query(
      `UPDATE subscriptions SET state = $1 WHERE subscription_id = $2`,
      [mapState(event), event.data.subscription_id]
    );
    await dbClient.query(
      `INSERT INTO billing_outbox (aggregate_id, event_type, payload) VALUES ($1, $2, $3)`,
      [event.data.subscription_id, event.type, event.data]
    );
    await req.commit();                                            // ✅ atomic commit
    res.status(200).json({ processed: true });
  } catch (err) {
    await dbClient.query('ROLLBACK');                              // ✗ key not retained
    res.status(500).json({ error: 'Processing failed' });
  } finally {
    dbClient.release();
  }
});

The critical property: the idempotency key is only durably retained when the business logic commits. If processing throws, the rollback removes the claim so the provider’s retry can succeed cleanly.

Why the claim and the mutation share one connection

The subtle part of the middleware is that it hands the same open client to the route handler through req.dbClient rather than pulling a fresh connection from the pool. This is deliberate and load-bearing. The INSERT ... ON CONFLICT and the UPDATE subscriptions must run inside one transaction on one physical connection, because the row lock the insert takes only protects the subsequent writes if they are part of the same transaction. If the route handler grabbed a second connection, the claim would commit independently, and a crash before the subscription update would leave a retained key with no state change behind it — the worst outcome, because the provider’s retry would now be acknowledged as a duplicate and the mutation would be lost forever. Passing the connection through the request object keeps the claim and the effect atomic, which is the entire reason to write the key inside the transaction rather than in a separate SET NX before it.

Isolation level and the ON CONFLICT race

Under the default READ COMMITTED isolation, two concurrent inserts of the same event ID are serialized by the unique index itself: the first acquires the row, the second blocks until the first commits or rolls back, then re-evaluates the conflict. If the first commits, the second sees the conflict and its rowCount is zero, so it takes the duplicate-ack path. If the first rolls back, the second wins the claim and proceeds — which is exactly why a mid-processing failure must roll back rather than leave a half-written row. You do not need SERIALIZABLE here, and reaching for it would only trade this clean behavior for spurious 40001 serialization failures you then have to retry. The one caveat is that the losing transaction holds a lock while it waits, so keep everything between BEGIN and the ON CONFLICT cheap; do no network calls, no external API round-trips, and no logging that can stall while the lock is held.

Keeping side effects out of the transaction

A frequent mistake is to send the customer’s receipt email or call the tax service from inside the route handler, before req.commit(). Those calls are not transactional — you cannot roll back a sent email — so they belong strictly after the commit or, better, as an outbox row that a separate worker drains once the transaction is durable. That is precisely what the billing_outbox insert buys: the intent to send is written atomically with the state change, and a downstream relay turns it into the actual side effect exactly once. Emitting the effect inline means a rollback leaves you having emailed a customer about a charge your database no longer believes happened.

Verification & Testing

The three tests prove the three properties: a concurrent double-delivery yields exactly one row, a tampered body is rejected, and a mid-processing failure leaves no orphaned claim so the retry succeeds. The panel lists them before the assertions.

Assert exactly-once under concurrency by firing the same event ID through two simultaneous requests and checking the row count:

-- After a concurrent double-delivery, expect exactly one row, status 'done'.
SELECT count(*) FROM webhook_events WHERE id = 'evt_1Mq... ';   -- expect 1
SELECT count(*) FROM billing_outbox WHERE payload->>'event_id' = 'evt_1Mq... '; -- expect 1

In an integration test, post a payload with a tampered body and assert 401 (signature forgery test). Post the same valid event twice sequentially and assert the second returns 200 with X-Webhook-Status: duplicate and that no second outbox row appears. Simulate a processing failure (throw inside the route) and assert the webhook_events row is absent afterward, proving a retry can still proceed.

Driving real concurrency, not simulated concurrency

The concurrency test only proves anything if the two requests genuinely race inside Postgres. Firing them with await one after the other tests nothing — the first has already committed before the second begins. Fire them with Promise.all([post(evt), post(evt)]) against a real database, not a mock, because the behavior you are validating lives in the unique index and the row lock, and a mocked pg client has neither. To make the race reliable rather than occasional, wrap the pool in a small proxy that inserts a short delay between the ON CONFLICT claim and the commit for the first request; that widens the window during which the second request is blocked on the index, so a broken implementation that pulls a second connection or checks Redis before Postgres fails deterministically instead of once in fifty runs. Assert both that exactly one webhook_events row exists and that exactly one billing_outbox row exists — the second assertion catches the subtler bug where deduplication works but a side effect still fired twice.

Property-based and replay testing

Beyond the three canonical cases, a replay harness that captures a day of real provider deliveries and re-runs them in shuffled order — including deliberate duplicates and out-of-order arrivals — surfaces ordering bugs the happy path hides. A customer.subscription.updated that lands before the customer.subscription.created it logically follows should not crash the handler or corrupt subscription_id state; it should either apply cleanly because your mutations are commutative on the fields they touch, or be parked for reprocessing. Property-based tests that generate random interleavings of N deliveries of the same event and assert the final state and outbox row count are invariant are cheap to write with fast-check and catch the class of bug where two code paths both look correct in isolation but compose badly under interleaving.

Gotchas & Production Pitfalls

The pitfalls here are Node/Express-specific traps around the raw body, stuck keys, and pool exhaustion under retry storms. The map groups them so each fix — mount raw, sweep stuck rows, and shed load — is obvious.

• Parsing before verifying: if you let express.json() run first, you HMAC re-serialized bytes that no longer match the provider’s signature and every webhook fails 401. Mount express.raw() on the webhook route only.
• Keys stuck in processing: an unhandled promise rejection between claim and commit can leave a row in processing forever, swallowing the event. Always wrap claim plus logic in one transaction and run a sweep that deletes processing rows older than the provider’s max retry window.
• Trusting Redis TTL alone: provider clock skew or a mid-cycle eviction lets a duplicate slip past a Redis-only check. Keep Postgres as the source of truth and treat Redis as a pre-filter.
• Connection-pool exhaustion under retry storms: a provider outage produces a flood of retries that drains the pool. Add ingress rate-limiting and a circuit breaker so the handler sheds load instead of hanging.
• Deadlocks on high-concurrency upserts: concurrent claims for adjacent keys can deadlock. Retry the transaction with exponential backoff on Postgres error code 40P01 rather than returning 500 to the provider.

The sweep job needs a status the retry can distinguish

Deleting stuck processing rows is necessary, but a naive sweep that deletes anything older than the retry window can race with a transaction that is legitimately slow — a large UPDATE subscriptions under lock contention might sit in processing for tens of seconds without being stuck. Give the sweep a generous threshold well past any healthy processing time, and have it delete rather than flip to done, because a deleted row lets the provider’s next retry re-claim cleanly while a row wrongly marked done would make the handler swallow the retry and lose the event permanently. If your provider’s retry ceiling is three days, sweeping rows still in processing after, say, fifteen minutes is safe: no honest transaction survives that long, and any row that old represents a process that died between claim and commit. Log the id and type of every swept row so a spike in sweeps surfaces as an alert rather than a silent stream of dropped billing events.

Handling body-parser ordering and framework upgrades

The express.raw() trap has a second edge that bites during upgrades: if a global app.use(express.json()) is registered before the webhook route anywhere in the middleware chain, it consumes the stream and req.body arrives at your handler already parsed into an object, so the HMAC runs over Buffer.from('[object Object]') and every signature fails. Mount the raw parser on the exact webhook path and register it before any global JSON parser, or scope the JSON parser to the routes that actually need it. When moving to Express 5 or swapping to Fastify, re-verify this ordering, because the default body-handling changes between frameworks and a silently re-parsed body produces a 100% 401 rate that looks like a rotated secret rather than a parser regression.

Return codes the provider actually reads

Providers treat your HTTP status as the retry signal, so the codes are part of the contract, not cosmetic. Return 200 for both a fresh success and a recognized duplicate, because in both cases the event is safely handled and further retries are wasteful. Reserve 5xx for genuinely transient failures you want retried — a deadlock you could not resolve, a pool timeout, a downstream dependency that was briefly unavailable. Never return 4xx for a duplicate: a 409 or 422 reads to most providers as a permanent rejection, which can disable the webhook endpoint after enough of them and silently sever your billing sync. The 401 on a bad signature is the one deliberate exception, and even then you are betting the forgery is not the provider itself sending with a secret you failed to rotate — so alert on any sustained 401 rate rather than treating it as background noise.

Frequently Asked Questions

Where should the idempotency claim happen? Before any side effect, and durably. Claiming after the work has started leaves a window in which a duplicate delivery can double-apply.

Should the handler return 200 for a duplicate? Yes. A duplicate is a successful outcome from the provider’s perspective, and returning an error invites redelivery that will also be a duplicate.

Does the claim need to be in the same transaction as the effect? Ideally, when both live in the same database. Where they cannot be, claim first and make the effect itself idempotent, so a crash between them is recoverable.

How should handler errors be distinguished from duplicates? Explicitly, in the response and in metrics. A handler that reports duplicates and genuine failures identically hides a rising error rate behind normal-looking traffic.

Related

• Idempotency & Event Deduplication
• Redis vs Postgres for Webhook Idempotency Keys
• Using the Outbox Pattern for Reliable Billing Events
• Syncing Subscription Status Across Microservices