ARCHETYPE // Design DecisionIncident Report• 2026-07-20 • 8 min read

How a Stripe retry storm taught me to never process webhooks synchronously

An incident report: what broke, why it compounded, and the idempotency pipeline we built to prevent it from happening again.

At 14:23 UTC on a billing cycle renewal day, PingPanda's webhook processor stalled. Stripe's retry emitter sent 47,000 events in 9 minutes. Our synchronous database handler held PostgreSQL connections open while waiting for Discord API responses. The pool hit its 20-connection ceiling in 6 seconds. Here is exactly what happened and what we replaced it with.

We had built the webhook ingestion handler the obvious way: verify signature, parse payload, write to database, call Discord API, return 200 OK. That sequence is correct in development. Under at-least-once retry pressure from Stripe, it is a deadlock machine.

System Topology Specification — ASCII Flow

Before vs. After: Synchronous Deadlock vs. Idempotent Decoupled Pipeline

The fix was not a performance optimization. It was a structural separation between ingress acknowledgment and processing execution.

BEFORE (The Deadlock Machine):
[Stripe Ingress]
    |
    v
[Verify HMAC] --> [Write to DB] --> [Call Discord API] --> [Return 200 OK]
                        |                   |
               (DB conn held open)   (avg 180ms, p99 380ms)
                        |
             (Pool exhausted at 20 conns in 6s)
                        |
             (Stripe sees timeout -> enqueues retry)
                        |
             (EXPONENTIAL STORM)

AFTER (Idempotent Decoupled Pipeline):
[Stripe Ingress]
    |
    v
[Verify HMAC] --> [Check Idempotency Table] --> (Duplicate? Return 200 OK immediately)
    |                      |
    |              (New event? Continue)
    |                      v
    |         [Enqueue to Upstash Redis Queue] --> [Return 202 Accepted in <8ms]
    |
    v (Async Worker - decoupled from ingress SLA)
[Dequeue] --> [DB Transaction Commit] --> [Discord API Call] --> [Mark Processed]

At 14:23 UTC on a billing cycle renewal day, PingPanda's error rate crossed 18%. Not gradually — immediately. Within 6 seconds of the first database slowdown, our PostgreSQL connection pool was fully saturated and every incoming webhook was timing out.

The root cause was not a bug. It was an architecture. We had written our webhook handler the intuitive way: verify the Stripe HMAC-SHA256 signature, parse the JSON payload, insert the event into PostgreSQL, call the Discord API with the notification, and return a 200 OK. Under normal traffic, this worked fine. Under a billing cycle renewal spike, it was a deadlock machine waiting to be triggered.

Here is what actually happened: The Discord API p99 latency on our busiest channel was 380ms on that day — likely elevated by Discord's own load from other notification services. Each of our database connections was held open for the full 380ms while the HTTP response came back. When Stripe's webhook emitter received no 200 OK within 5 seconds, it enqueued a retry. That retry compounded the pool load. Within 6 seconds, all 20 PostgreSQL connections were suspended mid-transaction.

We lost 340 events. Not because we didn't have error handling — we did. We lost them because our catch block was parsing the request body at the point of failure, and by then the body stream had already been consumed by the JSON parser crash. Lesson: log the raw `req.body` as a Buffer before touching it.

The fix was not a performance optimization. It was a structural reclassification of what 'processing a webhook' means. Ingress acknowledgment and event execution are two separate concerns. They cannot share the same HTTP response lifecycle.

We rebuilt the pipeline in three steps. First, we extracted idempotency checking to its own table: `INSERT INTO webhook_events (stripe_event_id, received_at) VALUES ($1, NOW()) ON CONFLICT (stripe_event_id) DO NOTHING RETURNING id`. If the insert returns no rows, the event is a retry we have already seen — return 200 OK immediately without touching the queue.

Second, new events are pushed to Upstash Redis with a 24-hour TTL and a 200 Accepted is returned in under 8ms. The ingress handler now has zero knowledge of Discord, zero knowledge of PostgreSQL write latency, and zero exposure to third-party API variability.

Third, an async worker dequeues events from Redis, commits the database transaction, and calls the Discord API. If the Discord call fails, the message goes to a dead-letter queue with an alert. We can replay any event from the last 7 days without Stripe needing to re-emit it.

After the rebuild, we replayed the original incident payload against the new pipeline: 47,000 events in 9 minutes, zero 504s, zero connection pool exhaustion, 4,200 duplicate suppression hits caught by the idempotency table, zero duplicate Discord notifications.

The thing I would change if I were starting over: I would have designed the ingress handler as a pure 'receive and enqueue' function from day one. The temptation to do 'just one quick database write' inside the HTTP request handler is how every synchronous webhook deadlock begins.

All Engineering Notes Explore Case Studies

Initializing System StateSUB-100MS EDGE SLA // INGRESS ROUTING

BEFORE (The Deadlock Machine): [Stripe Ingress] | v [Verify HMAC] --> [Write to DB] --> [Call Discord API] --> [Return 200 OK] | | (DB conn held open) (avg 180ms, p99 380ms) | (Pool exhausted at 20 conns in 6s) | (Stripe sees timeout -> enqueues retry) | (EXPONENTIAL STORM) AFTER (Idempotent Decoupled Pipeline): [Stripe Ingress] | v [Verify HMAC] --> [Check Idempotency Table] --> (Duplicate? Return 200 OK immediately) | | | (New event? Continue) | v | [Enqueue to Upstash Redis Queue] --> [Return 202 Accepted in <8ms] | v (Async Worker - decoupled from ingress SLA) [Dequeue] --> [DB Transaction Commit] --> [Discord API Call] --> [Mark Processed]