ZeonTools
What is the Thundering Herd Problem?
Imagine your backend API goes down for exactly 2 seconds due to a database lock. During those 2 seconds, a 3rd-party service (like Stripe or a custom microservice) attempts to send you 5,000 webhook payloads. Because your server is down, all 5,000 webhooks fail.
If the 3rd-party service uses a standard naive retry logic (e.g., "Wait exactly 5 seconds, then retry"), all 5,000 webhooks will wait exactly 5 seconds, and then smash your newly-recovered API at the exact same millisecond. This massive spike in traffic instantly crashes your server again. This phenomenon is known as the Thundering Herd Problem.
Exponential Backoff is Not Enough
Exponential Backoff solves part of the problem by increasing the wait time exponentially between failures (e.g., 1s, 2s, 4s, 8s). This prevents the server from being hammered continuously over a long outage. However, it does not solve the Thundering Herd. All 5,000 requests from our previous example will still retry simultaneously at the 1s mark, then simultaneously at the 2s mark.
To fix this, System Architects introduce Jitter. Jitter is a randomized variance applied to the calculated wait time. This scatters the retries across a wide time window, completely eliminating the traffic spike and allowing your database connections to recover peacefully.
AWS Standard Jitter Formulas Explained
The AWS Architecture blog established three standard mathematical formulas for applying Jitter to backoff algorithms:
- Full Jitter:
random(0, wait_time). This spreads the load perfectly across the available window. However, it means some retries might happen almost immediately (0ms), which could fail if the server needs a few seconds to boot. - Equal Jitter:
(wait_time / 2) + random(0, wait_time / 2). This is often the safest bet. It guarantees a minimum wait time (at least 50% of the backoff) while still spreading the load across the remaining 50% window. - Decorrelated Jitter:
min(cap, random(base, prev_wait * 3)). A highly complex algorithm where the current wait time is based on the previous wait time, dramatically smoothing out the curve over time and preventing large "clumps" of retries.
CRITICAL: Idempotency Keys
You cannot safely retry POST requests without implementing Idempotency. Imagine a payment processing webhook. The webhook hits your server, your server charges the customer's credit card, but right as your server tries to return a `200 OK`, a network timeout occurs. The sender assumes the webhook failed and retries.
If you blindly process the retry, you will charge the customer twice. Idempotency Keys solve this. The sender includes a unique header (e.g., Idempotency-Key: req_12345). Your database logs this key. When the retry comes in, you check the database. If req_12345 is already marked as `Completed`, you immediately return `200 OK` without running the payment logic again.
Circuit Breakers vs Retries
Retries and Circuit Breakers are complementary but distinct patterns. A Retry handles transient failures (a temporary network blip or a brief DB lock). A Circuit Breaker handles catastrophic failures.
If a downstream microservice is completely offline, sending 8 retries will just tie up threads and memory on your server. A Circuit Breaker monitors failures. If X failures occur within Y seconds, the circuit "trips" (opens). Once open, all requests immediately fail without waiting or retrying, protecting your server's resources. After a timeout, it allows a "half-open" test request to see if the service is back online.
Dead Letter Queues (DLQ)
What happens when the 8th and final retry fails? You must not drop the payload into the void. Enterprise systems route exhausted messages to a Dead Letter Queue (DLQ).
Using AWS SQS, Apache Kafka, or RabbitMQ, the failed JSON payload is stored safely in a DLQ. Engineers can then set up alarms on the DLQ, investigate why the payload failed (e.g., a bad schema or persistent outage), fix the issue, and manually replay the messages from the DLQ.
Case Studies: Stripe & Twilio
- Stripe: Stripe implements a highly robust exponential backoff. They retry webhooks for up to 3 days. They utilize Idempotency Keys extensively and strongly recommend their users implement equal jitter when calling the Stripe API.
- Twilio: Twilio provides a "Fallback URL" mechanism. If the primary webhook fails, they immediately hit the fallback. If that fails, they implement an exponential backoff curve, giving your primary servers time to restart.