The status page has gone green for most things. One slow service keeps returning errors, and every client that depends on it keeps asking again. That is a retry storm: failed requests are retried so eagerly that the retries become the load, and the dependency cannot recover while they keep arriving. GitHub's own report on its August 17 outage describes one, and it is worth reading with your own services open next to it.
Recovery was slower because everyone asked again
The GitHub report says the outage lasted 7 hours and 47 minutes. It began when traffic reached a new peak and a critical infrastructure component failed to scale with it. GitHub is plain that neither of its two August incidents came from a code or configuration change. Both were capacity failures.
The retries come in during recovery. Most services were back earlier that day, but some Copilot services took longer, and errors in those services triggered a client-side retry loop that increased traffic. GitHub says it had to mitigate that behaviour before it could safely restore traffic.
So the obvious reading, that the system ran out of room, is true and incomplete. Capacity failed first. The retries made the way back harder. The report does not say which clients retried or how the loop was built, and this post will not guess. What it does say is what GitHub changed: consistent retry limits, retry budgets and variable timeouts across service-to-service calls, to prevent retry storms and cascading load.
Why backoff alone does not stop a retry storm
Most advice on this topic stops at exponential backoff and jitter. Backoff makes each client wait longer between attempts. Jitter keeps a thousand clients from waking up on the same second. Both are good, and both are about when one client asks again.
Neither says how much asking is too much. A fleet of clients that each back off politely can still send a steady stream of retries at a service that has no spare capacity, because every individual client is behaving correctly. Google's SRE book describes a cascading failure as one that grows over time through positive feedback, and notes that a server answering late can miss its clients' deadlines, so the work is wasted and the clients retry, adding more overload.
Then there are layers. Say a browser, a gateway and a service each try a call three times. The service at the bottom can see up to 27 attempts for a single request, and not one layer did anything unreasonable by its own rules. Backoff decides when each client asks again. A budget decides whether the whole fleet should.
Give retries a budget and one owner
A retry budget is a cap on how much of your traffic is allowed to be retries. While the dependency is healthy, nobody notices it. Once the cap is spent, a failed call stays failed: the client gives up locally and returns the error, without sending anything.
Google's SRE book has a close cousin of this in client-side throttling. Each client keeps the last two minutes of how many requests it attempted and how many the backend accepted. When attempts reach twice the accepted count, new requests are rejected at the client with a rising probability, so they never reach the network. It is a published, working example of a client protecting a backend from its own persistence.
The second half is ownership. Decide which layer retries a given call and make every other layer pass the error up. The owner should be the layer that knows whether the operation is safe to repeat, and it should read one policy, not a hand-written loop. GitHub's wording is worth noticing here: consistent limits, across service-to-service interactions.
Retry only what is safe to repeat
A retry is a second attempt at something that may already have happened. Reading a record twice costs nothing. Charging a card twice, or creating an order twice, costs a customer. A call that must be retried needs an idempotency key, which is an identifier the receiving service uses to recognise a repeat and return the first result instead of doing the work again.
Two limits are worth stating. Retries help with transient failures, a dropped connection or a brief overload, and do nothing for a dependency that is genuinely down. And a budget caps the proportion of retries, not the absolute volume, so if your normal traffic is itself climbing, the retries climb with it.
When a retry is not safe or the budget is spent, fail fast and say why. We build document workflows with the same instinct. In Stay Funded 360, a missing receipt blocks the download and names the record, instead of letting an incomplete packet through. A visible stop is easier to fix than a quiet loop.
A one-page retry check for your own services
Sit with one service and the dependency it calls most. Go through these in order:
List every layer that retries a call to that dependency, including SDK defaults, the proxy and the load balancer.
Write down each layer's attempts, delays and timeout.
Pick one layer to own retries and set the others to pass errors up.
Set a budget, and decide what the caller does once it is spent.
Mark which calls are safe to repeat, and add idempotency keys to the ones that must be.
Make the dependency return errors in a test environment and watch how many requests leave your service.
The last step is where most teams find the surprise. The number they expected is rarely the number they see.
Back at that green status page with one slow part: the clients now get a small, fixed allowance of retries and then go quiet, and the dependency gets room to recover. If your own workflows chain several tools that each retry on their own, our services page shows how we map them, and we can walk through yours in a 30-min call.
Frequently asked questions
What is a retry storm?
A retry storm is when clients retry failed requests so often that the retries add enough load to keep the failing service down. The original fault may be gone, but the extra traffic stops the recovery.
How do you avoid a retry storm?
Limit retries with a budget, let one layer own them, and use backoff with jitter inside that limit. Also test how many requests your service sends while its dependency is returning errors.
What is a retry budget?
It is a cap on the share of traffic that may be retries. When the cap is reached, further failed calls return an error immediately instead of being sent again.
Does jitter fix retry storms?
Not by itself. Jitter spreads retries out in time so clients do not hit the service together, but it does not reduce how many retries are sent.
Are retries safe for payments and writes?
Only with an idempotency key, so the receiving service can tell a repeat from a new request. Without one, a retried write can happen twice.
