Retries multiply across layers, so budget them instead of counting

The idea

Most services retry with a fixed count: try up to three times, back off, give up. Each team picks that number locally, and locally it is reasonable. The trouble is that retries add load exactly when a dependency has the least capacity to absorb it.

In the worst case this produces a metastable failure: a brief slowdown triggers retries, the retries keep the dependency overloaded, and the system stays down after the original cause is gone.

A retry budget changes the question from "how many times may this request retry" to "what share of my traffic may be retries". The client tracks retries against first attempts over a sliding window. While retries stay under a small fraction of requests, it retries normally. Past that fraction, it stops retrying and fails fast. A common implementation is a token bucket where each successful request deposits a fraction of a token and each retry spends a whole one.

An example

Take a chain: gateway, then an orders service, then an inventory service, then a database. Every layer retries up to three times, so up to four attempts each.

• The database slows down and inventory queries start timing out.
• Inventory makes four attempts per query.
• Orders sees inventory fail and makes four attempts, each causing four database attempts: sixteen.
• The gateway does the same: sixty-four database queries for one user click.
• Users see a spinner and refresh, multiplying it again.

The database needed less load and received far more. With a budget at each layer, retries during the incident are bounded by a small share of normal traffic. Once failures exceed it, every layer fails fast, the database drains its queue, and recovery starts on its own.

Where it breaks

Budgets are a ceiling, not a design. Several failure modes survive them.

• Low traffic clients have a noisy ratio. Without a small floor, one failure can drain the bucket and block the next legitimate retry.
• Retrying the wrong errors still wastes budget. A validation error will fail every time, and a timeout on a non-idempotent write may already have succeeded on the server.
• Server-side queues keep working on requests whose callers already gave up. The budget limits new attempts but does nothing about wasted work without deadline propagation and load shedding.
• When one bad host is the problem, the fix is retrying elsewhere or hedging, and a budget tuned for fleet-wide overload can be too stingy.
• Silent fail-fast hides trouble. If the budget trips without a metric, the dashboard shows clean errors and nobody sees the dependency drowning.

In practice

I would start by deciding where retries live in the chain, and I prefer exactly one layer, usually the one closest to the failing dependency. Everything above it passes errors through.

• Put a retry budget in the shared client library, not in each call site, so the policy cannot drift between teams.
• Keep a per-request cap as well. The budget limits the aggregate, the cap limits any single request.
• Retry only errors known to be transient, and only operations that are idempotent or carry an idempotency key.
• Use exponential backoff with jitter so retries do not arrive in synchronized waves.
• Propagate a deadline with each request and stop retrying once the remaining time cannot cover another attempt.
• Emit a metric when the budget is exhausted and alert on it. It is one of the earliest honest signals that a dependency is in trouble.
• Test it: inject latency into a dependency in staging and confirm load at the bottom of the chain stays flat.

Retry in one layer, cap retries as a share of real traffic, and treat an exhausted budget as an alert rather than a quiet fallback.