How One Flaky API Call Turned Into 14,412 Silent Retries

Retry logic is supposed to make a system more resilient. Without a hard cap, it can just as easily turn one bad response into an unbounded amount of quiet, expensive noise. This is the shape of that failure: an agent's call to a third-party API started returning a degraded-but-technically-valid response late on a Friday. By the time anyone looked again on Monday, the retry logic had fired 14,412 times against that same call.

Backoff isn't the same thing as a limit

The retry logic had exponential backoff, which is the part that made this look fine from the outside: request volume didn't spike, alarms tuned to detect a burst of traffic never tripped, nothing about the pattern looked like an incident. Backoff was doing exactly its job — spacing requests out so a struggling dependency doesn't get hammered. What it wasn't doing was ever stopping. Each retry waited a little longer than the last, which meant the failure got quieter and slower over time instead of louder, which is precisely the opposite of what you want from a signal you're relying on to get noticed.

A retry loop with backoff but no cap doesn't fail loudly and it doesn't fail quickly. It fails at a rate specifically engineered to stay under everyone's attention threshold.

Over roughly 60 hours, that's a retry landing every few minutes on average by the end — not fast enough to look like a spike, not slow enough to stop before the API usage bill turned it into a five-figure line item for a task that, correctly configured, should have failed once, logged clearly, and stopped.

What actually should have happened

Three separate things, any one of which would have prevented this on its own:

A hard retry ceiling — not "keep backing off forever," but "after N attempts, stop and surface this as a failure." Exponential backoff without a maximum attempt count isn't resilience, it's a failure mode wearing resilience's clothes.

A cumulative-cost or cumulative-attempt alert, independent of per-request volume — something that asks "how many times has this specific operation retried today," not just "how many requests per minute are we sending." The second question never noticed anything wrong. The first one would have flagged it within the first few hundred attempts.

And a monitor watching the actual task, not the retry mechanism underneath it — an agent run that's supposed to complete in seconds and is instead still "in progress," retrying, three days later, is a clear signal on its own if anything is tracking run duration against an expected ceiling.

The pattern worth remembering: retry logic that's tuned to avoid looking like an incident is very good at not looking like an incident, right up until someone opens the bill.