The Alert That Recovers Itself Is the One That Gets You

A payment-sync job at a mid-size e-commerce shop had a monitor on it that fired 11 times over the course of one week. Eleven pages. Eleven red alerts in the on-call channel. And 11 times, by the time anyone actually opened the alert, it had already gone green again on its own. Nobody investigated any of them, because there was nothing left to investigate — the dashboard said the job was fine now, and "fine now" felt like the end of the story.

On the 12th firing, it didn't recover. The job had actually died — genuinely, not transiently — and it sat dead for six hours before a customer support ticket about missing order confirmations forced someone to go look. Six hours, on a monitor that had cried wolf eleven times that same week, each time training the team a little more thoroughly to not bother checking.

A flapping alert is a training program, and it's training you wrong

Every monitoring setup is implicitly teaching the humans who watch it how much attention each alert deserves. A page that's real every single time it fires teaches "drop what you're doing." A page that's fine again by the time you look teaches "this one clears itself, check later" — and "check later" quietly becomes "don't bother checking," because nobody has infinite attention and an alert with an eleven-for-eleven track record of self-resolving has, by simple pattern-matching, earned a lower priority than everything else competing for the same on-call brain.

The problem isn't that the alert fired 11 times. It's that firing 11 times and recovering 11 times is functionally identical, from the page, to a monitor that's broken — and the fix for a broken monitor is to start ignoring it.

This is alert fatigue's quieter cousin. The classic version is "too many alerts, all noise, all ignored." The flapping version is worse because it doesn't look like noise — each individual alert looks real, gets a real color, has a real timestamp — right up until the one time it isn't a false alarm, and by then the team's calibration has already adjusted to treat that alert type as low-stakes.

"Recovered" is not one thing

The deeper issue is that a monitoring dashboard usually collapses two very different situations into the same green checkmark. One is: the job failed, something noticed and fixed the underlying cause (a dependency came back, a retry succeeded against a real recovery, someone's automation kicked in), and the job is now actually healthy. The other is: the job failed, kept getting reattempted, and one of those reattempts happened to land inside the check window before a timeout — so the monitor reports "recovered" while the actual root cause was never addressed and is sitting there, ready to fire again the moment conditions drift half a percent.

Those two are indistinguishable on a dashboard that only tracks current status. They are not indistinguishable in terms of risk. A job that recovered because someone fixed something is done. A job that recovered because it outlasted a timeout window is a coin flip that happened to land on the good side, and coin flips don't stay good forever — they regress to the mean, usually at the worst possible time, like six hours before a support queue fills up with angry customers.

What to actually check on a flapping alert

Before dismissing a self-resolved alert as "handled," it's worth running through a short set of questions — ideally before the third occurrence, not the twelfth:

• Has this specific alert fired three or more times in the last week? A monitor with a real flapping pattern behind it should get treated as an open incident, not eleven separate closed ones.
• Did it recover because something was fixed, or because it just outlasted a retry window or timeout? If nobody can answer that question, assume the second one — it's the one that fails you.
• Did a human actually look at the underlying cause, or did the green checkmark end the conversation before it started? "It's green now" is a status, not an investigation.

None of this requires new tooling so much as a different rule: treat repeated flapping on the same check as its own incident, with its own ticket, independent of whatever color the dashboard happens to be showing at the moment someone glances at it. A monitor that fires and clears three times in a week isn't three resolved problems — it's one unresolved problem that hasn't finished happening yet.