The Monitoring Bug Hiding in Your Check Interval

Most of the time, a job's duration and your check interval have nothing to do with each other — the job finishes well within the gap between checks, and everything behaves the way you'd expect. The bug only shows up on the runs where that stops being true: when a job occasionally takes longer than the interval between checks, and a "still running" status from one check gets carried into the next one as if it were still fresh information.

How a dead job inherits a healthy status

Say a job normally takes 90 seconds and gets checked every 5 minutes. Most checks land cleanly: the previous run finished ages ago, the current state is unambiguous. Now say one particular run hangs — maybe a downstream dependency stalls — and the check that fires while it's still in flight sees a legitimate "in progress" state and reports it as such, correctly, because the job genuinely was in progress at that moment.

The problem isn't that one check reported "in progress." It's that the process then died silently, and nothing re-evaluated the status until the next scheduled check — which, depending on how state gets carried forward, can end up looking at a stale "in progress" flag and treating it as still current, rather than noticing the process behind it is gone.

Depending on exactly how the monitoring is wired, this can persist for more than one cycle. Each check that reads the same stale "in progress" marker without independently verifying the process is still alive just re-confirms a status that was accurate once, at a moment that's now several cycles in the past. The job is dead. The monitor keeps reporting the last thing it knew, because nothing forced it to re-derive the current truth from scratch.

What closes the gap

The fix is to make "in progress" a claim with an expiration, not a persistent state. If a job is supposed to take 90 seconds and a check finds it "in progress" at the 6-minute mark, that's not still a valid "in progress" — it's a hung or dead job wearing an old status. Comparing the reported start time against an expected maximum duration, on every check, turns a stale status into an active failure signal instead of a silently-inherited "it's fine."

The more general lesson: any monitoring design that carries forward a status rather than re-deriving it fresh each time has a version of this bug lurking somewhere, and it's specifically invisible during normal operation — it only shows up on the runs that overlap your check interval, which by definition are the outliers, which is exactly when you most need the monitor to be right.