Livelock vs. Crash: Why "It's Still Running" Doesn't Mean It's Working

Crashes are the easy failure mode. Something throws, the process exits, your process manager notices, your monitoring notices, you get paged. It's unpleasant, but it's legible — there's a clear moment where "working" became "not working," and usually a stack trace pointing at why.

Livelock doesn't give you any of that. The process is still running. CPU usage might even look normal. Nothing crashed, nothing logged an error, nothing exited. It's just stuck — spinning through a loop that never makes progress, waiting on a lock it'll never get, or bouncing between two states that each look fine in isolation but never resolve into an actual result.

The dashboard can't tell the difference

Picture an agent process that pulls a task, starts working on it, and reports back when done. Now picture that same agent stuck in a retry loop against a dependency that's degraded but not fully down — every attempt technically completes, gets a response, and immediately triggers another attempt because the response doesn't satisfy whatever condition it's checking for. The process is alive. It's making requests. Your uptime check, if it's just asking "is this process running" or "does this port respond," sees a perfectly healthy service.

A crash announces itself. A livelock has to be caught doing nothing, which is a much harder thing to notice than catching it doing something wrong.

This is the same shape of problem as a cron job that logs "started" and never checks whether it finished — the presence of activity gets mistaken for the presence of progress. A livelocked agent isn't idle. It's busy. It's just busy in a circle.

What actually catches it

The fix isn't a smarter uptime check — it's checking a different thing entirely. Instead of "is the process alive," you want "has this process produced a result recently." That's a fundamentally different signal: a heartbeat check-in that only fires on genuine completion, not on every loop iteration. A livelocked process still loops. It stops checking in.

The same principle applies at the level of individual runs, not just the whole process: track duration, not just outcome. A job that normally finishes in 40 seconds and is now sitting at 20 minutes with no output yet is a strong signal on its own, independent of whether it ever technically errors out. If you're only watching for a failure event, and the loop is specifically the kind that never produces one, you'll wait a very long time for an alert that isn't coming.

The uncomfortable part is that this failure mode is invisible by construction — it exists precisely in the gap between "the process crashed" and "the process finished," which most monitoring setups don't watch at all. Worth asking about your own setup: if a process got stuck in a loop tomorrow and never crashed, never errored, and never technically timed out — would anything tell you?