3 Ways Your Green Dashboard Is Lying to You

A dashboard full of green checkmarks feels like evidence. It isn't, necessarily — it's evidence that a specific, narrow question got a specific, narrow "yes." The gap between that narrow question and the broader thing you actually care about is where outages hide. Three of the most common versions of that gap:

1. The health check pings the load balancer, not the app

A huge number of "is my service up" checks are really "does something respond on this port." If that something is a load balancer or reverse proxy sitting in front of your actual application, it can keep answering "yes" long after the application behind it has stopped doing anything useful — hung on a database connection, stuck initializing, crash-looping in a way the proxy doesn't notice. The proxy isn't lying. It's answering the question it was asked, which is a different question than "is my application working."

2. Cron logs "started," never checks it finished

A common cron setup logs a line the moment a job kicks off, and that's the line something downstream greps for to confirm "the job ran today." The job starting and the job finishing are two different events, and a lot of monitoring only ever observes the first one. A job that starts and then hangs, times out, or gets OOM-killed partway through still produced that "started" log line — the exact same evidence a genuinely successful run would have left behind.

3. The agent reports in every run, even the ones that fail silently

An agent or worker process that pings a status channel on every execution looks reassuring — constant activity, regular messages, nothing looks quiet. But "I ran" and "I ran correctly" are different claims, and a check-in that fires unconditionally on every attempt will keep firing right through a run that returned malformed output, an empty result set, or a wrong answer that happens not to throw. The activity is real. It just isn't evidence of correctness.

• A health check that hits your actual application logic — not just "is something listening" — catches failure mode #1.
• A check-in that fires on completion, with an expected duration window, catches #2 — a hung job stops checking in instead of leaving a stale "started" line behind.
• Validating the shape of the output, not just the fact that something ran, catches #3 — did it produce the rows/records/result you'd expect, not just "did it exit without an exception."

None of these are exotic failure modes. They're the default shape of the simplest, cheapest health check you can write — which is exactly why they're so common. The fix in each case is the same instinct: ask what specific claim your check is actually verifying, and whether that claim is the one you actually care about, or just the easiest one to measure.