The Job That "Succeeded" for 11 Days While Doing Nothing

Here's a failure mode that doesn't look like a failure: a nightly job that pulls data from an upstream API, writes it to a table, and exits 0 every single time — for 11 days — while doing absolutely nothing useful for 10 of them.

The upstream API rotated an auth token. The job's client library handled the rotation gracefully in one specific sense: it didn't throw. Instead of a 401, the now-unauthenticated requests started getting back a 200 with an empty array — a perfectly valid, perfectly parseable, perfectly empty response. The job looped over zero records, found nothing to insert, and exited cleanly. Green checkmark. Every night.

Why "it ran" was the wrong question

Whatever was watching this job was watching for the right kind of thing in the wrong place. Exit code 0 was the entire signal. Everything about that signal was technically true — the job did run, it did complete, it didn't error. The thing exit code 0 can't tell you is whether the job accomplished anything, because "accomplished anything" isn't a concept that exists at the process level. It's a concept that exists at the level of your actual data.

The job never errored, because from its own point of view, nothing ever went wrong. It asked a question, got an answer, and the answer happened to be "nothing." That's not a failure. That's a query result.

This is what makes silent failures so much more dangerous than loud ones: a crash gets escalated immediately because something is actively broken and demanding attention. An empty-but-successful run doesn't demand anything. It just quietly doesn't do its job, and keeps not doing its job, until someone downstream notices the data is stale — which in this case took 11 days, because the table still had 10 days of history sitting in it and nobody was looking closely enough to notice the row count had stopped moving.

What would have caught it on day one

Not better error handling — the job never errored, so there was nothing to handle. What actually would have caught this is checking the shape of success, not just its existence: did this run write rows, not just "did it exit cleanly." A one-line assertion — fail loudly if a run that's supposed to produce ~500-2,000 records produces zero — turns an 11-day silent gap into a same-night page.

The more general version of that rule: for any recurring job, "succeeded" and "did something" are two different claims, and most monitoring only checks the first one. If your job could technically report success while doing nothing at all — an empty response treated as valid, a loop that runs zero times, a write that silently no-ops — that's the exact gap where an 11-day outage hides in plain sight, wearing a green checkmark the entire time.