Your Heartbeat Grace Period Is a Decision Nobody Made

Every heartbeat monitor has a grace period: how long after a check in was expected before the monitor decides something is wrong. It's usually one field in a setup form, filled in once with whatever number felt reasonable at the time, and never looked at again. That number quietly decides two things at once: how often you get woken up for nothing, and how long a real failure gets to hide.

Most teams never notice they made that decision. They just live with whichever side of it they landed on.

The job that breaks every default

Take an hourly job that starts at the top of every hour and pings the monitor when it finishes. On a quiet hour it's done in 5 minutes. On a busy one, when the upstream export is large, it takes 40. Both are normal. Neither is a problem.

Now look at it from the monitor's side. The monitor doesn't know when the job started. It only knows when the last ping arrived, and it expects the next one about 60 minutes later. If the 1:00 run was a fast one and pinged at 1:05, and the 2:00 run was a slow one and pinged at 2:40, the gap between pings was 95 minutes. The monitor was told to expect 60.

The spread between your fastest and slowest normal run gets added straight onto the interval the monitor sees. A job with 35 minutes of natural variance needs at least 35 minutes of grace just to stay quiet on a perfectly healthy night.

Option one: tight grace, nightly false alarms

Say you set the grace period to 10 minutes, because 10 minutes feels responsive. Now every time a slow run follows a fast one, the monitor fires. On this job that might be two or three times a night, always between midnight and 4am when the batch exports are biggest, and always resolving on its own a few minutes later when the ping finally lands.

The first week, someone checks every one of those alerts. By the second week they check the first one of the night and ignore the rest. By the third week the monitor is muted, or routed to a channel nobody reads, or the grace period gets bumped to something enormous in a hurry at 2am. The tight setting didn't make the monitoring more sensitive. It made it get switched off.

Option two: loose grace, a two hour blind spot

So you go the other way and set the grace period to 60 minutes, or just call it 2 hours to be safe. The false alarms stop completely. Everything is green and quiet.

Then one night the 1:00 run dies on startup. Bad config push, a missing environment variable, a container that never gets scheduled. It doesn't matter which. The last ping was at 12:10. The monitor expects the next one around 1:10 and then waits out its grace period. With 60 minutes of grace, nobody hears about it until around 2:10. With 2 hours, it's past 3.

And that's the good case, where the job crashed immediately. If it hung instead, sitting there holding a lock and doing nothing, the next run at 2:00 might also fail to start because the lock is still held. Two missed runs, a backlog building, and the monitor still showing green because it's patiently waiting out a number someone picked to avoid being paged.

A loose grace period doesn't reduce failures. It just moves them from "you find out now" to "you find out after it's already cost you two hours of data."

Why there's no right number

The trap is that both of these are the same field. One number is being asked to answer two different questions:

• Did the job start when it was supposed to?
• Did the job finish within a reasonable amount of time?

The first question has a tight answer. The job is scheduled for the top of the hour, so if it hasn't started by 2 or 3 minutes past, something is wrong. The second question has a loose answer, because run time genuinely varies from 5 to 40 minutes. A single grace period measured from the last completion ping has to be loose enough for the second question, which makes it far too loose for the first.

Getting out of the tradeoff

The fix is to stop asking one number to do two jobs. Ping twice: once when the run starts, once when it finishes. Then each signal gets its own timing.

• Start ping, tight window. Expected every 60 minutes, grace of 3 to 5 minutes. Scheduling is predictable even when the work isn't, so this can be strict without generating noise. A run that never starts is caught within minutes.
• Finish ping, measured from the start. Instead of "60 minutes after the last finish," the rule becomes "no more than N minutes after this run's own start." Now the variance between runs doesn't stack up, because each run is judged on its own clock.

For N, don't use the average run time. Use something near the slowest normal run you've actually observed, plus a margin. If the last month of runs topped out at 40 minutes, a 50 minute ceiling stays quiet on every healthy night and still catches a hung run before the next one is due to start. That's the property that matters most: a hang gets reported before it can block the following run, instead of after.

Compare the two setups on the 1am crash. With a single completion ping and 2 hours of grace, you hear about it after 3. With a start ping and a 5 minute window, you hear about it at 1:05. Same job, same failure, and the new setup also produces fewer false alarms than the 10 minute version did, because the slow runs that used to trip it are now well inside their own 50 minute ceiling.

Check the number you already have

If you have heartbeat monitors running today, it's worth pulling up the grace periods and asking where each one came from. Look for two patterns. Grace periods that are round numbers like 1 hour or 24 hours on jobs whose timing nobody measured are usually the loose kind, picked to make alerts stop. Monitors that are muted, or whose alerts land somewhere nobody reads, are usually the tight kind, abandoned after one noisy week.

Either way, the fix starts with the same step: look at how long the job actually takes across a few weeks of real runs, including the slow ones, and separate "did it start" from "did it finish." Once those are two signals instead of one, the grace period stops being a compromise and becomes two small, boring numbers that are each obviously right.