The cron job that stopped running

Every scheduled job you run has two failure modes, and your monitoring probably only sees one of them.

The first is loud. The job runs, throws, exits non-zero, and cron mails the output to whoever still reads root's mailbox — or your error tracker gets a report, or the CI step goes red. Unpleasant, but visible.

The second is quiet. The crontab entry didn't survive the server migration. The container's scheduler sidecar isn't in the new compose file. The OOM reaper took the worker at 02:58 and systemd didn't restart it. The job doesn't fail; it never starts. No exit code, no exception, no log line — because nothing ran to produce one.

That second failure mode is indistinguishable from a quiet, successful night. You find out when someone asks why the invoices didn't go out, and the answer is "since Tuesday, apparently".

Inverting the check

An HTTP check works because we can call your service. A job can't be called: it's a process that exists for ninety seconds a day inside your network, and nothing outside can poll it.

So the check runs the other way. okokumo gives each heartbeat or cron check a unique ping URL, and your job calls it after a successful run. We don't measure the job; we measure the arrival of its pings. If the ping doesn't turn up when it should, the check goes down and alerts fire through the same path as any other incident.

The absence of a signal becomes the signal.

The ping endpoint is deliberately dumb: the token in the URL is the whole credential, no session or API key involved, and both GET and POST are accepted so a one-line curl works anywhere. An unknown token gets a flat 404 — we don't confirm which tokens exist.

The crontab line

0 3 * * * /usr/local/bin/nightly-backup.sh && curl -fsS https://app.okokumo.com/ping/<your-token>

Two details in that line matter more than they look.

&&, not ;. The ping is chained to the job's success. A backup that exits 1 must not report healthy — a job that fails and pings anyway is worse than no monitoring, because now you have evidence you were fine.

The ping is at the end, not the top. Ping after the work is committed. Pinging on start tells you the scheduler fired, which is a real thing to know, but on its own it certifies a process that may have died halfway through.

If you want both, the same token accepts a start ping:

curl -fsS $PING/start && ./nightly-backup.sh && curl -fsS $PING

The completion ping subtracts the start time and records the run's duration, so the check's history shows how long each run took — which is the data you'll want when you're deciding what the grace window should be.

What "overdue" actually means

For a check with a plain period, the deadline is arithmetic on one row:

last ping + period + grace < now  →  down

Two consequences worth internalising. A check that has never pinged is measured from its creation time, so a heartbeat you set up and then forget to wire into the job goes down on its own rather than sitting green forever. And the clock runs from the last ping, not from the scheduled time — which is why the grace window has to absorb your job's own duration, not just its lateness.

There's no consecutive-failure threshold here and no second region to confirm with. An HTTP check needs two failing executions from two regions before it transitions; a heartbeat has exactly one observer, your own job, so silence past the deadline is the transition. Multi-region confirmation falls back to single-observer behaviour for precisely this case. That makes the grace window the only tuning knob you have — and the only thing standing between a slow backup and a 3am page.

Sizing the grace window

Start from the run you'd be annoyed about, not the average one.

Take the longest normal run in the last month. Add the scheduler's own slop — a busy host, a queue that was backed up, a lock held by yesterday's run. Grace should cover that sum; the period covers the rest.

A concrete shape for a nightly backup that usually finishes in 40 minutes but has taken 70 on a heavy day:

• Period: 24 hours.
• Grace: 2 hours.

That pages you at about 05:00 if the 03:00 run never started or is still grinding an hour past its worst known duration. Tight enough to catch a missing Tuesday inside the same morning; loose enough that a slow month-end run doesn't wake anyone.

The failure mode of a too-generous grace is subtler than the failure mode of a too-tight one. Twelve hours of grace on a daily job means a job that died Monday night is reported Tuesday lunchtime — still useful, still far too late to re-run it before anyone noticed. Grace is bounded at 24 hours in the API, which is already past the point of diminishing returns for most daily jobs.

What happens when a run overruns. Nothing dramatic: at period + grace the check flips to down, alerts go out, and when the job finally finishes and pings, the check flips straight back to up — the transition reason is recorded on both sides, so the incident history shows a real overrun rather than a mystery blip. That's the signal you want if it happens twice. If it happens every month-end, your grace is sized to the average run rather than the bad one.

When a period is the wrong shape

0 3 * * 1-5 is 24 hours from Monday to Tuesday and 72 hours from Friday to Monday. No single period describes that. Set 24 and it pages every Saturday; set 72 and a job that stopped on Tuesday goes unnoticed until Friday.

So a cron check can take the crontab expression itself plus the IANA timezone your server runs it in, and the deadline becomes the next scheduled run plus grace, recomputed after every ping. A weekday-only job stays quiet all weekend, and a skipped Tuesday is caught inside Tuesday's own window. The incident reason names the run that was missed, in your schedule's own timezone, rather than just reporting silence.

You can check what an expression actually fires with the cron expression tester before you rely on it — day-of-month and day-of-week are OR'd, not AND'd, which surprises everyone exactly once.

Where to start

Pick the one job whose silent failure would cost you the most — backups, invoicing, the nightly export a customer reads — and give it a heartbeat this afternoon. One curl at the end of the command, a period, a grace window sized off its worst real run. Five minutes of work against the class of outage nobody notices.

More on the mechanics on the cron job monitoring page, and on how the scheduler works if you like knowing what's underneath.

---

okokumo is infrastructure monitoring hosted in France — HTTP, heartbeat, TLS and domain checks, probed from two EU regions. Heartbeat and cron checks are available on every plan, including Free.