Designing a check scheduler

Every monitoring product has the same core loop: some set of checks need to run on their own schedule — every 30 seconds, every 5 minutes, every 24 hours — and something has to decide, continuously, which ones are due right now. It sounds trivial until you have to run it at scale without either missing a check or running it twice.

The naive version, and why it doesn't scale

The obvious first draft is: loop over every check, compare its last-run time to its interval, fire the ones that are due. That works fine at ten checks. At ten thousand, you're doing a full table scan every tick, most of which is wasted work — the vast majority of checks aren't due on any given second.

The fix is to stop asking "which checks are due" from scratch every time, and instead let each check carry its own answer: a next_run_at timestamp. The scheduler's only job becomes: find rows where next_run_at <= now(). That's an indexed query, not a scan.

The double-fire problem

Once you're not doing a full scan, a new problem shows up. If the scheduler ever runs more than one instance — for redundancy, or just because a tick overlaps a slow one — two processes can both read the same "due" row before either has marked it as claimed. Now the same check fires twice.

The fix has to happen in the same statement that reads the row, not after it. In Postgres that's FOR UPDATE SKIP LOCKED:

UPDATE checks
SET next_run_at = now() + (interval_seconds * interval '1 second')
WHERE id IN (
    SELECT id FROM checks
    WHERE enabled AND next_run_at <= now()
    FOR UPDATE SKIP LOCKED
)
RETURNING *

SKIP LOCKED means a second scheduler instance hitting the same due row just skips it instead of blocking — it's already being claimed by someone else. And because the next_run_at advance happens in the same UPDATE that claims the row, there's no window between "read" and "mark as done" for a second instance to sneak in. One statement, one claim, no race.

Jitter, or: don't let checks created together fire together

If ten checks get created in the same minute with a 60-second interval, naively they'll all land on the same next_run_at forever — every tick after that is a small burst instead of steady load. The fix is cheap: spread each check's next run by a small random fraction of its own interval, applied in the same UPDATE:

next_run_at = now()
    + (interval_seconds * interval '1 second')
    + (interval_seconds * 0.10 * (random() * 2 - 1) * interval '1 second')

±10% of the interval is enough to break up synchronized bursts without meaningfully affecting how "due" actually behaves from a user's perspective.

Scheduling is not execution

The scheduler's only output is a job on a queue — it doesn't make the HTTP request, check the certificate, or ping the heartbeat URL itself. That work happens in a separate probe worker, reading off Redis. Keeping those two concerns apart means the scheduler can tick every second and stay cheap (an indexed row claim, not a network call), while the actual check execution — which can be slow, can time out, can hit a flaky network — never blocks the next tick.

What this bought us

A scheduler that ticks every second, claims only the rows that are actually due, never double-fires even with multiple instances or an overlapping tick, and spreads load instead of bursting it — all from one UPDATE ... RETURNING statement per tick. No cron-style external scheduler, no distributed lock service, no separate "is this claimed" table.