Pause the alerts, not the check

2026-09-04

engineeringalerting

You are migrating a database at 2am. You know the service will refuse connections for a few minutes somewhere in the middle, and you know your monitoring will notice, so you do the obvious thing: you disable the check. Run the migration. Turn it back on. Nothing pages, nobody wakes up, and the next morning the uptime report says 99.98%.

That number is wrong, and not in a way anyone will catch.

A disabled check doesn't record downtime. It doesn't record uptime either — it records nothing. The hour you spent deliberately breaking things produced zero samples, so it carries zero weight in every average computed afterwards. And the hole is never in a random place. It is always exactly where you changed something, which is the first hour you'd want to look at when latency is worse next week and you're trying to remember when it started.

The second problem is the one that actually bites: you forget to turn it back on. A check that isn't running looks precisely like a check that has nothing to report — silence and health are the same shape — and you find out weeks later.

So okokumo has two different verbs. Disabling a check stops probing. A maintenance window keeps probing and stops paging.

One nullable column

The entire storage model is migration 0025:

ALTER TABLE checks ADD COLUMN maintenance_until TIMESTAMPTZ NULL;

CREATE INDEX idx_checks_maintenance_ending ON checks (maintenance_until)
    WHERE maintenance_until IS NOT NULL;

No maintenance_windows table, no start time, no recurrence rules. The only question the alerting path ever asks is "is this check in a window right now", and one timestamp answers it:

// inMaintenance reports whether a check is inside a window right now. A nil or
// past timestamp means no — which is why a window needs nothing to end it.
func inMaintenance(check db.Check) bool {
	return check.MaintenanceUntil != nil && check.MaintenanceUntil.After(time.Now().UTC())
}

The comment is the design. A window needs nothing to end it. There is no expiry job, no scheduler entry, no cleanup task that can fall over and leave a check silently muted forever. A window that lapses while the process is restarting is still lapsed, because "lapsed" is a comparison against the clock rather than a state somebody has to write.

It also settles a question that a windows table leaves open. Your two-hour job is running long, so you click "2 hours" again. Does that mean four hours total, or two hours from now? A single column can only express the second one, which is also what anyone re-arming a window actually means:

until := time.Now().UTC().Add(time.Duration(req.Hours * float64(time.Hour)))

The tests are named after the two things that would make this feature a lie if they broke: TestStartMaintenanceMeasuresFromNow and TestStartMaintenanceLeavesTheCheckEnabled.

Suppression belongs in the dispatcher

The probe scheduler knows nothing about maintenance. Neither does the state machine that decides a check is down. Both keep working exactly as they do on a normal Tuesday, which is the whole point — the uptime history for that hour is real, measured data, including the part where your own migration took the service down for ninety seconds.

Suppression happens one layer later, in the alert dispatcher, which already has the check row loaded because it needs it to build the notification:

// In a maintenance window: the transition is still recorded — the history
// should show what actually happened during the change — but nobody is
// paged for it.
if inMaintenance(check) {
	reason := "in a maintenance window until " + check.MaintenanceUntil.Format(time.RFC3339)
	d.record(tr.OrganizationID, check.ID, &tr.ID, nil, db.DeliverySkipped, reason)
	return nil
}

Two alternatives were available here and both are worse.

Don't write the transition at all. Then the incident history is blank during the change, and the check page can't tell you that the service went down twice while you were working on it. That's the same hole as disabling the check, moved one table over.

Write it but leave it unclaimed. The transition would sit in the queue and fire the instant the window closed. A page at 04:00 about something that broke at 02:15 and recovered at 02:16 is worse than no page: it's an alert that has already stopped being true by the time it's read.

So the transition is recorded and marked as handled, and the skip is written to the alert-delivery log with the reason. Suppressed on purpose is still an alert that reached nobody, and every alert that reaches nobody should be able to say why.

The hole this digs, and the sweep that fills it

Here's the part that makes maintenance windows more than a mute button.

A check goes down during your window. The alert is suppressed, correctly. Your migration finishes, you go to bed — and the service is still refusing connections, because you broke something you haven't noticed yet.

The window lapses. Now nothing happens. There is no new transition to alert on: the check went up -> down once, during the window, and it has been down ever since. Nothing changed, so nothing fires, so nobody is ever told. Left there, a maintenance window would be a way to hide an outage indefinitely — you'd have muted the alert and the retry at the same time.

The feature has to answer that itself, and it does, on every dispatcher tick:

err := d.DB.WithContext(ctx).Raw(`
	UPDATE checks SET maintenance_until = NULL
	WHERE id IN (
		SELECT id FROM checks
		WHERE maintenance_until IS NOT NULL AND maintenance_until <= now()
		FOR UPDATE SKIP LOCKED
	)
	RETURNING *`).Scan(&ended).Error

Every window that has run out gets cleared, and the rows come back. Anything still down among them produces an alert: still down now that the maintenance window has ended.

The single most important thing in that query is that it clears the column in the same statement that returns the row. The sweep only ever sees a lapsed window once, because by the time anything else looks, maintenance_until is already NULL. A second API instance running the same sweep in the same second gets an empty result rather than sending a duplicate. Without that, each tick would re-find the same lapsed window and re-page every thirty seconds until someone fixed the outage — repeat-until-acknowledged is a legitimate design, but it should be a decision, not an accident of a WHERE clause.

That alert is sent inline rather than through the transition queue, for the same reason the pause notification is: there is no transition row behind it, because nothing changed state. That's the entire problem it exists to solve.

Three tests, which are really three sentences about what a window means:

Test	What it pins down
`TestDispatcherSuppressesAlertsDuringMaintenance`	No page, but the transition is recorded and marked
`TestDispatcherReportsACheckStillDownWhenTheWindowEnds`	Still broken afterwards pages — exactly once, then never again
`TestDispatcherStaysQuietWhenTheWindowEndsHealthy`	Recovered before the window ended, so nothing to say

The second one ticks twice on purpose and asserts the delivery count is still 1.

What your visitors see

If the check feeds a public status page, there's a second decision to make, and it goes the other way from pausing.

A paused check shows up as "unknown". Whether a customer chose to stop monitoring something is between them and their monitoring — it isn't a statement they're making to their visitors, and a paused check produces no results anyway, so the data can't distinguish it from a probe that has gone quiet.

A declared maintenance window is the opposite. It is a statement, deliberately made, to exactly those visitors. Someone watching a component wobble during planned work should be told it's planned — that is most of what a status page is for. So the component reads Under maintenance.

What it must never do is launder a real failure into a planned one:

switch {
// A component under planned maintenance doesn't make the page read "major
// outage" — that's the point of declaring the window. It also can't hide
// one: any *other* component being down still wins.
case down > 0:
	return OverallMajor
case degraded > 0:
	return OverallDegraded
case operational > 0:
	return OverallOperational
case maintenance > 0:
	return OverallMaintenance
}

Maintenance is last. The page-level banner only reads "Under maintenance" when every component on it is in a window; one component in maintenance and another genuinely down still reads "Major outage" (TestRollupMaintenanceNeverMasksADownComponent). In the severity ordering that reconciles the banner with hand-written incidents, maintenance sits below operational: it can describe a page with nothing else to report, and it can never make a page look worse than it is.

The automated incident drafter follows from that without extra code. It reads component state from the same function the public payload uses, so a component under maintenance produces no draft incident — automation can't disagree with the page it's writing about.

What this deliberately doesn't do

A window is per check. There's no "maintenance mode" for a whole environment, so a ten-service deploy means arming ten windows, or using the API to do it. Whether that becomes a group operation depends on whether anyone actually asks.

There are no recurring windows. A nightly batch job that always makes a service unreachable between 03:00 and 03:10 is not really planned maintenance, it's a service whose check should be configured for it — a cron check, a longer failure threshold, or a probe pointed somewhere more honest.

A single window is capped at 30 days, which is long enough for any real migration and short enough that a forgotten window can't mute a check for a year. Scheduling one is restricted to owners and admins: it's the one setting that makes a check stop talking, and it shouldn't be a one-click action for anyone who happens to be in the org.

And windows are always relative to now. You can't arm one for next Tuesday at 23:00. That's the honest cost of the single-column model — I'd rather add a schedule when someone needs it than carry a windows table for a feature nobody asked for.

okokumo is infrastructure monitoring hosted in France — HTTP, heartbeat, TLS and domain checks, with alerts confirmed across EU regions before they reach you.