A canary for alerting: breaking our own API every morning

2026-08-31

engineeringalerting

Every system has failure modes its own dashboards can show you. A monitoring product has one they cannot: silence. If the alerting pipeline breaks, the symptom is that nothing happens — which is indistinguishable from a week where nothing needed to happen. Uptime graphs stay green. Error rates stay flat. The product looks healthy precisely because it has stopped doing its job.

We learned this the way most people do.

The alert that reached nobody

On 2026-08-11, a TLS check in one of our beta testers' organizations went down and no email arrived. The check was for a certificate about to expire. The timeline, read out of production rows rather than reconstructed from memory:

04:36:28Z — the fr-par probe records a failure.
04:41:53Z — nl-ams confirms it, and the check transitions up → down.
04:41:54Z — the dispatcher claims the transition.
Nothing else.

Every part of that is correct behaviour. Multi-region confirmation worked exactly as designed: one region observing a failure is not enough, a second region agreed five minutes later, and only then did the state change. The state machine wrote the transition. The dispatcher picked it up within a second.

Then it looked up where to send the alert, and found nowhere to send it. The organization's only email channel had a validated_at of 2026-08-17 — created on 29 July, confirmed six days after the outage it should have reported. Our channel lookup filters on validated_at IS NOT NULL, deliberately: we will not send alerts to an address nobody has confirmed, because that is how a monitoring tool becomes a way to mail strangers. So the query returned zero rows, and dispatch ended in a bare return nil.

No email. No log line. And alerted_at was set on the transition, so it could never be retried — the row looked, forever, exactly like one that had been delivered.

Every component was healthy. Every test passed. The probes were right, the confirmation logic was right, the filter was right. The outcome was wrong and nothing recorded it. We found out six days later because a human asked why he hadn't been emailed.

The detail that makes this a class of bug rather than one bug: successful deliveries logged nothing either. So even with the logs in front of you, "sent" and "swallowed" were the same absence of a line. That is fixed now — there is a row per delivery attempt, including the skips, with the reason — but fixing the record does not fix the underlying problem, which is that we had no way to know the pipeline still worked.

Why health checks cannot see this

We had two layers of health checking at the time, and both were green all morning.

GET /healthz answers whether the process is alive. GET /healthz/deep verifies the monitoring loop from the inside: is the scheduler still claiming due checks, is the queue draining, has each probe region reported recently, is the dispatcher keeping up with transitions. It is a genuinely useful endpoint and it would have returned {"status":"ok"} throughout, because every one of those components was fine.

Outside that, a third-party monitor polls us from infrastructure we don't own, which closes a gap nothing internal can: if the box is gone or the network path to it breaks, no code of ours is in a position to notice. It, too, would have been green.

The gap all of them share is the same one. They check that the parts work. They do not check that the product's output arrives. For a monitoring product the output is the alert, and nothing we had exercised it end to end.

/healthz/deep does assert that now — but only because the canary below gives it something to assert about, which is the last section of this post rather than this one.

A scheduled failure

So we added an endpoint whose entire job is to break on a timetable:

// InCanaryWindow reports whether t falls in the daily failure window.
func InCanaryWindow(t time.Time) bool {
	utc := t.UTC()
	return utc.Hour() == canaryHour && utc.Minute() < canaryWindowMinutes
}

// Canary returns 500 during the daily window and 200 otherwise.
func (h *Handler) Canary(c echo.Context) error {
	if InCanaryWindow(h.now()) {
		return c.JSON(http.StatusInternalServerError, map[string]string{
			"status": "canary window — scheduled failure",
		})
	}
	return c.JSON(http.StatusOK, map[string]string{"status": "ok"})
}

That is the whole thing. A check in our own organization points at it, named "Canary Test (will fail once a day)" so nobody mistakes it for an incident. Every morning it goes down and recovers, and a real alert has to travel the entire path — scheduler, probe, state machine, dispatcher, channel — to arrive in an inbox.

The window is converted to UTC inside the predicate rather than taken from the host clock's zone, because a box configured for Paris would otherwise move the failure by an hour, or by two for half the year.

The inversion is the point. Instead of asking "is anything broken", which silence answers wrongly, we arrange for something to break on purpose and ask "did the alert arrive". The absence of an event becomes the signal.

Why not just a test button

We have one of those. Every alert channel can send itself a test message, and it is the right tool for "did I paste the webhook URL correctly".

It would not have caught this. A test send proves the code path you invoke from a UI, with the arguments that UI hands it. It does not exercise a real transition row, or deduplication, or the alerted_at claim, or the validated_at filter — which is to say it skips precisely the parts that swallowed the real alert. Passing a test send while dropping every genuine alert is not a hypothetical state; it is exactly the state we were in for six days.

A canary exercises the path production uses, with production's configuration, on production's schedule, at an hour when nobody is watching. Which is when the original failure happened.

The details that turned out to matter

A fixed window, not a random one. You can only notice a missing event if you know when to expect it. Predictability is the feature; a canary that fires "sometime daily" cannot be distinguished from one that has stopped.

Ten minutes, not one. A check needs two consecutive failures before it changes state — the default, and the reason one network blip does not page anyone. So the failure window has to be wide enough for the check to land inside it at least twice: interval times threshold, plus margin, because scheduling carries jitter on purpose. A one-minute window gets sampled once or not at all and would never reliably produce a down state. Ten minutes leaves room at the intervals a dogfood check plausibly runs on. A canary that only sometimes fires teaches you to ignore it, which is worse than not having one.

No special casing. It is served by the same router and exposed through the same reverse-proxy path list as /healthz, so the canary rides the code under test rather than a private lane beside it. canary is also a reserved status-page slug, so no customer can claim the name.

The clock is a dependency, and pretending otherwise cost us the test. The first version of this endpoint read time.Now() directly, and so did its test — which meant the test asserted whichever branch the clock happened to be in. For 23 hours and 50 minutes a day it exercised only the 200 path. An inverted comparison, the one bug that would silently disarm the whole mechanism, would have shipped green.

The clock is now injected, and the window is asserted at both edges: 07:59:59 up, 08:00:00 down, 08:19:59 down, 08:20:00 up, plus two Paris-local timestamps to prove the UTC conversion. Inverting the comparison now fails the suite, and so does dropping the .UTC() — both checked by making the mutations and watching red appear, because a test you have never seen fail is a test you are guessing about.

Correction: the window was too narrow (2026-09-08)

As first published this said ten minutes, justified as interval × threshold — a five-minute check needing two consecutive failures. That arithmetic was wrong, and it took six weeks and a degraded health check to notice.

It assumes each execution samples both regions. It does not. The two regions consume one shared queue and alternate, so an execution yields a single result from a single region, and a ten-minute window holds roughly two executions in total — about one per region. Going down needs both the consecutive-failure threshold and two distinct failing regions, and at two executions neither has any margin.

On 2026-09-08 both in-window failures drew nl-ams, fr-par had a healthy sample inside the confirmation lookback, and the check never transitioned. No alert, no delivery, and twenty-five hours later the canary component went degraded — which is the mechanism in the next section doing exactly its job, on a day when alerting was working perfectly. Reading back further, the day before had fired only by luck: fr-par was silent for half an hour, so the single-region escape hatch confirmed on the failure count alone. Three consecutive days, three different mechanisms, one of them absent.

The window is twenty minutes now, sized against the per-region cadence rather than the check's own interval, and the dogfood check runs every minute instead of every five. The uncomfortable part is that the original reasoning looked right, was written down, and was never checked against a real window's rows — which is the same failure this post opens by describing, one level up.

Who watches the canary

A canary converts a silent failure into a missing event. But a missing event is still silence unless something is expecting it — so for a while the expectation was a human one: no canary email over coffee means something is wrong with alerting itself. Better than nothing, and clearly worse than a check.

The obvious next move is to point the canary's alert at a heartbeat URL, so a missing alert makes a second check go down. We didn't build that, for a reason worth stating: it needs a webhook channel and a heartbeat check created by hand in the right organization, and it ends with a down check whose own alert has to travel the pipeline we are trying to verify. More moving parts, same recursion.

What /healthz/deep carries instead is a canary component that goes degraded when either:

no alert for the canary check has been delivered in over 25 hours, or
no enabled check points at /canary at all.

The second condition matters more than it looks. Deleting, renaming or disabling that check silences the daily alert exactly the way a broken pipeline does, and a watchdog that can be quietly removed is not a watchdog. The check is found by what it points at rather than by a configured id, so there is no second setting to get wrong and no way to be half-configured.

Only delivered rows count toward the first condition. A fresh skipped or failed row is evidence of the opposite — it is precisely what the 2026-08-11 incident would have written, had it written anything — so neither can mask a pipeline that is dropping every alert.

That endpoint is where the recursion terminates, because it is already polled from outside our infrastructure by a service that alerts through a channel we do not run. An alert about alerting cannot be trusted to arrive through the alerting you are asking about; handing the signal to somebody else's pager is the only version of this that holds. What stays ours is that monitor's own configuration — an account, a keyword rule, a phone number — which cannot live in our repository and still be independent of it.

What it cost

Two handler functions, a pure predicate, one health component with two SQL queries, one check in one organization, and one email a day you learn to expect. What it buys is the difference between believing alerting works and being able to point at this morning and say it did.

One last thing, in keeping with the subject. Writing the component's tests, four of them passed on the first run and I nearly moved on. They were passing because a uuid scan error made the component report degraded — the right verdict for entirely the wrong reason, and indistinguishable from the outside. The healthy case was the one that couldn't fake it, and it failed, which is how the bug surfaced. Those tests now assert why the canary is unhealthy rather than only that it is. Verification that can only ever confirm what you expected is the thing this whole post is about.

While writing the first draft I checked, at 08:00:13 UTC: /canary answering 500 on both environments, right on schedule.