Multi-region confirmation without false positives

The fastest way to make a monitoring product worthless is to page someone at 3am for something that was already fine by 3:01. Do it twice and they add a filter. Do it three times and they stop reading the alerts, which means the next page — the real one — goes unread too.

So the interesting problem in monitoring isn't detecting failure. A single HTTP request detects failure. The problem is deciding that what you saw was real.

Two gates, not one

okokumo puts a failing check through two independent gates before it will call anything down.

The first is consecutive failures. A check has a failure_threshold, default 2, and a single failed probe never changes state on its own. This catches the shortest class of false positive: a dropped packet, a mid-deploy blip, one unlucky TCP handshake.

The second gate is regional agreement, and it's the one that matters. Every probe result carries the region it came from — fr-par or nl-ams today. Before a check flips to down, we ask whether the failure is visible from more than one place:

SELECT
    COUNT(DISTINCT probe_region) FILTER (WHERE NOT success) AS failing_regions,
    COUNT(DISTINCT probe_region)                            AS active_regions
FROM check_results
WHERE check_id = $1 AND time > now() - interval '15 minutes'

Two numbers: how many regions are currently seeing this thing fail, and how many regions are reporting at all. If two or more regions are failing, it's real. If only one of two is failing, it isn't — that's a network path problem between one datacentre and your server, not your server being down.

Your users in Berlin don't care that Paris briefly couldn't reach you. So we don't page you about it.

The part where we deliberately weaken the rule

Here's the decision I find most interesting, and the one I'd argue about with someone who hadn't thought it through.

What should happen when the second probe is itself down?

If you require two regions to agree, full stop, then a dead probe box means every check in the system silently stops alerting. No failure can ever be confirmed, because confirmation is impossible. Your monitoring goes quiet, and quiet looks exactly like healthy.

That's a far worse failure than a false positive. A false positive costs someone an annoying notification. A silent monitoring system costs them the outage they bought the tool to catch.

So the rule has an escape hatch:

if counts.ActiveRegions <= 1 {
    return true, nil
}
return counts.FailingRegions >= 2, nil

With only one region reporting, the consecutive-failure count decides alone. We fall back to being a single-region monitor — noisier, but awake. The same branch covers heartbeat checks, where the "probe" is your own cron job calling us and there is no second region by definition.

Failing open here is a deliberate trade, not an oversight. When the safety mechanism can't function, the system reverts to the less safe behaviour rather than to no behaviour.

Why fifteen minutes, and not "simultaneously"

The query has a 15-minute window, which deserves an honest explanation rather than a hand-wave.

Both probe regions consume from one shared Redis queue. That means they alternate executions rather than both checking every target at the same instant — whichever worker is free takes the next job. A check on a 60-second interval gets hit from Paris, then Amsterdam, then Paris, not from both at once.

So "confirmed by two regions" can't mean "both failed simultaneously", because they're never scheduled simultaneously. It means both regions saw a failure recently enough to be talking about the same event. Fifteen minutes is wide enough to span alternating executions at any supported interval, and narrow enough that yesterday's incident doesn't confirm today's blip.

It also self-heals: a region that went quiet half an hour ago drops out of the window, active_regions falls to one, and the system slides into single-region mode without anyone configuring anything.

Four cases, four tests

The whole behaviour is four scenarios, and each one is a test:

Situation	Confirmed?
One region active, threshold met	Yes — fall back to count alone
Two regions active, one failing	No — regional blip
Two regions active, both failing	Yes — real outage
Second region last seen 30 min ago	Yes — stale region drops out of the window

The second row is the one that earns its keep. It's also the one that's invisible when it works: nobody notices the page they didn't get.

What this doesn't solve

Two regions in the EU is not the same as global coverage. If your service is broken only for users in São Paulo, okokumo will not tell you, and I'd rather say that plainly than imply otherwise. Two independent regions catch the common case — one network path going bad — and that's the case that generates most false positives.

More regions is a straightforward extension: the query counts distinct regions, so a third probe changes a deployment, not a line of logic. The threshold of two stays sensible as that number grows.

---

okokumo is infrastructure monitoring hosted in France — HTTP, heartbeat, TLS and domain checks, with alerts confirmed across EU regions before they reach you. The scheduler behind these probes is described here.