alerting

4 posts tagged alerting. All posts

Verifying an okokumo webhook — the signature, the retries, and the duplicate you'll eventually get

2026-09-28

Checking an HMAC signature takes three lines, and most receivers still get it wrong by checking it against re-serialised JSON. How to verify okokumo's X-Sentinel-Signature against the raw body in Node, Python and Go, which responses we retry and which we don't, and why your receiver has to cope with the same alert arriving twice.

guidealerting

Pause the alerts, not the check

2026-09-04

Disabling a check during planned work stops the probing too, which leaves a hole in the uptime history exactly where the interesting hour was. How okokumo suppresses paging without suppressing measurement — and what has to happen when the window closes and the thing is still down.

engineeringalerting

A canary for alerting: breaking our own API every morning

2026-08-31

A monitoring product's worst failure is silence, because a broken alerting pipeline looks exactly like a quiet week. Here is the alert of ours that reached nobody, why health checks could not see it, and the endpoint that breaks on purpose every morning so that silence becomes something a pager outside our own infrastructure can see.

engineeringalerting

Multi-region confirmation without false positives

2026-08-13

A monitoring tool that cries wolf gets muted, and a muted tool is worse than none. How okokumo requires two probe regions to agree before it pages you — and why it deliberately stops requiring that when a probe dies.

engineeringalerting