Why we delete your job output after 30 days

When a scheduled job calls POST /ping/<token>/fail, it can attach its own output — up to 10 KB of whatever it wrote to stderr on the way down. That is the single most useful thing a heartbeat can carry. It is also the only place in okokumo where a customer's own text lands in our database, and it is usually the text you would least like to leave lying around: a stack trace with your source paths in it, an internal hostname, a connection string that found its way into an error message.

We keep it for 30 days. Check results — our own measurements, the latency and status code and the error our probe recorded — we keep for 90. This post is about why those two numbers are different, and about the part almost nobody publishes: how far past each one a record can actually survive.

The number nobody prints

Every monitoring product tells you its retention window. The windows are easy to compare and they are all, in a narrow sense, true.

None of them tell you the error bar, and every one of these systems has one. Nothing deletes rows at the instant they turn 90 days old. Something runs on a schedule, or in batches, or drops storage a block at a time, and the real answer is always "the window, plus however long the mechanism takes to get round to it". A retention number quoted without that second part is a marketing figure: it is the number the mechanism aims at, not the number it hits.

Ours are on the security page, and the precise bounds are written into the DPA:

• Check results: 90 days, and a result may survive up to eight days past its 90th.
• Captured job output: 30 days, and it may survive up to one day past its 30th.

Both are floors. Nothing is deleted early. The overshoot is the honest part.

Why the two error bars are different sizes

Because two different mechanisms enforce them, and the mechanism is what sets the bar.

Check results live in a TimescaleDB hypertable, and Timescale deletes by dropping a whole chunk of storage once every row inside it is past the window. The chunk interval on check_results is seven days. So a result that turns 91 days old sits there until its chunk's youngest row is also past 90 — which can be another week. Then add the retention job's own schedule: it is a background job that looks once a day, so a chunk that becomes eligible a minute after a run waits nearly another full day. Seven plus one is eight.

Captured job output is not deleted that way. It is nulled out of the rows by a purge loop in our own code:

// failureBodyPurgeTick is how often the loop looks for bodies past the
// window. Daily is ample against a 30-day cutoff — a day's lateness is
// invisible.
failureBodyPurgeTick = 24 * time.Hour

One mechanism, one pass a day, one day of slack. Same product, same database, different bar — which is precisely why quoting one number for both would have been the wrong thing to do.

The reason the output gets nulled rather than the row deleted is worth a sentence, because it is the same distinction the whole post turns on. "This check went down on 3 March and recovered 40 minutes later" is a small, genuinely useful row, and it is what your incident list and your status page are computed from. The 10 KB of stderr stapled to it is the part with the exposure in it. So the row survives its full window and the body does not. The incident record outlives the output that explains it, on purpose.

Why 30 and not 90

The body answers one question: why did this run fail? That question gets asked while the incident is live, and sometimes in the week after during a post-mortem. Nobody debugs last quarter's cron failure from its stderr.

90 days is the floor for check results because something reads them that far back — a Tower status page renders 90 one-day cells per component, and shortening the window would blank the left-hand end of every published page. Nothing renders a 90-day-old failure body. Holding your log output for three months to serve a question asked within one week is storage we pay for and exposure you carry, with nothing on the other side of it.

That is the whole argument. It is not a privacy feature we built to have one. It is that the data has a shorter useful life than the row it is attached to, so it gets a shorter window.

We found out our own promise was false, twice

This is the part that makes the post worth writing, and neither half is flattering.

The first one was the code. The failure body is written to two places in one transaction — check_results.error and state_transitions.failure_body — because the alert dispatcher runs asynchronously and renders your notification from the transition without racing the results table. The purge cleared one of them. The other copy fell through to the 90-day Timescale policy instead.

So the identical text — same stack trace, same hostnames — was being deleted from one table at 30 days and from the other at up to 98, while three documents said 30. It was found in review, before launch. The fix was to extend the purge so both copies expire on the same clock, read once from the database so they cannot drift apart. What we did not do was soften the sentence in the DPA to match the code. That option was on the table, it was one line, and it was the wrong one: the number was right, the mechanism was wrong, and you fix the mechanism.

The second one was this very page's arithmetic. The first version of the DPA clause said a check result could outlive its 90th day by up to seven days. That reasons about the chunk width and stops there — it forgets that the chunk is only dropped when the retention job next looks, and that job runs daily. The real bound is eight. A reviewer caught it by reading both numbers off a live database rather than off our own comments:

select schedule_interval, config->>'drop_after'
  from timescaledb_information.jobs where proc_name='policy_retention'
-> 1 day | 90 days

Seven instead of eight is a small error and it flattered us, which is exactly why it needed correcting: it is a contractual document promising a tighter ceiling than the mechanism delivers. The clause now names both causes — the block width and the daily pass — so you can do the arithmetic yourself instead of trusting our total.

Measured, not asserted

Both windows are checked against the running database rather than against our intentions. /healthz/deep reports each one as its own component — this is production, right now:

"retention": { "ok": true,
  "detail": "drop_after=90 days want=90d enabled last_success=0.1d ago" },
"failure_body_purge": { "ok": true,
  "detail": "no captured body older than 30d" }

The enabled and last_success parts are not padding. A retention policy can sit in the catalog with the correct 90-day window and be switched off — alter_job(id, scheduled => false) leaves the row looking perfect while the job never runs again. An existence check passes against that. So the health check asserts the policy ran, and measures staleness against the policy's own schedule rather than a hard-coded day, and the purge component goes red if any captured body older than 30 days is still readable anywhere.

That is the same reasoning as the canary we break every morning: a promise that nothing checks is a promise you find out about from a customer.

What this is not

It is not a competitive claim. Several competitors keep check results far longer than 90 days — that is a real advantage if you need year-over-year latency history, and we do not offer it. Our windows are the same on every plan, free included; we do not sell retention as an upgrade tier. And none of this touches erasure: a request under Article 17 deletes your organization's data immediately and in full, ahead of any window.

The claim is narrower and it is the only one we can actually back: we publish the error bar because we measured it, and when the measurement disagreed with the document, the document changed.

---

okokumo is infrastructure monitoring hosted in France — HTTP, heartbeat, TLS and domain checks, with alerts confirmed across EU regions before they reach you. Retention in full.