Custom domains, on-demand TLS, and the endpoint that says no

Customers on Business can serve their okokumo status page from their own hostname — status.theircompany.com rather than a slug on ours. They add one CNAME, press Verify, and the page is live over HTTPS a few seconds later on a certificate we obtained for them.

The feature is four lines of Caddy config. The design is one HTTP endpoint, and all of it lives in what that endpoint refuses.

The four lines that make it dangerous

Caddy calls this on-demand TLS: instead of listing every hostname up front, you serve a catch-all and obtain certificates on first contact.

{
	on_demand_tls {
		ask http://localhost:8080/v1/tower/tls-check
	}
}

:443 {
	tls {
		on_demand
	}
	# …routing to the status page
}

Delete the ask line and that config still works. It also becomes a machine that orders a Let's Encrypt certificate for any hostname a stranger points at our box. No account needed — just a DNS record.

The damage isn't the stray certificates, it's the rate limit. Let's Encrypt caps issuance per account, and an account spending its budget on hostnames belonging to nobody can't issue for the customer who signed up this morning. One stranger with a wildcard record breaks issuance for everyone.

The endpoint that says no

Before ordering for a hostname it hasn't seen, Caddy GETs the ask URL with ?domain=<name> and proceeds only on a 2xx. Ours is fail-closed in every direction: unknown host, unverified host, missing parameter, database error, or a caller that isn't loopback all return 404.

The interesting refusal is the last one.

// RemoteAddr, NOT c.RealIP(). Echo's RealIP trusts X-Forwarded-For and
// X-Real-IP before falling back to the peer address, and no IPExtractor is
// configured — so RealIP here would be spoofable with a header, turning
// this into a public "is this domain registered with okokumo" oracle. The
// TCP peer can't be forged.
if !isLoopbackAddr(c.Request().RemoteAddr) {
	return echo.NewHTTPError(http.StatusNotFound, "not found")
}

This endpoint answers a question worth something to an attacker: is this hostname an okokumo customer? A 200 for status.acme.com says Acme is a customer, pays for a tier that includes custom domains, and runs a status page. That's a customer list, queryable one name at a time.

c.RealIP() reads X-Forwarded-For first. Behind a proxy that's what you want. On an endpoint whose whole security model is "only the local Caddy may ask", it means anyone can claim to be local with a header. It's a one-word difference in the code and the entire difference in the threat model.

We proved the gate in production on one hostname, before and after. status.okokumo.fr was CNAME'd at the prod box and, while unverified, got no certificate at all — the handshake failed with no peer certificate available. Verified, it issued in 8.9 seconds. Same hostname, same box, same config; the only change was one timestamp in a database row.

One deliberate omission: the gate does not check whether the page is published. An unpublished page still gets its certificate, so publishing is instant rather than paying for ACME on the first real visitor. The page still 404s while unpublished, so the certificate protects nothing that shouldn't exist.

One record, and comparing the right thing

The usual custom-domain setup is two records: TXT to prove ownership, CNAME to route traffic. We ask for one. A CNAME pointing at our status host proves both at once — you can't point someone else's DNS at us, so routing is ownership here. A TXT token would prove the same fact twice, in exchange for a longer setup and one more thing to get wrong.

The trap is how you check it. The obvious way is wrong:

name, err := net.LookupCNAME(domain) // don't

LookupCNAME follows the chain to the final canonical name, which needn't be ours. A customer whose provider flattens records, or who chains through a CDN, ends up with a canonical name that isn't status.okokumo.com even though traffic arrives here perfectly.

So we resolve both sides — theirs and our own status host — and compare addresses. That checks the property we actually care about, traffic arrives here, and it holds for a plain CNAME, a chained CNAME, an ALIAS at an apex, or a customer who used an A record. Resolving our own target instead of hardcoding an IP means rebuilding the box doesn't invalidate every verified domain. We never connect to those addresses, only compare them, so a domain pointed at 127.0.0.1 just fails to intersect.

Two details that only surface in use. We query public recursors explicitly rather than the system resolver, because a customer who added their record thirty seconds ago would otherwise be told "not found" for the length of the box's negative cache. And we resolve our own host first: if that fails the outcome is "temporary, ours", not "not found", because blaming the customer for our resolver outage is the worst available answer.

What you can't claim

Validation runs first, so a hostname that can never carry a certificate gets rejected in a form rather than discovered as a failed ACME order.

Apex domains are refused — they can't reliably CNAME. The tempting check is counting labels, and it's wrong: acme.co.uk is an apex with three labels, and no label count distinguishes it from status.acme.com. The public suffix list does.

Our own domains are refused too, and one case earns the rule: rejecting everything under okokumo.com covers status.okokumo.com itself, the CNAME target. Without it a customer could claim the very name they're pointing at and build a resolution loop.

Normalisation is load-bearing rather than cosmetic. Caddy sends the SNI host lowercased, so a row stored as Status.Acme.com would never match the ask endpoint — and that failure presents as "verified, but TLS doesn't work", which is a miserable thing to debug.

Verification rots

Verification is a snapshot, and a row that says verified forever decays. The one that would end up in a postmortem: if the domain changes hands and the new owner points it back at us, we serve the previous customer's page to them.

So a goroutine re-runs the DNS check on every verified domain daily, revokes after seven days of continuous failure, and emails the owner after the first, so a bad week sends two emails rather than seven. DNS propagates in minutes, so a week of unbroken failure is unambiguous while still forgiving a customer mid-migration.

The branch I like most is the one that does nothing:

case result.Outcome == VerifyTemporary:
	// Our lookup failed, not their DNS. Neither punish nor forgive: the
	// failure streak is left exactly as it was, so a resolver outage on our
	// side can't revoke a working customer, and can't reset the clock on a
	// genuinely broken one either.

Due-ness lives in a custom_domain_checked_at column rather than a time.Ticker, and that's not fussiness. We deploy several times a week, and a 24-hour timer in a process that restarts every couple of days almost never fires — the feature would have looked implemented, passed its tests, and quietly never run.

The loop only ever revokes; it never re-verifies. A customer whose domain was revoked presses Verify again. Automatic re-verification would mean a hostname could silently start serving someone's page months after they stopped expecting it to.

What revocation actually does

I had this wrong in my own notes, twice, and only measuring fixed it.

What I'd assumed: clearing the verification leaves the certificate in Caddy's storage, so a revoked hostname keeps working until it expires ninety days later, with failing renewals in the meantime. Both halves were false.

Measured against a real Caddy 2.11.4 — two-minute internal certificates, ask endpoint flipped to deny mid-life — Caddy re-consults the ask endpoint at the renewal point, is refused, then declines to renew and refuses the handshake for the certificate it already holds:

tls.on_demand  certificate should not be obtained
tls            on-demand certificate issuance denied
http.stdlib    certificate is not allowed for server name

So the real timeline is: content stops immediately, because the public handler gates on the same column the ask endpoint does; TLS keeps terminating from the cached certificate until it enters its renewal window, roughly day 60 of 90, serving our branded 404 over valid TLS; then the hostname goes dark by itself, with no certificate storage to clean up and no ACME order attempted, since permission is checked before the order.

Revocation needing no cleanup is a nice property. It's also one I'd have kept believing the opposite of, indefinitely, without a container and two-minute certificates. Assumptions about someone else's state machine are worth what you paid for them.

What this doesn't solve

TLS revocation isn't instant. Content stops at once, but a revoked hostname keeps presenting a valid certificate for up to ~60 days. If you need that to stop on command, this design doesn't do it, and the honest fix is deleting from certificate storage rather than leaning on the renewal gate.

Apex domains aren't supported. You need a subdomain in front of something.

A brand-new domain's first request can fail during a deploy, because first contact needs the ask endpoint and the API restarts. One retry fixes it, which is small comfort if it's someone's first impression.

---

okokumo is infrastructure monitoring hosted in France — HTTP, heartbeat, TLS and domain checks, with alerts confirmed across EU regions before they reach you. The confirmation logic is described here.