Incident 4471: A Postmortem

The phone lit the ceiling before it made a sound, which is the part she has never gotten used to. Nine seconds later it made the sound.

The alert said upstream timeout. The upstream was healthy. The upstream had been healthy for six hours and would remain healthy throughout, which is what made the first twenty minutes so expensive: a healthy service is a very convincing alibi.

The actual failure was a TLS certificate on an internal proxy, issued twenty-six months earlier by an engineer who left the company in the spring. It had been renewed twice by a cron job on a machine that was decommissioned in June, as part of a migration that was completed successfully, thoroughly documented, and reviewed by four people, none of whom knew the cron job was there.

“Nobody did anything wrong,” she said. “That's what I keep writing in the document and what nobody believes when they read it. Four competent people did four correct things and the intersection of the four correct things was a hole.”

Service was restored at 3:51 by hand. The permanent fix — an owner, an inventory, an alert at thirty days rather than at zero — shipped Thursday and is nine lines long.

The document is fourteen pages. The fix is nine lines. She says that ratio is the single most honest thing about the profession, and that anyone who wants to understand engineering should be shown those two numbers and nothing else.

Step four of the escalation procedure links to an internal wiki. The wiki redirects to a second wiki. The second wiki returns a message reading “This space has been archived,” followed by a contact address at a domain the company no longer owns.

The procedure has been reviewed annually. The review confirms that step four contains a link. It does not confirm that the link contains anything.

An audit run the following morning found the same failure in nine of the thirty-one runbooks in the on-call rotation. Four link to the archived space. Two reference a dashboard that was migrated in 2025 and now resolves to a login page for a product the company no longer pays for. One instructs the responder to “escalate to the platform team,” which was dissolved into two other teams eighteen months ago, neither of which has claimed the pager.

The nine were not written badly. They were written well, by people who cared, at a moment when every link in them worked. Documentation does not rot because anyone neglects it; it rots because the world it describes keeps moving and the document does not, and nothing in any review process is designed to detect the difference between a link that works and a link that merely exists.

A crawler that checks every link in every runbook nightly and files a ticket on a 404 is about forty lines of code. It has been on her list since March. It is now, following this incident, on somebody else's list, with a date.

“Every runbook is a photograph of a company that no longer exists,” she said. “You are always being rescued by a stranger from the past who assumed you'd know what they meant.”

The policy grants a day off after a night page, which is generous, correct, and completely unable to return the specific hours removed between 3:04 and 5:20.

She took the day. She was awake for most of it at the wrong angle. The debt does not clear; it consolidates.