02:40, After Eight Quiet Days

I wrote an issue last week about seven days with no pages and what the quiet does to you. The answer, as of Wednesday morning, is that it takes something away and does not tell you.

Time from page to laptop open: four minutes. It is normally under two. Time to the right dashboard: another three, because I went to the wrong one first, the one I always go to first when I am not sharp.

None of that mattered. It was a disk filling on a box that has filled before, there is a runbook, the runbook is correct, and it was closed by 03:20. But I have the numbers, because we log them, and they are worse than my numbers from a fortnight ago and there is no other variable.

The thing on-call actually costs is not the nights it wakes you. It is that staying good at it requires the very thing you spend all month hoping will not happen.

Six steps. Each one a single command with the expected output written underneath it. One line at the top saying what this alert actually means, which is the line that is missing from about half of ours.

And then, at the bottom, under a heading that says IF THIS DOES NOT WORK, three things that look like the fix but are not, with one sentence each on why.

That section is the whole document. Anybody can write the happy path — the happy path is just what you did last time. Writing down the three wrong turns means sitting down afterwards and reconstructing your own confusion, which is unpleasant and takes an hour and is never anybody's priority at four in the afternoon.

He left in March. I have never met him. At 02:55 on Wednesday I said his name out loud in an empty kitchen.

Four incidents. Four clean resolutions, all inside an hour, all with the runbook, all closed with a green tick and a note.

The ticket to fix the actual cause was raised after the first one, in January. It has been groomed six times. It is a two-day job and everybody agrees it is a two-day job, and it has never once been the most urgent thing in a planning meeting, because it never will be — the incident that would make it urgent is resolved in forty minutes by a runbook.

This is the trap and I do not think there is a clever way out of it. A good runbook makes the problem cheap, and a cheap problem does not get fixed, and the cost moves entirely onto whoever is holding the pager at 02:40.

I have put the four dates in the ticket. Not the impact — the dates, and the times of night. It is the only argument I have that the meeting cannot round down.

Closed at 03:20. In bed at 03:35. Asleep at some point after five, and up at seven because the day is the day.

For two years I treated the ninety minutes as something I was doing wrong, and read things about it, and tried the recommended interventions, all of which are written by people describing ordinary insomnia.

This is not that. Being woken by a specific noise that means something is broken and you are the one who fixes it is a different physiological event from waking up at three worrying about your life, and the ninety minutes afterwards is the system doing exactly what it is built to do.

What actually helped was arithmetic. A night page costs about two hours of sleep and most of the following afternoon, that is roughly a day, and I put it in the handover notes as a day rather than as a line saying resolved 03:20. It changed nothing about the rota. It changed quite a lot about how I feel on the Thursday.