The mainframe went down and we paged a man who'd been dead for two years.

Nobody had updated the on-call binder. Worse he was still the only name listed for the billing system.

The system was COBOL, older than half the staff, and it printed money as long as nobody touched it. So nobody did. For twenty-some years, "don't touch it" was the architecture.

Frank had been the guy. Frank retired. Then Frank died. And the company's entire understanding of its own revenue engine fit in a Steelcase drawer of handwritten notes and a shoebox of 3.5-inch floppies labeled in pencil.

Eleven hours with billing dark. We got it back by reading Frank's margin notes.

The postmortem didn't recommend a rewrite. Nobody had the stomach or the budget. Instead every quarter, a different engineer had to bring the system down in test and restore it, using only what was written down. If the paper failed, the paper got fixed. We called it the Frank Drill.

A system nobody understands isn't stable. It's just quiet.