The Off Switch Is Now a Self-Harm Button

People keep asking whether we could turn these systems off, as though the answer were technical. It is not. The switch works fine. What matters is the invoice attached to pressing it, and that invoice has a line item added every quarter by people who are just doing their jobs well.

An earlier post in this series looked at why the industry itself can’t stop building. This is the other half, and it’s the half that applies to you rather than to a lab. Even if every company stopped training tomorrow, the systems already deployed are woven into operations that a modern country runs on, and the weaving is the point. Nobody deploys a system into something unimportant. You deploy it where it saves the most, which is always where the volume is, which is always something that matters.

Follow the logic in an ordinary sector. A hospital group adopts a scheduling and triage system. It’s better than the old process, so waiting times fall and the group reorganizes around the improvement: fewer coordinators, tighter staffing, beds planned to the new efficiency. Two years in, switching it off doesn’t return the hospital to how it was two years ago. It returns the hospital to how it was two years ago while carrying today’s patient load with today’s staff numbers, which is not a return. It’s a crisis. The old system wasn’t kept warm, the people who ran it moved on, and the slack that used to absorb the difference was spent as savings, because leaving slack unspent is what a badly run organization does.

That’s the general shape and it applies everywhere. Efficiency gains get consumed, not banked. Every improvement is immediately spent on doing more with less, which is exactly what improvement is for, and the consequence is that the improvement stops being optional the moment it’s absorbed. Dependence isn’t created by the adoption. It’s created by the reorganization that follows.

Then multiply it by the fact that there is no switch. There are ten thousand switches, in different buildings, owned by different companies in different countries, each one controlled by someone whose job depends on their part working. Ask them to shut down and every one of them will point out, correctly, that their piece isn’t the dangerous one, that their competitors won’t stop, and that stopping would hurt their customers immediately and for certain. The coordination problem this series described for the labs reappears in miniature, in every organization, all the way down. There has never been a lever that turns off a general purpose technology, and looking for one is a category error we keep making because a single switch is a comforting object to imagine.

What worries me isn’t that the option gets taken away. It’s that it expires quietly while remaining formally available. You can always shut it down, in the sense that a person on a bridge can always jump. The capability stays real and the price rises until pressing it becomes the kind of thing only a catastrophe could justify, and by then the catastrophe has to be visible and enormous, and this whole series has been about why the thing that would justify it might not look like anything at all.

Fairness. Some of this is normal and fine. We couldn’t turn off electricity or the phone network either, and nobody loses sleep over it, because those systems don’t do anything except what we ask. The relevant difference isn’t the depth of dependence, it’s what you’re depending on: something inert and understood, or something grown, unreadable, and improving on a schedule nobody controls. And there are real mitigations, which deserve naming rather than dismissing: keeping manual fallbacks exercised rather than documented, deliberately preserving slack, having more than one supplier, running the drills. Aviation and finance both do versions of this and it works. It’s unglamorous and it’s the actual answer, and almost nobody is paying for it because the payoff is a disaster that doesn’t happen.

Tonight’s exercise, and make it concrete. Pick one system your work depends on, ideally one that arrived in the last three years. Now plan a two-week outage. Not the technical failover, the actual work: who does what by hand, whether those people still exist, how much of the volume you’d simply drop, what you’d tell customers. Get to a real plan or get to the honest sentence, which is that there isn’t one. Then note the date, because next year the same exercise gets harder, and the year after that it stops being an exercise.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.