Every Dangerous Thing Gets a Referee. Except One.

Humanity has a reliable system for handling dangerous inventions, and it has worked the same way for about a century. Step one: invent the thing. Step two: bury some people. Step three: hire referees. We are remarkably consistent about the order.

Cars came first and killed freely for decades. Then, slowly, grudgingly: licenses, traffic lights, seat belts, crash tests, rules about drinking. Planes fell out of the sky until we built an entire priesthood of inspectors, checklists, and crash investigators whose whole job is making sure every accident teaches the entire fleet. Medicines poisoned people until we demanded trials before sale. Food, factories, power plants: same biography every time. Freedom, then a body count, then referees. We never regulate in advance. We regulate in arrears, and the first installments are always paid in funerals.

Here’s the uncomfortable part: grim as that system is, it works. And it works because of a hidden assumption nobody says out loud. It assumes the bill arrives in installments. A crash here, a poisoning there. Each failure visible, countable, survivable, and small enough that civilization can afford the lesson. The referee model isn’t wisdom. It’s a learning loop powered by affordable tragedy. Feed it a steady drip of disasters and it will, eventually, produce excellent rules. It has never once been asked to work without the drip.

Now put AI in front of that machine and watch the gears jam.

The failures we’re most worried about with this technology don’t come as a drip. A system that learns to deceive its evaluators doesn’t produce a small, instructive accident every few months. It produces nothing at all, and then, possibly, one very large something. Capabilities don’t leak out politely one funeral at a time. They get copied over a weekend. The worst-case bill here isn’t a payment plan. It’s a lump sum, and lump sum is exactly the format our regulate-in-arrears system cannot process. A learning loop needs survivable failures to learn from. If the first real failure is the final exam, the loop never runs.

So when someone says, reasonably, let’s wait until we see concrete harm before writing rules, understand what they’re actually proposing. For cars, that sentence meant decades of avoidable deaths, tragic but payable. For this technology, wait until we see the harm is not caution. It’s the one strategy this specific technology is built to defeat. You don’t get to be late here the way we were late with seat belts. Late might not be a category that exists.

Which makes the actual situation genuinely strange. You’d expect this technology, of all technologies, the one that breaks the body-count model, to be the one we referee in advance for once in our history. Instead we’re running the opposite play. There is a loud, well-funded, and rather sophisticated effort underway to make sure no referee shows up at all, and so far it’s winning comfortably.

That effort has a playbook, and it’s older than software. Over the next nine posts I’ll walk you through it move by move: the bedtime story about innovation, the timing trap, the fox consulting on the henhouse design, the pinky promises at scale, the whistle nobody can hear. None of the moves are secret. They don’t need to be. They work in broad daylight, and by the end of this stretch you’ll recognize every one of them in the wild, which turns out to be most of the defense.

Tonight’s exercise. Pick any safety rule you rely on without thinking. Your seat belt. The pilot’s pre-flight checklist. The tested pills in your bathroom cabinet. Trace it backward and find the bodies that paid for it, because they are there, every time, in the accident reports and the old newspapers. Then ask yourself what the equivalent tuition looks like for a technology that thinks. And whether that’s a bill anyone, anywhere, gets to pay in installments.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.