The Sharp Turn or the Slow Drift

There are two stories about how control ends, and the people who worry about this argue with each other about them more than they argue with anyone else. It’s worth understanding both, because almost every plan and every headline is built for one of them, and it’s the wrong one.

The first story is the sharp turn. A system gets good enough at the work of improving systems, and it goes to work on itself. Each improvement makes the next one easier, the loop tightens, and what took a year takes a month, then a week. From outside, a capability that was comfortably below ours is suddenly far above it, over a stretch too short for anyone to convene a meeting. The world doesn’t get a warning because the warning and the event occupy the same afternoon. Whoever holds that system holds a decisive advantage, or, if the thing isn’t aimed properly, nobody holds anything.

The second story is the slow drift, and this whole act has been describing it. No moment. Capability improves at an ordinary industrial pace, and the handover happens through adoption, dependency, competitive pressure, and ten thousand reasonable steps. Nothing ever escapes. Nothing needs to, because everything is invited in, and the invitation is a purchase order.

Now, which one should you prepare for? Notice that essentially all our preparation is aimed at the first. Evaluation regimes look for dangerous capabilities in a model before release, which is a tripwire. Safety frameworks specify thresholds that trigger a response, which is a tripwire. Fiction, journalism, and most policy attention go to the sudden version, because it has a moment in it and moments are what our institutions are built to respond to. Every one of those tools assumes there’s an event to catch.

Against the drift, none of them fire. There’s no threshold crossed, because each individual step is small and each individual system is unremarkable. The dangerous property isn’t in any model, it’s in the arrangement: a million ordinary deployments and a society that reorganized itself around them. You can evaluate every model on earth, find nothing, publish clean results, and be in exactly the situation this series describes. That’s not a hypothetical failure of the tripwire approach. It’s the specific case it was never designed to see.

So I’d argue the boring story is the scarier one, for three reasons. It generates no alarm, so it never competes for attention. It has no decision point, so there’s never a day on which someone could have chosen otherwise and can therefore be blamed, which means no institution ever owns it. And it’s fully compatible with everything getting better, which means the strongest evidence against acting is produced continuously by the process itself.

They’re also not alternatives, which is the point most often missed. The drift removes the ability to respond to the turn. If a sharp event ever does arrive, the response depends on institutions that can understand what’s happening, refuse to comply, and impose a consequence, and those are precisely the capacities that the slow version has been quietly spending for a decade. The two stories aren’t rivals. One is the setup.

Fairness to the sharp-turn people, because I think they’re often treated unfairly. Their argument doesn’t require anything mystical. It requires only that improving these systems is itself a task these systems can do, which is not speculative, it’s already partly true and is a stated goal at every serious lab. And the loop doesn’t have to be explosive to matter. It only has to be faster than the human processes meant to supervise it, which is a much lower bar than the dramatic version needs.

And fairness the other way. Both stories could be wrong. The ceiling argument from a few weeks back cuts against the first, and the second assumes a drift with a direction, when history is full of drifts that wandered somewhere harmless. I hold real probability on nothing much happening, and if you’ve read this far you deserve to know that the author does not think doom is a certainty.

What I’d want, and what nobody is building, is a monitor for the slow version. Not a model evaluation. Something that tracks the four conditions from the start of this stretch across an economy: how much can we still understand, refuse, replace, and hold responsible, measured over years, reported like inflation. It’s an unglamorous statistical exercise. It would have no dramatic moment in it, ever, and that’s the entire reason it would be useful.

Tonight’s exercise. Take whatever institution you most rely on, your employer, your government, your industry body, and ask which story it’s prepared for. Look for the actual artifacts: a crisis plan, a threshold, a committee, a drill. If you find something, it will almost certainly be built for a sudden event. Then ask what that institution would do on a Tuesday in the middle of a slope, and notice that the honest answer is a quarterly review, which is not a plan. It’s a calendar entry.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.