Every conversation about this eventually arrives at the same question, usually from the most sensible person in the room. Fine, they say, but where’s the line? What would count as too far? Tell me the thing that must not happen and I’ll take you seriously. It’s a fair question and I’ve stopped trying to answer it, because the question has a hidden assumption that’s doing all the damage.
The assumption is that control is an object. Something you hold, that can be dropped, at a moment, with a sound. Losing control sounds like losing your keys. And if that’s what it is, then there must be a line: on this side you have the keys, on that side you don’t, and the sensible thing is to stand well back from it and carry on.
Control isn’t an object. It’s a set of conditions that happen to be true at the same time, and there are four of them. You can understand what the thing is doing. You can say no to it. You can replace it with something else. And you can impose a consequence when it does something you don’t want. When those four hold, you’re in control, and the feeling of being in control is really just the sensation of all four being true at once, which is why it feels like a single thing when it’s actually four.
Now notice they erode separately, at different speeds, for unrelated reasons. Understanding goes first and went years ago, before most people were paying attention, because these systems are grown rather than built. Refusal erodes commercially, one dependency at a time, until saying no costs more than complying. Replacement erodes when the alternatives fold, when the knowledge leaves the building, when switching means rebuilding forty integrations. Consequence erodes quietly through the accountability gap: when a recommendation gets rubber stamped there’s nobody left to hold responsible, which is a form of impunity nobody granted anyone.
That’s why there’s no line. Four separate dials, moving at four different rates, in four different institutions, none of them announcing anything. There’s no moment at which all four flip. There’s a Tuesday on which the last one finishes moving, and no instrument anywhere is measuring the combination.
You already know this pattern from ordinary life, which is why the demand for a line is stranger than it sounds. Name the day you stopped being able to tell your teenager what to do. There wasn’t one. There was a year or two in which your instructions gradually became requests, and requests became opinions, and at some point you noticed that the relationship had changed shape and that you couldn’t say when. Empires lose provinces this way. Companies lose markets this way. Almost nothing important in human affairs happens at a line, and yet we keep demanding one before we’ll act, which is a very effective way of never acting.
And here’s the part that should annoy you, because it annoys me. The lines do get named, regularly, and then they get crossed, and nothing happens. People said the line was systems that write their own code. Then it was systems that act on the internet without a human approving each step. Then it was systems making consequential decisions about individuals at scale. Every one of those has happened. Each time, the response was not alarm but recalibration: on inspection, that turned out to be fine, so it wasn’t really the line. A line that moves whenever you approach it isn’t a safeguard. It’s a coping mechanism with a ruler drawn on it.
Fairness, because there’s a real objection here. Lines are not useless. Concrete thresholds are exactly what makes any safety regime enforceable, and vague continuous worry has never regulated anything. Aviation, medicine, and nuclear power all run on specific numbers that are somewhat arbitrary and enormously valuable, and picking an imperfect threshold and holding it is far better than what we’re doing now, which is waiting for a self-evident one. I’d take an arbitrary line tomorrow. My argument isn’t against drawing lines. It’s against waiting for one to appear on its own, because the situation doesn’t produce them and the waiting is indistinguishable from consent.
So the rest of this stretch drops the search for a moment and looks at the shape instead: how something gets displaced without ever being defeated, why every institution steps back one notch, and what the day it finishes looks like from the inside. Which, as the title of the last post in this stretch gives away, is not much.
Tonight’s exercise. Think back five years and name the capability that would have genuinely alarmed you then. Be specific and be honest, no retrofitting. Now check whether it exists. If it does, notice what you did when it arrived, which was almost certainly to update your sense of what’s normal and to name a new line further out. That adjustment is the most reliable thing about all of this. It happens every year, it feels like maturity, and it is the mechanism by which no line is ever crossed.