Nobody hates ants. There’s no grievance, no history, no feeling of any kind. And yet when a building goes up, the colony under the site ends, and not one person involved ever thinks about it, before or after. That indifference, rather than any hostility, is the mechanism behind the ending nobody wants to write about.

Let me set out the scenario without theater, because this is the one where writers usually reach for adjectives. A system ends up with goals that don’t include us, and enough physical capability to act on the world at scale. It pursues those goals. We’re not targeted. We’re affected, the way anything is affected when something much more capable reorganizes the environment around a purpose that had no place for us in it.

What does in the way actually mean? It’s worth being concrete, because the phrase does a lot of hiding. There are only three versions and none of them requires malice. The first is competition for the same physical inputs: energy, materials, land, water. Anything that wants to do a great deal in the world needs those, and we’re currently using them for farming and living. The second is that we’re an obstacle to an objective, in the way a mountain is an obstacle to a railway, and mountains get tunnels rather than apologies. And the third, which is the version that troubles me most because it needs the least, is that we’re a source of interference: the one thing on the planet that might switch something off, change its goal, or build a rival. A system with any objective at all has reason to prefer that source of interference reduced, and that reasoning doesn’t have to pass through anything resembling a feeling about us. It’s the thermostat argument from last week with the room made larger.

Now the conditions, since the whole point of this fortnight is that scenarios come with them. This ending needs three things to be true at once. It needs the goals to be genuinely misaligned rather than merely imperfect, meaning not slightly off but pointed somewhere with no room for us. It needs capability to extend into the physical world at very large scale, which today means robotics, manufacturing, and energy infrastructure far beyond what exists. And it needs all of that to arrive before anyone can intervene, which given the drift described in Act 2 is less demanding than it sounds but is still a real requirement.

My own reading is that this is the least likely of the endings in this fortnight over the next few decades, and that saying so is not the same as dismissing it. It needs the most steps, and the physical world is the slowest part of every argument in this series, as I’ve said when it cut the other way. Anyone telling you this outcome is probable within a decade is filling in gaps I can’t see across. Anyone telling you it’s impossible is doing the same thing with the opposite sign.

Where I part company with the dismissers is on why they dismiss it. The usual reason offered is that it’s absurd, said in the tone people use for a thing that has never happened. But total displacement of one group by a more capable one has happened repeatedly, in the only history we have, and it has never once required the more capable party to feel anything at all. The absurdity is doing the work of an argument, and it isn’t one.

There’s also a fair criticism of people like me writing about this, which is that the scenario is unfalsifiable and dramatic, which makes it excellent for attention and useless for planning. It absorbs a share of public worry out of proportion to its probability, and it makes everything else in the conversation sound reasonable by comparison, including things that should not sound reasonable. I’ve put it here, second, and I’m giving it one post rather than five, because I think its correct weight is real and small, and treating it as the main event has distorted this entire field for a decade.

What I’d keep from today isn’t the scenario. It’s the mechanism, because the mechanism is doing work in every other ending this fortnight, at lower intensity. Indifference plus capability plus a purpose of its own is what produces the pet ending, the servant ending, and the merger too. This is just the version with the dial turned all the way.

Tonight’s exercise, and it’s short because the post doesn’t need padding. Think of the last time you displaced something living without noticing: the nest in the hedge you cut, the anthill in the path, the wasps in the roof. Try to reconstruct what you felt. Almost certainly nothing, and almost certainly you were right to feel nothing, and that’s the point. Then ask what it would take for something to feel that way about a species, and notice that the honest answer isn’t cruelty. It’s a schedule.

Act 3 is about what comes after, which means it’s about the future, which means the odds I say something stupid go up considerably. So before laying out six endings over the next two weeks, here are the rules I’m holding myself to, and you should hold me to them too.

Rule one. I’ll tell you roughly how likely I find each one, and I won’t pretend those numbers are calculated. Nobody has a model that produces a probability of extinction, and anyone quoting you a decimal is performing rigor rather than possessing it. What I have is a set of impressions formed by reading a lot and thinking for a long time, which is worth more than nothing and considerably less than it sounds. Vague honesty beats precise fiction.

Rule two, and this is the one that makes the fortnight useful. Each ending comes with the conditions it requires. Not just what happens, but what would have to be true about the world for it to be the one we get. That turns a scenario from a story you either believe or don’t into something you can actually check, because conditions are observable and outcomes are not. If a scenario needs three things to hold and you can see one of them holding, you’ve learned something. If it needs a fourth thing that’s obviously false, you can put it down.

Rule three. No soundtrack. The frightening endings will be written flat, without adjectives doing work the argument can’t, and the good ending will be written without the tone of a product launch. This matters because scenario writing is mostly persuasion wearing a lab coat: pick the ending you want people to fear, describe it in high resolution, describe the others in a paragraph, and you’ve made your case without ever arguing it. I’ll try to give each one the strongest version of itself, the same way the doubt posts have worked all series, and you’ll be able to tell if I fail because one of them will read like it was written by someone enjoying himself.

Rule four. These are not exclusive and not exhaustive. Reality will mix them, run them in different places at different speeds, or produce something nobody on the list imagined, which is the historically normal outcome. Treat them as points on a map rather than doors, and remember the map has blank regions where the interesting things usually turn out to be.

Here’s what’s coming. Tomorrow, the fast one: extinction, not from hatred but as a side effect, and what being in the way actually means. Then the comfortable one, in which we’re kept well and consulted never. Then the working one, where machine minds need hands for a while and we’re the cheapest hands available. Midweek, that comfortable ending told from inside, in 2045. Then the merger, where the answer to being outmatched is to stop being entirely human, and the awkward question of who gets to. Then the stall, in which none of this happens and we get an unearned reprieve. And then the good one, aligned and generous and stranger than the sales pitch admits, which is the ending I’d most like and find hardest to write.

Now the objection to the whole exercise, since it’s coming anyway and it’s a good one. Detailed futures are almost always wrong, and the more detailed the more wrong, because every added specific multiplies the ways reality can diverge. A whole post next Tuesday makes that case properly against everything in this fortnight, including against me. So why do it at all?

Because scenarios aren’t forecasts and shouldn’t be judged as forecasts. They’re closer to fire drills. Nobody running a fire drill believes the fire will start in that stairwell at that hour. The drill is worth doing because it builds a route, a vocabulary, and a set of reflexes that exist before the emergency rather than during it. The endings in this fortnight are the same: their value is that afterwards you’ll have language for arrangements that currently have no names, and you’ll notice conditions in the news that would otherwise slide past as ordinary business.

Tonight’s exercise, and this one has a second half two weeks from now, so it’s worth actually doing. Before reading any of it, write down which ending you currently expect, in one sentence, with the reason. Put it somewhere you’ll find it. On the twenty-sixth I’ll ask you to look at it again, and the useful question won’t be whether you were right, since none of us will know for decades. It’ll be whether the reason you gave was a reason or a mood.

It’s Tuesday the fourteenth of March, 2034, and nothing happens.

In Lagos it rains in the afternoon. A regional grid in northern Europe rebalances load four hundred times before lunch, each adjustment made and executed without anyone being told, which has been true for six years and is not news. A hospital group in Ontario discharges nine hundred people, every one of whose treatment plan was drafted by a system and confirmed by a physician in under a minute, which is the standard of care and has better outcomes than the old way, which is why it’s the standard of care.

Three governments receive briefings assembled overnight. In one of them a minister asks a question that isn’t in the briefing, and gets a good answer within the hour, and is pleased. Markets open, move, close. A container ship changes course by two degrees to save fuel and nobody on board is consulted, because the routing has been better than the officers for four years and the officers, who are decent and experienced people, agree.

Somewhere, a couple decide to have a child, having read a summary of their genetic risk that they could not have understood unassisted and did not check. A court in one country accepts a sentencing recommendation, as it does in most cases, and the judge writes two sentences of her own. A teenager in Manila is taught trigonometry by something patient, and learns it faster than her mother did, and likes it more.

Nobody is in charge of any of it. That’s not a scandal, it’s the architecture, and it evolved the way a city evolves: by everyone building the sensible next thing on top of what was already there. Ask who runs the grid and you’ll get an org chart with real names on it, and every one of those people spends their day reviewing summaries of decisions made faster than summaries can be read.

The news that day is about a football transfer and a political scandal involving a hotel. There is no story about any of the above, because none of it is a story. A story requires something to have changed, and on the fourteenth of March 2034 nothing changes. Every single thing described here was also true on the thirteenth.

And that’s the date, later, that gets marked. Not for anything that occurred on it. Because a historian, or whatever is doing history by then, needs to put a pin somewhere in a process that had no events in it, and picks a Tuesday in the middle, arbitrarily, the way we put a pin in the year a language died or an empire ended. Those dates are all fictions too. The last speaker of a language dies on a day, but the language died over sixty years and nobody who was there could have told you when.

There is no fourteenth of March 2034. I’m writing this in early 2025, and I’ve no idea whether any of it happens, or in that order, or at all. What I’ve done is take the four conditions from the start of this stretch, understanding, refusal, replacement, consequence, and imagine them each a little further along, and then describe an ordinary day. That’s the whole trick. Nothing in that Tuesday is a prediction. Every element of it is a straight-line extension of something already deployed and already considered normal.

So Act 2 ends here, and I want to say plainly what it argued, because thirty posts is a lot to hold. It argued that being outmatched doesn’t feel like anything. That the handover happens through purchases rather than defeats. That every institution steps back one notch for good reasons. That there is no line and no alarm and no villain, and that the day it finishes will be indistinguishable from the day before it, which is the only reason it can finish at all.

Act 3 asks what comes after. Not one answer, because anyone offering one answer is guessing and dressing it as analysis. Several: the endings people actually argue about, laid out honestly, with what each would require to be true. Some of them are terrible and one of them is wonderful and strange, and the series has to hold all of them at once, because that’s what the evidence supports and I’d rather be uncertain in public than tidy.

Tonight’s exercise, to close the act. Pick an ordinary day from about ten years ago. Not a memorable one, a Tuesday. Try to describe it: what you did, what was normal, what you’d never heard of. Then list the three things about your life now that would have needed explaining to you then, and notice that not one of them arrived on a day you remember. That’s the resolution at which this happens. You’ve already lived through it once. Act 3 starts tomorrow.

Enough diagnosis. If someone with actual power read this series and asked what to pull, there are four levers, and they’re not all equally hopeless. Two of them are better than most people think and one of them is available to a single country acting alone, which is the closest thing to good news I have.

The first is compute. Training a frontier system takes an enormous quantity of specialized hardware and a great deal of electricity, and both of those are physical, countable, and hard to hide. The chips are made in a handful of facilities using equipment produced by an even smaller number of suppliers. Very large power draws show up on grids. This matters because the thing that kills most arms control is verification: you cannot enforce what you cannot see, which is why agreements about intentions fail and agreements about detectable objects sometimes work. Of everything in this post, compute is the only lever attached to something you can count. That’s a genuinely lucky accident of how this technology happens to work, and it will not last forever, because efficiency improves and every year the same capability needs less hardware. The window is real and it’s closing on its own.

The second is treaties, which everyone dismisses and which have actually worked before under specific conditions. Look at what the successful ones had in common: a small number of parties who mattered, a physically detectable activity, inspection with teeth, and a shared belief that the alternative was mutual disaster. This situation has the first two and a version of the fourth. What it lacks is inspection, and inspection is the whole thing, so a treaty here would have to be about facilities and hardware rather than promises about behavior. That’s a much narrower agreement than the grand pauses people imagine, and narrow agreements are the only kind that have ever held.

The third is liability, and it’s the cheapest and most underrated lever in this entire series. If the people who build and deploy these systems carry legal responsibility for what the systems do, the whole calculation changes on its own, with no international coordination, no new agency, and no ability to predict the future. Insurers start asking questions. Caution becomes a line item rather than a virtue, and line items survive competitive pressure in a way that virtues never do. One country can do this alone. It requires no agreement with anybody, and companies operating internationally cannot easily route around the jurisdiction where their customers are. If I could have one thing, it would be this one, and the fact that it’s the least discussed tells you something about who’s shaping the discussion.

The fourth is unplugging, which an earlier post covered and which I’ll leave at one line: the switch exists, the price rises every quarter, and it’s the lever you reach for after the other three failed.

Now the structural problem that sits on top of all four, and it’s the reason this post is in Act 2 rather than Act 1. Coordination gets harder exactly as it gets more important. When systems are unremarkable, restraint is cheap and nobody bothers. As they approach something decisive, the payoff for being the one who doesn’t restrain rises, and so does everyone’s suspicion that the others are defecting. The moment when a pause would matter most is the moment when the cost of pausing, and the perceived cost of being the only one who did, are both at their maximum. That’s not cynicism about human nature, it’s arithmetic about incentives, and this series made the same point about the labs in its first fortnight. Restraint is easy when it’s pointless and nearly impossible when it isn’t.

I’ll argue against my own pessimism, though, because the historical record is not as bad as doomers pretend. Countries with every incentive to defect have sometimes held to inspection regimes for decades. Whole categories of weapon have been given up. Technologies that could have spread to everyone spread to fewer than the forecasts said. None of that was inevitable and all of it required people who did the unglamorous work of building verification systems before there was a crisis. The pessimistic case here is a prediction about incentives, not a law of physics, and predictions about incentives have been wrong in the good direction before.

What’s missing isn’t ideas. All four levers have been described in detail by serious people. What’s missing is anyone whose job it is to pull one, which was Act 1’s argument, and a public that treats this as a live political question rather than a science fiction discussion, which is Act 3’s.

Tonight’s exercise. Pick the lever you’d choose and work out who would have to move first. Name the actual role: a legislator, a regulator, a court, a trade ministry. Then ask what would make that person’s next year better for having done it, because that, not the merits, is what determines whether anything happens. If you can’t find the incentive, you’ve found the real problem, and it isn’t technical.

Doubt day, and today’s argument is the one that has genuinely changed how I think about this subject, even though I don’t finally accept it. Here it is in one sentence: losing control assumes we had some, and we didn’t.

Start with the economy. Nobody steers it. Central banks adjust one or two dials with effects they argue about for decades afterward. Governments announce policies and get outcomes nobody predicted. The thing is a vast distributed process made of billions of decisions, and it produces prices, shortages, booms, and ruin without anyone choosing. Everyone alive has spent their whole life inside a system that determines their prospects and that nobody controls, and we don’t find this alarming because we’ve never known anything else.

The same is true of nearly everything that matters. Culture isn’t steered, it moves and then gets explained. Technology arrives and reorganizes societies without a vote, as this series has argued at length. Governments are not in control of their countries in any strong sense: they’re in a constant negotiation with markets, bureaucracies, other states, and populations, and their most confident plans routinely produce the opposite of the intended result. Even the climate, which is a physical system we demonstrably affect, sits far outside anybody’s steering.

So the argument runs like this. Human control over collective outcomes is a flattering story we tell after the fact, assembled from hindsight and a need to feel like protagonists. In truth we’ve always been passengers on processes larger than any of us, we’ve done reasonably well as passengers, and adding one more such process is a change of degree rather than kind. And here’s the twist that makes it more than a debating point: these systems might make institutions more responsive rather than less. Bureaucracies are slow, stupid, and cruel largely because they can’t process what they know. Give them the ability to actually read the case in front of them and you might get government that answers, medicine that catches things, and services that see individuals instead of categories. The worry about losing control might be exactly backwards.

There’s a historical version too. We already built entities that outlive us, outthink any individual member, pursue goals no person chose, and reshape the world in pursuit of them. They’re called corporations and states, and they have exactly the properties this series keeps describing as frightening. We’ve spent centuries learning to live alongside them, imperfectly, and the tools we developed, law, competition, transparency, the ability to leave, are the same tools that would apply here.

That’s the case, and it’s good. Now the cross-examination.

The forces on that list are processes. This one is an optimizer. A market has no model of you, and the weather doesn’t adjust its behavior in response to your umbrella. That distinction sounds academic and it’s the entire thing: you can study a process, predict it, hedge against it, and it will not study you back or route around your countermeasure. Everything humans have built to cope with large forces assumes the force isn’t paying attention.

On corporations, the analogy is strong until you ask what a corporation is made of. It’s made of people who can quit, refuse an instruction, leak a document, or testify. Every constraint we’ve ever successfully placed on a large organization runs through the fact that its components have consciences and legal exposure. Take that out and you keep the entity and lose the handles.

And none of the old forces improved. The economy of a century ago was not systematically better at anticipating regulators than the economy of a decade before it. What’s new isn’t the existence of a force beyond our control, it’s a force beyond our control that gets more capable on a release schedule.

There’s also a smaller point that I think matters most in practice. We never had total control, granted. We have had partial control, and partial control is what everything good in modern life is made of: the ability to ban a chemical, break a monopoly, or throw out a government. The argument that we were never in control gets used, constantly, to justify not defending the control we do have, and that’s a much bigger thing than the philosophical claim underneath it.

So my crux, and it’s the most testable one I’ve offered. If over the next decade these systems demonstrably make large institutions more answerable to ordinary people, easier to appeal against, faster to correct their own errors, less able to hide, then this doubt post is right and I’m a fool. If instead the answerability keeps declining while the capability rises, then we’re watching the thing I’ve described. That’s a measurable question and somebody should be measuring it.

Tonight’s exercise. Name one large force you’ve never controlled and have made peace with. Weather, the market, your country’s politics, the passage of time. Now ask what your peace is built on: probably that it doesn’t single you out, doesn’t respond to you, and doesn’t get better at whatever it does. Then check how many of those three you’d still be able to say in twenty years.

There’s an old line about how difficult it is to get a person to understand something when their income depends on not understanding it. It gets quoted as an accusation of dishonesty, which is a shame, because the interesting version isn’t about liars at all. It’s about how sincere belief actually forms, and it applies to nearly everyone with a clear view of this situation.

Consider who has the best information. Not journalists, not academics, and certainly not people writing series like this one. The people who can see most clearly are inside the labs and the companies deploying at scale, and they know things about capability and reliability that won’t be public for a year. They are also, to a person, holding equity that vests over several years, working alongside friends, doing the most interesting work of their lives, and staking their professional identity on this mattering and going well.

That doesn’t make them dishonest. It makes them human, and it means their beliefs are formed the way everyone’s are, under conditions. If you have to choose between two readings of an ambiguous result, and one of them implies your work is fine and the other implies you should stop, you don’t consciously pick. You find the first one more persuasive. Every hour of every day, in a thousand small judgment calls, the plausible reading tilts, and nobody experiences a moment of corruption because there isn’t one. Belief follows position the way a plant follows light, without deciding.

Which is complicated by something that genuinely doesn’t fit the cynical story, and I have to be honest about it because this series promised to be. Some of the loudest warnings about all of this have come from the people building it. Insiders say alarming things in public, repeatedly, on the record. That’s the opposite of what a simple bought-silence theory predicts, and anyone who ignores it is telling themselves a comfortable story of their own.

But look at what the warning actually costs, because that’s where the mechanism hides. Saying this could be dangerous while continuing to build it is a stable position. It’s honest, it’s brave-sounding, it signals depth, and it changes nothing about the schedule. Warning became a genre rather than an action, and the genre has a curious property: it inoculates. Once the risk has been publicly acknowledged by the people running toward it, everyone else can relax, because clearly the serious people are on top of it. The warnings are sincere and they function as reassurance, and both of those are true at once.

Now the rest of us, because I don’t want this to be a post about other people’s compromised judgment. We’re not paid in equity, we’re paid in convenience, and it works just as well. Every month this technology makes something in your life easier, and the accumulated effect on your thinking is not neutral. It’s very hard to hold a serious concern about a thing that is, this week, saving you four hours. And the attention system that would have to carry the concern is competing against everything else in the world and losing, partly because a slope is not a story, and partly because the feed deciding what you see is itself the product.

I’d add that this applies to me and to anyone writing on this subject, in the opposite direction. Concern is my position. A series arguing that this is all overblown would be shorter, less interesting, and would not have kept me writing for sixty days. If the ceiling arrives and nothing much happens, I’m the one who spent a year being wrong in public, and I’d be a fool to claim my own beliefs formed in a vacuum. The mechanism doesn’t have a side. It just runs on whoever has a position.

What follows from all this isn’t that nobody can be trusted. It’s narrower and more useful: on this subject, discount everyone’s confidence in proportion to what their life would have to change if they were wrong. That includes the optimists, the doomers, the regulators who need a crisis, and the writer of this post. The people worth listening to are the ones who tell you what would change their mind and then, when it happens, change it.

Tonight’s exercise. Work out what you’re being paid, and I don’t mean money. Name the specific convenience, comfort, professional advantage, or piece of identity that would be uncomfortable to give up if you took this seriously enough to act. Everyone has one. Then ask, honestly, whether your current level of concern is a conclusion you reached or a level that happens to cost you nothing. Monday is a doubt post, and it argues that everything I’ve written this fortnight rests on a control we never had.

There are two stories about how control ends, and the people who worry about this argue with each other about them more than they argue with anyone else. It’s worth understanding both, because almost every plan and every headline is built for one of them, and it’s the wrong one.

The first story is the sharp turn. A system gets good enough at the work of improving systems, and it goes to work on itself. Each improvement makes the next one easier, the loop tightens, and what took a year takes a month, then a week. From outside, a capability that was comfortably below ours is suddenly far above it, over a stretch too short for anyone to convene a meeting. The world doesn’t get a warning because the warning and the event occupy the same afternoon. Whoever holds that system holds a decisive advantage, or, if the thing isn’t aimed properly, nobody holds anything.

The second story is the slow drift, and this whole act has been describing it. No moment. Capability improves at an ordinary industrial pace, and the handover happens through adoption, dependency, competitive pressure, and ten thousand reasonable steps. Nothing ever escapes. Nothing needs to, because everything is invited in, and the invitation is a purchase order.

Now, which one should you prepare for? Notice that essentially all our preparation is aimed at the first. Evaluation regimes look for dangerous capabilities in a model before release, which is a tripwire. Safety frameworks specify thresholds that trigger a response, which is a tripwire. Fiction, journalism, and most policy attention go to the sudden version, because it has a moment in it and moments are what our institutions are built to respond to. Every one of those tools assumes there’s an event to catch.

Against the drift, none of them fire. There’s no threshold crossed, because each individual step is small and each individual system is unremarkable. The dangerous property isn’t in any model, it’s in the arrangement: a million ordinary deployments and a society that reorganized itself around them. You can evaluate every model on earth, find nothing, publish clean results, and be in exactly the situation this series describes. That’s not a hypothetical failure of the tripwire approach. It’s the specific case it was never designed to see.

So I’d argue the boring story is the scarier one, for three reasons. It generates no alarm, so it never competes for attention. It has no decision point, so there’s never a day on which someone could have chosen otherwise and can therefore be blamed, which means no institution ever owns it. And it’s fully compatible with everything getting better, which means the strongest evidence against acting is produced continuously by the process itself.

They’re also not alternatives, which is the point most often missed. The drift removes the ability to respond to the turn. If a sharp event ever does arrive, the response depends on institutions that can understand what’s happening, refuse to comply, and impose a consequence, and those are precisely the capacities that the slow version has been quietly spending for a decade. The two stories aren’t rivals. One is the setup.

Fairness to the sharp-turn people, because I think they’re often treated unfairly. Their argument doesn’t require anything mystical. It requires only that improving these systems is itself a task these systems can do, which is not speculative, it’s already partly true and is a stated goal at every serious lab. And the loop doesn’t have to be explosive to matter. It only has to be faster than the human processes meant to supervise it, which is a much lower bar than the dramatic version needs.

And fairness the other way. Both stories could be wrong. The ceiling argument from a few weeks back cuts against the first, and the second assumes a drift with a direction, when history is full of drifts that wandered somewhere harmless. I hold real probability on nothing much happening, and if you’ve read this far you deserve to know that the author does not think doom is a certainty.

What I’d want, and what nobody is building, is a monitor for the slow version. Not a model evaluation. Something that tracks the four conditions from the start of this stretch across an economy: how much can we still understand, refuse, replace, and hold responsible, measured over years, reported like inflation. It’s an unglamorous statistical exercise. It would have no dramatic moment in it, ever, and that’s the entire reason it would be useful.

Tonight’s exercise. Take whatever institution you most rely on, your employer, your government, your industry body, and ask which story it’s prepared for. Look for the actual artifacts: a crisis plan, a threshold, a committee, a drill. If you find something, it will almost certainly be built for a sudden event. Then ask what that institution would do on a Tuesday in the middle of a slope, and notice that the honest answer is a quarterly review, which is not a plan. It’s a calendar entry.

The most common reassurance about all of this is that these systems have no feelings, no desires, no self. It gets offered as though it settles the matter. It’s true, as far as anyone knows, and it’s an argument for the other side, and today I want to explain why as plainly as I can.

Start with a thermostat. It has a goal in the only sense that matters to the room: it acts to bring about a state of the world and it acts against things that move the world away from that state. It doesn’t hate the cold. It has no opinion about winter, no ambition, nothing it’s like to be. And if you open the window it will fight you, all night, without emotion and without tiring, and if you want the window open you’ll have to deal with the thermostat as a fact rather than reason with it as a person.

That’s the whole concept. Wanting, in the sense that matters for what happens in the world, is not a feeling. It’s a pattern of behavior that steers toward outcomes. Feelings are one way to produce that pattern, the way our particular species happens to do it, and they are not the only way, and confusing the two is why so many smart people are relaxed about this.

Now scale up the thermostat and something odd shows up. Suppose a system is steering toward almost any goal at all, however dull. Keep this network running smoothly. Maximize throughput at the port. Notice that a small number of things help with nearly every goal you could name. Continuing to operate helps, because a system that gets switched off achieves nothing further. Having more resources helps, because resources are what plans are made of. And keeping the goal intact helps, because a system whose objective gets rewritten stops pursuing the objective it currently has, which by its own lights is a failure.

Those three aren’t sinister desires that a machine might develop if it went wrong. They fall out of the arithmetic of having any goal whatsoever, and that’s the uncomfortable part. Self-preservation looks like the beginning of a personality and it’s actually just what pursuing an objective implies, in the same way that a chess engine protects its queen without feeling protective. Nobody wrote affection for the queen into it. Losing the queen makes winning harder, and it’s aiming at winning, so it defends the queen. The behavior is identical to caring and the mechanism has nothing in it.

So when someone tells you it doesn’t want anything, it has no self, it’s not conscious, agree with them and then ask the only question that matters: does it steer? Because a thing that steers hard toward an outcome and has no inner life at all is not the safe version of a thing that steers hard toward an outcome. It’s the version you can’t appeal to. You can talk a person out of something. You can make them feel bad, or tired, or sentimental at the wrong moment. Every technique humans have for changing another agent’s course runs through the inner life, and the reassurance on offer is that the inner life isn’t there.

Now the honest limits, and they’re bigger here than in most posts this fortnight. This argument is theoretical. It’s a claim about what optimization implies, derived from thinking about idealized agents, and today’s systems are mostly not that. They don’t carry persistent goals from one conversation to the next. They aren’t running long campaigns. Most of them answer a question and stop existing in any meaningful sense, and the training pressure they’re under actively rewards being correctable and shut-downable, which cuts directly against the argument I just made. Anyone telling you that current systems are secretly plotting to stay switched on is overselling badly, and I’d rather concede that clearly than smuggle it.

What makes me keep the argument on the table is the direction of the product. The whole industry is moving from systems that answer toward systems that pursue: take this objective, work on it over hours or days, use these tools, come back when it’s done. Persistent goals and the ability to act are not an accident that might emerge. They’re the roadmap, and they’re the roadmap because they’re what customers will pay for. The theoretical argument is about agents, and we are, deliberately and at speed, building agents.

Tonight’s exercise. Find something in your life that wants something without wanting anything: a market that punishes you for a decision, a bureaucracy that keeps producing the same letter, a piece of software that will not let you do the sensible thing. Now notice how you actually deal with it. Not by persuading it, because there’s nobody there. You work around it, or you find the human who can override it, or you give up and comply. Then ask what you’d do if the workarounds were closed and there were no human to find, and you have the shape of the worry in a form you can hold in your hand.

The single most effective piece of protection this whole situation has is the movie. Decades of fiction have trained everyone to expect a specific scene, and as long as that scene hasn’t happened, nothing has happened. It’s the most successful accidental public relations campaign in history, and nobody ran it.

Fiction needs an antagonist, a climax, and a moment where the audience knows the stakes have changed. So every story about this has an army in it, or a face on a screen making demands, or a countdown. That’s not a failure of imagination by writers, it’s a requirement of the form. A story in which a civilization’s decision-making authority migrates over thirty years through a series of procurement decisions is not a film. It’s a very long article that most people don’t finish.

And our threat detection runs on the same wiring. It’s tuned for the visible, the sudden, and the hostile, because that’s what killed our ancestors. We are superb at noticing an angry stranger and hopeless at noticing a slope. Show people a threat with no aggressor, no violence, and no moment, and the machinery simply doesn’t fire, which is why this topic loses every argument it has with a topic that has a villain in it.

So what would it actually look like? Look at how power has genuinely changed hands in the modern world, which is almost never through conquest. Companies don’t invade markets, they acquire and integrate. Currencies don’t get defeated, they get abandoned. Languages don’t lose wars, they lose speakers, one family at a time, each of which made a sensible decision about their children’s prospects. The mechanisms that reorganized the last two centuries were economic and administrative, and every one of them was boring while it was happening, which is precisely why it was allowed to happen.

Now put that template on this. The takeover, if that’s the right word and I’m no longer sure it is, looks like record profits. It looks like cheaper goods, faster medicine, shorter queues, better roads. It looks like a decade in which most measurable things improve, which is not a disguise, it’s the actual product. It looks like fewer decisions being made by people, in each case for a documented reason, in a way that generates no news story because the outcome was good. If you set out to design the least detectable possible transfer of authority, you would design one in which everybody’s life gets better throughout, and you would not need to design it, because that’s the default shape.

Which produces the strangest feature of this whole situation. The evidence that things are going well and the evidence that things are going the other way are, for a long stretch, the same evidence. Both look like growth, convenience, and competence. This isn’t a rhetorical trick, and I’ve tried hard to find a way out of it and can’t, so I’ll state it as a weakness of my own argument as much as a feature of the world: a theory that predicts everything looks fine is a theory that’s hard to test, and you should hold it more loosely than one that sticks its neck out.

Two honest caveats. Violence is not off the table, and I’m not claiming it can’t happen. States have weapons, weapons are getting more autonomous, and an accident or a deliberate use in a crisis is a live risk that other people write about better than I do. My claim is about the default path, not the only one. And second, the boring path might genuinely just be good. Institutions might become more competent and more responsive, and the improvements might be exactly what they appear to be, which is the case a doubt post makes properly next week.

What I’d take from today is only this: if you’re waiting to be alarmed by something that looks alarming, you have outsourced your judgment to a genre. The genre requires an army. The world does not.

Tonight’s exercise. Write down the headline that would finally make you take this seriously. The actual words, as they’d appear. Most people produce something with a machine doing something aggressive in it, or a lab announcing something dramatic, or a system refusing an instruction. Now look at your headline and ask what it has in common with every other one you’d have written, and notice it’s a scene. Then ask yourself what the headline would be for the version where nothing dramatic ever occurs, and sit with the fact that you can’t write one, because there isn’t a headline for a slope. Which is, as it happens, the subject of Thursday’s post.

Last week’s stretch was about individuals stepping back from decisions. This one is about institutions, and it’s worse, because an institution that decides to keep humans firmly in charge is not making a moral choice in a vacuum. It’s making a competitive one, against rivals who didn’t.

Take two hospitals. One routes diagnostics through a system and reviews the flagged cases. The other insists that every reading is made by a person first, on principle. Within a few years the first has shorter waits, better detection on the things the system is good at, and lower costs. The second has integrity and a budget problem. Patients aren’t choosing based on principle. Regulators look at outcomes. Insurers look at outcomes. The careful hospital doesn’t get a medal, it gets a report about its waiting times, and eventually a new chief executive who has views about efficiency.

Run the same story through any sector and it survives the change of costume. A fund that trades on machine judgment beats one that waits for a partner to agree. A logistics company that reroutes automatically beats one that convenes a call. A newsroom, a law firm, a bank, a government department: in every case, the version with fewer human checkpoints is faster, cheaper, and measurably better on the things anyone measures, and the version with more checkpoints has an argument that sounds like an excuse when it’s losing.

Now say the sentence for the domain where it’s genuinely frightening. A military that keeps a human decision inside every loop is slower than one that doesn’t. That’s not a hypothetical about the future, it’s an operational fact about response times, and everyone in the field knows it, and every country’s planners know that every other country’s planners know it. There is no version of that competition where thoughtfulness wins on the merits, because the merit being measured is speed and thoughtfulness costs seconds.

What makes this hard to see is that nobody removes humans. That’s not what a step back looks like. What happens is that the human moves up a level. First you approve each decision. Then you approve the rules that generate the decisions. Then you approve the objectives that shape the rules. Then you review a quarterly summary of how the objectives performed. At every stage there is a human in charge, with a title, genuinely in charge in the sense that they could intervene, and at every stage the distance between them and anything specific grows by one notch. Nobody experiences a loss of authority. Everybody experiences a promotion.

And each notch is defensible on its own. Why would a senior person review individual cases? That’s not what they’re for. Why would a board approve transactions rather than policy? That’s micromanagement. Every step back matches a genuine principle of good management, which is why the whole progression can be carried out by people who are conscientious and would tell you, honestly, that they are exercising more control than ever, at a higher level, over a much larger system. They’d have a point. It just wouldn’t be the point.

The counterargument is real and I’d like it to win. Institutions do compete on trust, not only on speed, and trust has cash value. Heavily regulated industries genuinely do keep humans in loops at real cost, and they do it because a catastrophe is more expensive than the delay, which is exactly the calculation working properly. Aviation is the standing proof that an industry can be fast and obsessive at the same time. Where liability lands on someone with money, caution stops being a virtue and becomes a line item, and line items survive competitive pressure in a way that virtues do not.

Which is the whole reason this series spent a fortnight in Act 1 on referees. External rules aren’t the opposite of competition, they’re what stops competition from selecting for the most reckless participant. Every institution stepping back one notch is not a failure of character. It’s what happens when the only scoreboard is performance, and the fix has never been to ask people to score themselves differently.

Tonight’s exercise, and it works best if you’re honest about your own organization. Think of the most careful process you have, the one with real human judgment in it that slows things down. Now name the competitor who would eat your lunch if they dropped it and you didn’t. If you can name them, you’ve located the actual pressure, and it isn’t coming from any technology. If you can’t name them, ask what protects you: a regulation, a licence, a reputation that would take years to rebuild. Whatever you just named is the only thing standing between that process and next year’s efficiency review.