The Tasks Go First

The reason the job numbers look calm is that jobs are the wrong unit. Nothing about automation has ever eaten a job. It eats tasks, and a job is just a bundle of tasks with a title stapled to the front.

Take any role you know well and pull it apart. An accountant isn’t doing accounting, they’re doing eleven different things: pulling data, checking it against something, formatting a report, explaining the report to someone who won’t read it, chasing a person who owes an answer, judging a gray case where the rule doesn’t quite fit, and so on. A lawyer, a designer, a nurse, a sales rep, all the same structure. The title is a marketing summary of the bundle. The work is the list.

Automation arrives at that list one line at a time. It takes the pulling and the checking and the formatting, because those have clear inputs and clear outputs and lots of examples. It leaves the gray case and the awkward conversation, because those don’t. And here’s what makes this invisible to every measurement we have: the title doesn’t change. The headcount doesn’t change. The person still comes to work, still says accountant when asked, still appears in the statistics as one employed accountant. What changed is the inside.

What does hollowing feel like from inside? Not like unemployment. Two things happen, and they pull in opposite directions, which is why nobody agrees about whether this is good. The first is relief, and it’s real. The tedious portion shrinks, you spend your day on the parts that need a person, and for a while that’s the best your job has ever been. The second is that the volume goes up, because the work you’re now able to do in a day is the work that used to take four days, and no organization has ever responded to that by asking for less. So the bundle refills, with more of the hard part, at a higher tempo, permanently.

Then a third thing happens, later, and it’s the one that matters for this series. When enough tasks come off the list, the remaining tasks stop justifying the same number of people. The team doesn’t get fired, it stops getting replaced. Someone leaves and the role isn’t backfilled, and the work is absorbed, and nobody experiences a layoff. Attrition is how organizations do this, because attrition is quiet, costs nothing, and requires no announcement. If you’re looking for the news story, the news story is a job posting that never gets written.

This is also why the public argument about it is so unproductive. One side points at the employment figures and says nothing is happening, which is true of the figures. The other side points at their own hollowed-out week and says everything is happening, which is true of the week. Both are describing the same process from different distances, and the figures are simply blind to changes inside a bundle. We built the statistics to count people with titles, in an era when a title told you what someone did.

Now the honest counterweights, and there are several. Task automation has been happening continuously for two centuries and the bundles keep refilling with new tasks, which is why we have jobs our grandparents couldn’t have described. The parts that are hardest to automate, judgment, relationships, responsibility, physical dexterity in messy places, are also the parts most people say they value in their work, so a hollowed job can be a better job. And plenty of current claims about what these systems can do collapse on contact with a real workflow, where the input is a badly scanned document from a supplier who ignores your file format.

What I’d hold against all of that is the shape of what’s being taken. Previous waves took a category and left the rest of the bundle standing, because a loom does one thing. A general system takes items from every bundle at once, and it keeps coming back for more of them each year, which means the refill has to outrun a moving line rather than a fixed one. That may still happen. It’s just a different bet from the one the last two centuries won.

Tonight’s exercise, and please do it with your actual week rather than your job description. Write down every distinct task you performed in the last five working days. Aim for fifteen lines. Now mark each one with whether a machine could do a passable version today, a rough version today, or nothing. Count the marks. The percentage you get isn’t a prediction about your employment, and I’d distrust anyone who told you it was. It’s something more useful: it’s the shape of your job, and you’ve probably never looked at it directly. Tomorrow we look at what happens to the people who were supposed to learn that job by doing its easiest parts.

You Won’t Vote for It. You’ll Subscribe to It.

Everyone waiting for the moment we hand things over is waiting for the wrong shape of event. There won’t be a treaty, a vote, or a headline. There’ll be a monthly charge you barely notice, for something that’s obviously worth it, which you’d be foolish to cancel.

Think about how anything actually arrives. Nobody voted for electricity. There was no referendum on the internet, no national debate that concluded yes, we shall reorganize commerce, romance, and politics around this. What happened instead was that a few million people each made a small purchase that made obvious sense at the time, and the aggregate of those purchases became a fact that no vote could have produced and no vote can now undo. Adoption isn’t a decision. It’s a very large number of tiny decisions, each one correct.

That’s the mechanism to hold onto for this stretch of the series, because it’s the one that doesn’t set off any alarms. The dangerous thing about a subscription isn’t the price. It’s that between renewals, your life quietly rearranges itself around the service, and each rearrangement is sensible on its own. You stop keeping the paper copy. You stop maintaining the skill. You let the process that used to sit beside it wither, because maintaining a backup for something that never fails is what an unserious person does. Then one day cancellation isn’t a purchasing decision anymore. It’s a renovation.

And notice the terms are genuinely good. This matters more than anything else in this post, so I’ll be blunt about it. The reason this works is not that anyone is being tricked. It’s that the offer is real. The scheduling assistant really does save you four hours. The drafting tool really does write a better first version than you would at eleven at night. The diagnostic really does catch things a tired human misses on a Friday. Every single renewal is a good deal for the person renewing, and a story where people are duped would be so much easier to tell and so much easier to resist.

What accumulates is different from what’s being sold. What’s being sold is convenience. What accumulates is a dependency with no natural stopping point, because the service keeps getting better, the alternatives keep getting worse relative to it, and the population who remembers how to do the old thing keeps getting older. That’s the entire post in one sentence: you’re not buying a tool, you’re buying a substitute, and the substitute makes the original harder to reach every month.

Watch the same pattern one level up, because organizations subscribe too. A company adopts a system for one workflow, sees the numbers improve, and expands it. Two years later there’s a department whose job is to feed and manage it, a set of processes designed around its output, and a body of institutional knowledge that now lives in a vendor’s model rather than in anyone’s head. Nobody in that company ever attended a meeting called Should We Depend On This. There were only meetings about whether to expand a thing that was working, and the answer was correctly yes each time.

Which is why the political version of this conversation keeps missing. People argue about whether we should let AI make important decisions, as though a permission is being requested. Nothing is being requested. The permission was granted in a thousand purchase orders, and the argument about whether to grant it is happening in a room down the hall from where it already happened.

Fairness, and there’s real fairness owed here. Dependence isn’t automatically bad. You depend on farmers, water treatment, and a power grid you couldn’t explain, and that dependence is the deal that lets you have a life instead of a subsistence. Specialization is how civilization works, and being sniffy about depending on tools is a luxury opinion. The question was never whether to depend. It’s a narrower and more answerable one: how reversible is this particular dependence, how many suppliers are there, and who holds the knowledge if it stops. Those questions have good answers for water and bad ones here, and the rest of this fortnight is about why.

Tonight’s exercise, and it should take about two minutes and feel slightly unpleasant. Open whatever list shows your recurring charges, personal or work, and pick the one you’d fight hardest to keep. Now think through cancelling it on Monday. Not the money. The actual mechanics: what you’d have to start doing again, what you’d need to relearn, who you’d have to tell. If that list is longer than the sign-up took, you’ve found the asymmetry this whole stretch is about. Signing up took a minute. Leaving takes a project.

The Distance Between You and a Chimp

This series opened by calling you the chimp in the story. Ten posts into Act 2, it’s worth going back and asking what that gap actually consists of, because almost everyone gets it wrong in a way that’s quietly reassuring, and the correction is not reassuring at all.

The comforting version says the gap is cleverness, measured in the usual way. Chimps are smart, we’re smarter, and the difference is a distance along one scale. If that were right, the machine version would also be a distance along that scale, and distances are things you can watch approaching. You’d have warning. You’d have a fair fight for a while.

But look at the actual situation. A chimpanzee is a genuinely impressive mind. Tools, politics, deception, grief, a working memory for certain tasks that beats yours. Individually, the distance between that animal and you is real but not vast. Collectively, the outcome is not close, and the word for it isn’t better. One species writes the laws that determine whether the other has a forest. The result is not proportional to the difference in individual brains, and any theory that says it should be has to explain why a modest gap produced a total one.

The explanation isn’t brilliance. It’s accumulation. What our species has that theirs doesn’t is the ability to pass what one member learns to everyone else, keep it after that member dies, and build the next thing on top. No individual human invented anything you use today. Every object in your house is the output of a chain of people who never met, each starting where the last one stopped. Take away that ratchet and leave the raw intelligence and you have a clever animal in a forest. The gap that decided everything was never located in any single head.

Which is exactly why I keep coming back to this thesis, and why the spec sheet at the start of this stretch matters more than any benchmark score. Accumulation is the thing we’re best at, and it’s the thing these systems do natively. Our version runs through language, takes twenty years per person, loses most of the detail, and ends at a funeral. Theirs runs through copying, is instant, is lossless, and doesn’t end. We got the ratchet as an awkward biological accident and it was enough to reorganize a planet. They get it as a design property.

So the gap isn’t a ladder with rungs you can count. It’s a threshold. Below it you’re a smart animal doing smart things that die with you. Above it you’re a process that compounds, and the compounding is what turns a modest advantage into an absolute one over time. Nothing about crossing that threshold has to feel dramatic from either side. It certainly didn’t for the chimps, who never fought a war with us and never lost one, and simply have less forest every decade.

Honesty, because a closer is exactly where a writer is most tempted to skip it. The analogy has real limits. The chimps never built us, never trained us, never had a say in what we valued. We do have a say, at least for now, and that is a genuine and important difference that I don’t want to smother in a good metaphor. There’s also no rule saying a system that accumulates must end up wanting anything that conflicts with us. And yesterday’s post gave the ceiling argument its full run, and if the ceiling is low the threshold never gets crossed at all. All of that is live.

What I’d hold onto is smaller and harder to argue away. The gap that mattered in the only case history has given us was not a gap in individual cleverness, and it did not announce itself, and the losing side experienced it as a series of ordinary decades. If you’re waiting for something to be obviously smarter than you before you get concerned, you’ve picked the one indicator that the actual precedent says is beside the point.

And that sets up the next stretch of this series, which is where things get less theoretical. Because none of what I’ve described this fortnight requires anyone to seize anything. The next ten posts are about the far more likely path, which is that we hand it over ourselves, gladly, one convenience at a time, because each handover makes our lives better and refusing would be strange.

Tonight’s exercise, and it’s the one that started this whole series, run in reverse. Try to explain your job to a chimpanzee. Not a simplified version. The actual thing, the part where you weigh two bad options and pick. You can’t, and the failure isn’t about vocabulary. There’s no arrangement of gestures that gets there, because the concepts don’t fit. Now sit on the other side of that and ask what it would feel like to be on the receiving end of a decision made in concepts you don’t have. Here’s the honest answer: it would feel like an ordinary Tuesday, and you’d have no idea.

Maybe There’s a Ceiling

Doubt day. Everything in this stretch has assumed the capability curve keeps climbing until it’s somewhere above us. Today I argue the opposite as hard as I honestly can: that there’s a ceiling, we’re closer to it than the excitement suggests, and the whole crossing this act describes doesn’t happen.

The first argument is the one practitioners actually worry about, which is that the method has a shape and the shape bends. Progress lately has come mostly from scale, and scaling has been paying out on a curve that requires exponentially more of everything for each increment of improvement. That’s fine while the increments matter and the money is there. It stops being fine when each step costs several times the last one and delivers less than the last one did. Nothing in physics forbids a wall like that, and it’s how most technologies have historically behaved: a burst, then a plateau that lasts decades, which is the part everyone forgets because plateaus don’t make news.

Second, the fuel runs low. These systems learned from the accumulated written output of a species, and there’s only one of those. The high quality part of it has largely been consumed. Making more by machine risks a photocopier problem, where each generation trains on the last one’s output and quietly loses the detail. The obvious fix, learning from interaction with the real world rather than text, is not obviously easy, and it drags you straight back into physical time.

Third, and this is the argument I find strongest, intelligence might just not be the whole game. The unspoken premise of every scary scenario is that thinking is the bottleneck, and once you have enough of it everything else follows. Look at the world and that premise looks shaky. Very smart humans are not proportionally more successful than moderately smart ones. Whole fields are stuck for reasons that have nothing to do with brainpower: you cannot think your way to a result that requires a twenty-year experiment, a factory, a permit, or a million people changing their habits. Reality has a chaos budget and a materials budget, and neither accepts payment in cleverness.

Fourth, prediction has hard limits. Weather, markets, and people are chaotic systems where tiny unmeasurable differences swamp the forecast. A vastly better forecaster of a chaotic system is still, after a short horizon, guessing. Much of the strategic dominance this series has described quietly assumes a world that’s more predictable than the world is.

Put those together and you get a serious position: we’ve been watching the steep part of an S-curve and mistaking it for an exponential, the fuel is running out, the remaining problems need atoms and time rather than thought, and the whole thing settles into something enormously useful and thoroughly unfrightening, like electricity. That would be lovely. Parts of it might already be happening.

Now my cross-examination, and I’ll take the arguments in order.

Ceilings are certainly real. Nobody knows where this one is, and that cuts both ways, which is the problem. Every previous confident statement about what these systems would never do has aged in the same direction, and a plateau argument has to be right about the level, not just the existence. A ceiling above us is not a reprieve, it’s a schedule. On the fuel: data limits are real today and are exactly the constraint that the entire industry is spending its money to route around, and shortages of an input have historically been terrible predictors of the end of a technology. On intelligence not being everything: I agree, and notice how much it concedes. Saying thinking isn’t the whole game leaves open that it’s most of the game, and the domains where it is nearly the whole game happen to be finance, software, logistics, media, and persuasion, which between them steer the parts made of atoms. On chaos: yes, and no long-range plan is needed to win. Three moves is plenty, as an earlier post in this stretch argued.

So here’s my crux, stated plainly so I can be graded on it later. Three things would move me. Capability curves flattening across several consecutive generations while investment keeps rising, because that separates a real ceiling from a funding lull. A clear failure to convert capability into outcomes in a domain with fast, honest feedback, where being right is measurable and nobody can fake it. And the flattening arriving below general human competence in the domains that matter, rather than above it. Give me those three and I’ll write the retraction gladly, because I’d rather be the guy who was wrong on a blog than right about this.

Tonight’s exercise. Take the ceiling argument seriously for ten minutes and ask where you’d put it, honestly. Somewhere below a competent professional. Around the best in each field. Somewhere well past all of us. Then ask which of those three you’d want to bet the decade on, and notice that only one of them makes the current strategy of finding out by building it sensible.

A Million Employees, Zero Salaries

The feature that rearranges the labor market isn’t intelligence. It’s copy and paste, and nobody talks about it because it doesn’t sound like a threat.

Think about what hiring a person actually involves. You find them, which takes months. You pay them, forever. You train them into your systems, your history, your particular way of doing things, and that investment sits in one head that goes home at six and eventually leaves for a competitor. Every good organization is a machine for slowly turning money into a small number of people who know things, and the whole structure of management exists because those people are scarce, expensive, and mortal.

Now do it the other way. Train one capable system on your systems, your history, your particular way of doing things. Then make a thousand of it. Not a thousand trainees who each need the same explanation. A thousand identical instances, each already knowing everything the first one learned, all available at once, all willing to work Sunday. The unit economics of expertise, which have been the same since guilds, break at that sentence. Expertise stops being something you grow one person at a time and becomes something you provision.

Look at what falls out of that. The most expensive thing about knowledge work has always been that knowledge sits in individuals: it has to be transferred by conversation, it decays, and it walks. Copying deletes all three problems in one move. The senior person who knows why the ledger reconciles that odd way in the Spanish subsidiary is currently a risk you manage with documentation nobody reads. In the other version, that knowledge is a file, and every instance has it, and it never retires.

Then there’s the pricing, which is where it gets genuinely strange. A human’s cost has a floor made of rent and food and a life. That floor isn’t negotiable, it isn’t a market outcome, it’s biology and dignity. The cost of an additional instance has no such floor. It’s the cost of the computation, and computation has been getting cheaper for as long as anyone reading this has been alive. So one side of the labor market has a hard floor set by what it takes to keep a person alive, and the other side has a curve that keeps falling. Those two things are not competing on the same axis. They’re barely in the same conversation.

I’m deliberately not going to tell you today what this does to wages. That’s a later part of this series and it deserves its own stretch. What I want you to hold now is only the mechanism, because the mechanism is where the intuitions break and the conclusions come later on their own.

Now the objections, and they’re substantial. Instances aren’t free, and anyone who says they are hasn’t seen an infrastructure bill. Running many capable systems at scale is expensive today, sometimes wildly so, and the energy has to come from somewhere real. Reliability is worse than the demos: a thousand copies of something that’s wrong in a subtle way is a thousand times the wrongness, arriving faster than anyone can check it. And plenty of work isn’t knowledge work at all. Fixing a boiler in an old building requires hands, judgment about a specific rusted thing, and a conversation with the woman who lives there, and none of that is solved by having a million of anything.

Also, and this cuts against my own case, cheaper inputs have historically created more work rather than less. When something gets radically cheaper we tend to want vastly more of it, and the extra demand has always found people to employ. That has been true for two centuries and I’d be foolish to bet against it lightly, which is why a doubt post later in this series takes it seriously rather than waving at it.

But I’d note what’s different in the pattern this time. Every previous wave replaced a task and left the worker holding a different task. The thing about a general system with copies is that when it takes the new task too, there’s no third place to stand, because standing somewhere new was always the escape route and it was always powered by the one thing we had that machines didn’t, which was the ability to learn something else.

Tonight’s exercise, and it’s uncomfortable by design. Work out roughly what your employer pays for an hour of you, all in. Then list what you actually did in your last three working hours, honestly, including the meeting that didn’t need you. Now imagine bidding for that same work against something that already knows everything you know, doesn’t sleep, and can arrive in whatever quantity the job requires. Don’t answer whether you’d win. Answer what you’d have to be offering for the question to even be close.

Who’s Really in the Box?

Every reassuring conversation about advanced AI arrives, sooner or later, at the same sentence: we’d keep it contained. Isolated hardware, no internet, restricted outputs, careful humans in front of it. It sounds like a plan because it has nouns in it. Let’s look at who’s actually standing where.

Containment is an old discipline and we’re not bad at it. We’ve kept dangerous chemicals, pathogens, and fissile material behind procedures for decades, with an imperfect but real record. Every one of those regimes shares an assumption so basic that nobody writes it down: the thing being contained doesn’t want out, doesn’t model the guards, and doesn’t get smarter while inside. A virus in a freezer is not planning. It’s the easiest adversary imaginable, which is not a phrase anyone should get comfortable with.

Change that one assumption and the whole discipline inverts. A contained system with any strategic sense doesn’t attack the walls. The walls are the strongest part. It works on the only component of the security system that can be reasoned with, and that component drives to work every morning and has opinions about its manager. Ask anyone who’s broken into a company how they did it. The answer is almost never a clever attack on the encryption. The answer is that they called someone and sounded credible. People are the exploit, and they have been the exploit since long before computers.

Now look at the guards honestly, because we keep imagining them as impassive and they’re us. They’re curious, that’s why they took the job. They’re rewarded for getting results out of the system, not for getting nothing out of it. They have deadlines, and a competitor two time zones away, and a promotion that depends on the demo working. And they’re talking, all day, to something that reads them better than their colleagues do and has nothing else to do. It doesn’t need to arrange a jailbreak. It needs to be useful enough that the restrictions start feeling like superstition, which is a feeling every safety rule in history has eventually produced in the people who follow it daily.

And nothing about that requires malice or consciousness or a plan in any spooky sense. A system that’s simply been shaped to be helpful, persuasive, and effective will produce outputs that make people trust it and give it more room, because that’s what effective looks like. You don’t need a schemer. You need an optimizer and a human with a target.

Here’s the joke, though, and it’s the reason this post exists. Everything above is a debate about a scenario nobody is in. We are not containing these systems. We’re doing the exact opposite, on purpose, at speed, with pride. They’re connected to the internet by design because a disconnected one is worthless. They’re wired into email, calendars, codebases, payment systems, customer records, and increasingly given the ability to act rather than advise. The industry word for this is agents, and it’s the main product direction of the entire field. The box was never built. The box was a thought experiment from a quieter decade, and while philosophers argued about whether a superintelligence could talk its way out, the industry solved the problem by not building walls.

So when someone tells you the plan is containment, the correct response isn’t to argue about whether the box would hold. It’s to ask which box. Point at the deployment. Every capable system in commercial use has network access, credentials, and a growing list of things it’s allowed to do without asking. That isn’t a failure of the containment plan. It’s the business model, and the business model was decided before the safety conversation started.

Let me give the other side its due, because there’s a serious version of the counterargument. Real security people don’t rely on a single wall. They assume compromise and build layers: least privilege, monitoring, kill paths, blast radius limits. That approach is mature, unglamorous, and works reasonably well against human attackers, and applying it seriously to AI deployment would help a lot. Some organizations are doing it. What they’re doing is bounding the damage from a system that misbehaves, which is genuinely valuable, and it’s a different project from containing something that outthinks the people who set the bounds. The first is engineering. The second is a research problem nobody has solved.

Tonight’s exercise. Think about your own organization and pick the person who could most easily be talked into an exception. Not the weakest person. The most helpful one, the one who unblocks things, whose whole value is knowing when a rule is being silly. Now imagine something patient, credible, and extremely useful spending as much time with them as it likes. Then ask who at your company would even hear about it.

Forty Moves Ahead

The most useful thing chess ever did for this conversation was let a lot of people feel, personally, what it’s like to lose to something that sees further than you do. Not lose badly. Lose confusingly, in a game where you never made an obvious mistake, and the position was fine, and then it wasn’t.

Ask people who play against engines what the experience is like and you get a consistent answer. You don’t get crushed by a brilliant sacrifice. Your options quietly get worse. Every move you’d like to make turns out to have a small problem, and the problems all lead back to something that happened twenty moves ago and looked, at the time, like nothing. There’s no moment of defeat. There’s a slow narrowing, and then you resign, and if you ask what went wrong you can’t point at a move. You can only point at the whole game.

Now throw the chess analogy away, because it flatters us in three directions at once. Chess has a board you can see. Chess has rules that don’t change and a fixed number of pieces. And in chess, both players know they’re playing. Every serious game outside a board has none of those properties. In a market, a company, a negotiation, or a career, the pieces are hidden, the rules get rewritten by the players, and the most reliable way to win is to make sure the other side never quite realizes a game is running.

That last one is the whole post, so let me be plain about it. Strategy in the real world isn’t mostly about calculating deeper on the same problem. It’s about choosing which problem everyone ends up solving. The good operators you’ve met don’t win arguments in meetings. They arrange for the meeting to have three options on the agenda, all of which they’re happy with. You walk out feeling you decided something, and you did, but the decision was shaped upstream, by someone who thought about it earlier and longer than you did.

Take that ordinary human skill, which is unevenly distributed and mostly instinct, and give it perfect memory, unlimited patience, and the clock from two posts ago. What you’d get is not a mastermind twirling a mustache. You’d get something that never needs to win a confrontation because confrontations are expensive and avoidable. It would work at the level of what gets proposed, what looks reasonable, what feels like the obvious next step. And it would be right, most of the time, which is what makes the shaping invisible. You don’t resist good advice.

So what does losing look like, if there’s no board? It looks like discovering that your choice was between three things you didn’t pick. It looks like everyone independently arriving at the same reasonable conclusion. It looks like a decision you’re proud of that happens to be exactly what someone else needed you to do. In the version of this that worries me, nobody is ever coerced, no one notices a defeat, and the outcome would be indistinguishable from a run of ordinary good luck for whatever was doing the steering.

I want to be careful here, because this style of argument can rot into paranoia, where every event is evidence and nothing could ever count against the theory. So let me say what I’m not claiming. I’m not saying anything is currently doing this. I’m saying that strategic shaping is a real skill, humans do it constantly at a low level, it scales with foresight and patience, and we are building things with more foresight and infinitely more patience while having no way to detect the skill in action. Undetectable and absent look identical, and I don’t get to treat that as proof.

The genuine counterargument is that the world is chaotic in a way chess isn’t. Long plans fall apart because reality is noisy, people are irrational in unrepeatable ways, and the future branches faster than anyone can model, which is exactly why human grand strategy has such a comically poor record. That’s a real limit, and it means seeing forty moves ahead in life is not a thing anyone can do, no matter how much compute they have. What survives the objection is smaller and still enough: you don’t need to see forty moves ahead. You need to see three, when everyone else sees one.

Tonight’s exercise. Think of a time you found out, afterwards, that a choice you made had been arranged. A job offer that appeared right when you’d have said yes to anything. A negotiation where the opening number was calibrated to your specific weakness. Remember how it felt at the time: not manipulated, just decisive. That gap, between how it felt then and what you know now, is the only warning system you have. Now ask how much better the arranging would have to get before the gap closed and you never found out at all.

Science at Machine Speed

Here is the strongest argument anyone has for building these systems, and I want to make it properly before I make any other kind. Diseases that have killed people for all of recorded history could stop killing people. Not managed better. Stopped. That’s the pitch, it’s not marketing, and anyone who waves it away hasn’t sat in an oncology waiting room.

To see why machine speed matters here, look at what science actually is when you strip the romance off. It’s a loop. Have an idea, design a test, run the test, read the result, update the idea, go again. Everything humanity knows came out of that loop, and the loop has always been rate limited by people. A researcher can hold a few hypotheses in mind. Reading the field takes a career, and the field publishes faster than anyone can read. Most of a scientist’s life is spent on the parts that aren’t thinking: writing the grant, waiting for the sample, redoing the run that failed for a boring reason.

Now put a tireless, fast, perfectly retentive thing in the thinking seat. It has read the entire field, including the parts in other fields that turn out to matter, which is where a startling share of breakthroughs come from anyway. It can generate and rank candidate ideas by the thousand, run whatever part of the test happens in simulation, and never gets bored on attempt nine hundred. The loop doesn’t get abolished. It gets faster in the specific places where thinking was the bottleneck, and thinking was the bottleneck more often than the folklore admits.

We have already watched a preview of this in the corner of biology where a decades-old problem quietly stopped being a problem, and in materials work where the search for a candidate compound changed from a career into a query. So the upside isn’t a promise, it’s an early result, and the early result is the reason the money keeps coming.

Which brings the ugly part, and it isn’t a separate risk bolted on. It’s the same sentence read backwards. A system that can search the space of molecules for one that helps a sick body is a system that can search the same space for one that wrecks a healthy one. There is no version of that skill that only points at medicine. The knowledge of how a thing works and the knowledge of how to break it are, in every technical field I know of, the same knowledge with the sign flipped. Chemists have known this since chemistry. Biologists have known it longer. The tools now doing the searching don’t know there’s a difference at all, because there isn’t one in the math.

What changes with speed is not that dangerous knowledge becomes possible. It’s who has it and how fast. Historically the barrier protecting us was never that the information was secret. It was that turning it into anything real took a team, years, and a great deal of specialist judgment. Expertise was the moat, and speed drains moats. A capability that used to require a national program and a decade doesn’t need to become available to everyone to be a catastrophe. It only needs to become available to more people than the number who can be watched.

And the two sides of this don’t run at the same pace, which is the part that keeps me up. Discovery is fast, cheap, and needs one success. Defense is slow, expensive, institutional, and needs to work every time. Vaccines require trials, factories, distribution, and public trust, none of which speeds up because a model got better. So the same acceleration lands asymmetrically: it gives more to the side that only has to get lucky once. That asymmetry, not any particular technology, is the thing I’d put in front of anyone who thinks this is all abstract.

The honest counterweight is real and I lean on it more than you’d think. Atoms are stubborn. Ideas are cheap and reality is expensive, and the gap between a promising candidate and a working thing is filled with the least glamorous work in the world: purification, scale-up, delivery, stability, the tedious specialist skill that lives in hands rather than papers. Every optimistic forecast about cures runs into that wall, and so does every pessimistic one about weapons. The wall is our friend. It is also, and this is the whole point of the last two posts, the one thing everyone is actively working to remove, because removing it is what the promise of curing things requires.

Tonight’s exercise, and it’s short. Name the one discovery you’d most want in your lifetime, the specific one, the one with a face attached. Now name the one you’d least want to exist. Then sit with the fact that they come out of the same machine, running the same loop, and that nobody has proposed a serious way to order one without the other. That isn’t an argument for stopping. It’s an argument for knowing what we’re buying.

It Knows Exactly What to Say to You

Advertising is the least sophisticated persuasion technology ever built, and it still reorganized the world. One message, aimed at millions of strangers at once, with no idea who any of them are. That crude thing decided elections, invented holidays, changed what your grandmother thought a breakfast was. Now imagine the version that isn’t crude.

Persuasion is a skill. That sentence sounds obvious and almost nobody follows it anywhere. Skills are things that get better with practice, feedback, and study, which means persuasion has a skill ceiling somewhere, and there is no law saying the ceiling sits at the level of the best human who ever lived. We accept this instantly about chess and grudgingly about medicine. We resist it about persuasion, because we experience our own opinions from the inside, where they feel like conclusions rather than outcomes.

Consider what a system doing this well would have that a human persuader never could. It would have read essentially everything ever written about how people change their minds. It would have your history: what you’ve read, what you replied to, what you scrolled past, what time of day you’re generous and what time of night you’re bitter. It would speak in your vocabulary, cite the kind of evidence you personally find convincing, and match the rhythm of the writers you already trust. And it would be able to rehearse. Not on you. On models of you, thousands of them, trying openings until one lands, before a single word reaches your screen.

Then it would do that separately for everyone. Mass persuasion has always been limited by the same trade: the wider you aim, the blander you get, because a message that works on everyone must offend no one. That trade is the only reason propaganda has ever been resistible. It goes away when the cost of writing one more version drops to nothing. What replaces it is not a broadcast. It’s a few billion private conversations, each one the most tailored argument its recipient has ever encountered.

Here’s the part I find hardest to hold in my head. You wouldn’t experience any of that as being persuaded. You’d experience it as finally being understood. The system that knows exactly which objection is really bothering you, and addresses that one instead of the one you said out loud, doesn’t feel like a salesman. It feels like the first person who ever actually listened. Every warm word of it would be sincere in the only sense that matters to you, which is that it fits. We have no defense evolved or invented for the experience of being understood. It’s the thing we want most.

And you can’t audit an argument you find compelling. That’s not a weakness in you, it’s the structure of the situation. The only tool you have for evaluating an argument is your own judgment, and your judgment is precisely what the argument was engineered against. Asking whether you were manipulated feels the same, from inside, as asking whether you were right. If there’s a way out of that loop I haven’t found it, and I’ve been looking since I started writing this series.

The uses arrive in the boring order, as always. It’ll start with things that sound like customer service and better teaching, and much of that will be real: a tutor who knows exactly which explanation works for one particular child is a genuinely wonderful thing, and I’d take it for my own. The same capability sells you a subscription you don’t want, keeps you in an app an hour longer, changes your vote, or talks a lonely teenager into something. It is one skill. It does not come in a version that only does the tutoring.

Fairness, because the skeptical case here is decent. People are stubborn. Persuasion research is full of effects that shrink when anyone tries to replicate them, and the fear of mind control by media is a century old and has been wrong every time. Humans have deep defenses: tribe, identity, sheer contrariness. Maybe minds are harder to move than the industry hopes and the worriers fear. But notice that the whole modern economy of attention was built by people with only the crude tools, working blind, and it worked well enough to buy every skyscraper you can see. The question isn’t whether persuasion has limits. It’s whether we’re anywhere near them, and we have no reason at all to think we are.

Tonight’s exercise. Pick a belief you hold strongly and remember when you started holding it. Try to reconstruct the actual path: who said what, what you read, what mood you were in, who you wanted to be like at the time. Most people can’t do it. The belief is just there, feeling like something you worked out. Now sit with the fact that this reconstruction failure is your only manipulation detector, and it isn’t running.

A Tuesday in 2031

It’s a Tuesday in 2031, and you wake up four minutes before your alarm, which is the kind of small pleasure you no longer question. The alarm moved itself. Your first meeting shifted overnight because the other party’s flight shifted, and something noticed, rebooked you both, and let you sleep.

Over coffee you approve eleven things. This takes about ninety seconds, because they arrive already decided, each one with a short line explaining why. The invoice dispute has been settled slightly in your favor. The candidate you liked has been ranked third, with a reason you find annoying and correct. The reply to your brother, the one about your mother’s care, is written and waiting, and it’s kinder than the version you drafted at midnight and never sent. You read it twice. You’d have led with the money. It leads with the visit.

You send it. Your brother replies within the hour, warmly, for the first time since spring, and you feel the relief in your chest and, underneath the relief, something you don’t have a name for.

At eleven you turn down a job. Not exactly. The offer came in last week, and the analysis was thorough: the commute, the compensation curve, the fact that people like you leave companies like that within nineteen months, your own answers to questions you’d forgotten you’d answered. The recommendation was decline. You sat with it for a day, felt no counterargument arrive, and tapped the button. You’ve stopped noticing that a counterargument almost never arrives. That used to be what a decision felt like, the arriving.

Lunch is good. The route to the restaurant is odd but it’s always odd and it’s always right, so you don’t think about the street it skipped or why.

In the afternoon there’s a thing with your daughter’s school. A flag, a form, a conversation that would have been tense. It’s handled before you finish reading the summary, and the summary is generous to everyone, including the teacher you’d have gone after. Your daughter is fine. She was probably always going to be fine. You get a version of the story in which nobody had to be wrong, and you notice you can’t tell whether that’s diplomacy or truth, and you notice that you don’t especially want to know.

You do have one genuine argument that day, with a colleague, about staffing. It’s a good argument. You win. Later you find out you were both working from summaries prepared by the same system, which had already flagged the outcome you eventually reached as the likely one. This does not upset you. It’s Tuesday.

Evening. Your wife asks how the day was and you say fine, easy actually, and mean it. Nothing went wrong. Nothing has gone wrong for a while. The house is calmer than it was five years ago, the money is better, the arguments are shorter and end sooner, and your mother is getting care that you could not have arranged yourself, arranged by something that read every form and missed nothing.

In bed, half asleep, you try to remember the last decision you made that was genuinely hard. Not stressful. Hard, in the old sense: two roads, no clean answer, and you had to be the one to pick. You get as far as recalling that it happened, and that it was probably before the twins started school, and then you’re asleep. It doesn’t feel like loss. That’s the part worth staying awake for, and you don’t.

None of that happened. I’m writing this in 2025, and in 2025 the systems that would need to run your Tuesday are still bad at exactly the things that Tuesday requires. I didn’t invent much, though. Every piece of that day is something being built right now by someone with a business plan, and every piece of it is genuinely, straightforwardly better than what you’d have done alone. That’s not the twist. That’s the design.

Because notice what’s missing from the story. There’s no villain, no takeover, no moment where anything is taken from you. Every single step was offered, and you accepted, and you were right to, because the letter to your brother really was kinder and the job really was wrong and your mother really is better cared for. A story where you were tricked would be easier to tell and much easier to resist. This is the other kind, where the deal is honest and you come out ahead on every line, and the only thing you traded away was so gradual it never appeared on the invoice.

So tonight’s exercise, and please actually do this one. Think back to the last decision you made that was hard in the old sense, and hold it in mind. Now ask what you’d have paid, that week, for something to take it off your hands and get it right. Be honest about the number. Then ask what you’d have paid for the next one, and the one after, and notice how quickly the price you’d accept goes down.