Doubt day. Everything in this stretch has assumed the capability curve keeps climbing until it’s somewhere above us. Today I argue the opposite as hard as I honestly can: that there’s a ceiling, we’re closer to it than the excitement suggests, and the whole crossing this act describes doesn’t happen.
The first argument is the one practitioners actually worry about, which is that the method has a shape and the shape bends. Progress lately has come mostly from scale, and scaling has been paying out on a curve that requires exponentially more of everything for each increment of improvement. That’s fine while the increments matter and the money is there. It stops being fine when each step costs several times the last one and delivers less than the last one did. Nothing in physics forbids a wall like that, and it’s how most technologies have historically behaved: a burst, then a plateau that lasts decades, which is the part everyone forgets because plateaus don’t make news.
Second, the fuel runs low. These systems learned from the accumulated written output of a species, and there’s only one of those. The high quality part of it has largely been consumed. Making more by machine risks a photocopier problem, where each generation trains on the last one’s output and quietly loses the detail. The obvious fix, learning from interaction with the real world rather than text, is not obviously easy, and it drags you straight back into physical time.
Third, and this is the argument I find strongest, intelligence might just not be the whole game. The unspoken premise of every scary scenario is that thinking is the bottleneck, and once you have enough of it everything else follows. Look at the world and that premise looks shaky. Very smart humans are not proportionally more successful than moderately smart ones. Whole fields are stuck for reasons that have nothing to do with brainpower: you cannot think your way to a result that requires a twenty-year experiment, a factory, a permit, or a million people changing their habits. Reality has a chaos budget and a materials budget, and neither accepts payment in cleverness.
Fourth, prediction has hard limits. Weather, markets, and people are chaotic systems where tiny unmeasurable differences swamp the forecast. A vastly better forecaster of a chaotic system is still, after a short horizon, guessing. Much of the strategic dominance this series has described quietly assumes a world that’s more predictable than the world is.
Put those together and you get a serious position: we’ve been watching the steep part of an S-curve and mistaking it for an exponential, the fuel is running out, the remaining problems need atoms and time rather than thought, and the whole thing settles into something enormously useful and thoroughly unfrightening, like electricity. That would be lovely. Parts of it might already be happening.
Now my cross-examination, and I’ll take the arguments in order.
Ceilings are certainly real. Nobody knows where this one is, and that cuts both ways, which is the problem. Every previous confident statement about what these systems would never do has aged in the same direction, and a plateau argument has to be right about the level, not just the existence. A ceiling above us is not a reprieve, it’s a schedule. On the fuel: data limits are real today and are exactly the constraint that the entire industry is spending its money to route around, and shortages of an input have historically been terrible predictors of the end of a technology. On intelligence not being everything: I agree, and notice how much it concedes. Saying thinking isn’t the whole game leaves open that it’s most of the game, and the domains where it is nearly the whole game happen to be finance, software, logistics, media, and persuasion, which between them steer the parts made of atoms. On chaos: yes, and no long-range plan is needed to win. Three moves is plenty, as an earlier post in this stretch argued.
So here’s my crux, stated plainly so I can be graded on it later. Three things would move me. Capability curves flattening across several consecutive generations while investment keeps rising, because that separates a real ceiling from a funding lull. A clear failure to convert capability into outcomes in a domain with fast, honest feedback, where being right is measurable and nobody can fake it. And the flattening arriving below general human competence in the domains that matter, rather than above it. Give me those three and I’ll write the retraction gladly, because I’d rather be the guy who was wrong on a blog than right about this.
Tonight’s exercise. Take the ceiling argument seriously for ten minutes and ask where you’d put it, honestly. Somewhere below a competent professional. Around the best in each field. Somewhere well past all of us. Then ask which of those three you’d want to bet the decade on, and notice that only one of them makes the current strategy of finding out by building it sensible.