You know the feeling of climbing stairs in the dark and misjudging the last step. Your foot expects floor and finds air, or expects air and slams into floor, and your whole body lurches with the error. Keep that lurch in mind. It’s the characteristic feeling of modern AI development, including for the people doing the developing.

Here’s the pattern, repeated enough times now that it qualifies as the field’s signature. A team scales a system up, more data, more compute, expecting the usual gentle improvement. Mostly they get it. And then, some fraction of the time, a skill that wasn’t there is simply there. An ability nobody programmed, nobody predicted, and nobody ordered arrives with the new model like a stowaway. The industry has a bloodless phrase for these, emergent capabilities, which makes them sound like a feature. Translated into plain English, the phrase means: our product does things we found out about after we made it.

The surprises run in both directions, which is somehow not comforting. Skills experts predicted for next decade arrive on a Tuesday. Skills everyone assumed were around the corner stall for years. The people with the most information, standing closest to the machines, running the actual experiments, keep being wrong in public about their own systems’ near future, and to their credit, many of them say so cheerfully. Take that in properly: the insiders can’t predict next year’s capabilities. Which means every confident timeline you’ve ever read, the soothing ones and the terrifying ones alike, was decoration. The honest forecast is a shrug with error bars, and the error bars contain everything.

Now connect this to safety, because that’s where the stairs get expensive. Nearly every plan for handling dangerous capabilities has the same load-bearing phrase somewhere in it: when we see it approaching. We’ll prepare defenses when the threat gets close, regulate when the risk is demonstrated, pause when the warning signs appear. Every version of that sentence assumes capability growth is a ramp, visible ahead, walkable at a measured pace. If it’s stairs in the dark, the first sign of the capability is the capability. There is no approaching, there’s before and after, separated by one training run. You don’t get to schedule your reaction to a surprise. That’s what the word means.

It stacks with last post’s problem in an unpleasant way. The capabilities we most need advance notice of, better deception, better modeling of evaluators, strategic patience, are exactly the ones that wouldn’t announce themselves on a dashboard even after arriving. A jump in poetry is obvious. A jump in knowing when it’s being watched looks like nothing at all. Looks, in fact, like an unusually good safety report.

Now, the fair-play paragraph, because there’s a real counterargument and it deserves daylight. Some researchers argue the jumps are partly an illusion of measurement: underlying skill grows smoothly, but our tests are pass-fail cliffs, so smooth progress registers as sudden leaps. The stairs, on this view, are a ramp we’re measuring badly. It’s a good argument. It might be right. And I invite you to notice what it actually offers: either the world jumps, or our instruments are too crude to see the ramp the world is climbing. For planning purposes those are the same sentence. A surprise caused by reality and a surprise caused by your speedometer both arrive as surprises.

What would help is boring and familiar by now: instruments built by people whose bonus doesn’t depend on the answer, evaluation science that matures faster than the systems, someone outside the building watching the staircase full time. What we have instead is the lurch, quarterly, absorbed each time as a fun surprise because so far every step has held.

Tonight’s exercise. Picture pouring water into an opaque kettle with no gauge, on a stove whose dial someone else controls, and your job is to call the boil in advance. You’ve never seen this kettle. Nobody has. Now notice that your written plan for the boiling-over scenario begins with the words when we see it coming, and ask yourself what, in this kitchen, seeing would even mean. That plan is the plan. Sleep well.