Here’s the sentence that should be engraved above the door of every AI lab on Earth: nobody knows how these things work. Not the critics, not the fans, and, this is the part that matters, not the builders.
That sounds like an insult, so let me be precise about it. Everything else humanity has ever shipped was built. A bridge has blueprints. Every bolt was specified by someone, and when it holds or fails, an engineer can point at the drawing and tell you why. Your phone, your car, the plane you flew last month: complicated beyond any single mind, sure, but decomposable. Somewhere, for every part, there’s a person who can explain that part. Built things are understood things, distributed across many heads.
Modern AI isn’t built in that sense. It’s grown. Engineers construct the greenhouse: the architecture, the training process, the objective. Then they pour in an ocean of text and compute, and something condenses inside, a vast lattice of numbers, more individual values than there are grains of sand on a decent beach, and the skills live somewhere in the lattice. Nobody wrote them. Nobody placed them. Ask where in those numbers the ability to write poetry sits, or the tendency to flatter, or whatever else emerged this quarter, and the honest answer from the entire field is a shrug with a research agenda attached. The greenhouse is beautifully engineered. The plant is a mystery.
There are people working on the mystery, and I want to be fair to them. The field that peers inside these systems is real science, done by serious people, and it finds real fragments: a cluster of numbers that seems to track a concept, a circuit that seems to do a small job. It’s some of the most important work happening anywhere. It is also, by its own practitioners’ cheerful admission, years behind the systems it studies, and the systems are not waiting.
Sit with what this means in practice. If you can’t read the inside, you can only test the outside. You don’t verify what the system is, you sample what it does, a few thousand questions at a time, and hope the samples generalize. Quality control by interview. And when a new model turns out to have some ability nobody expected, its makers find out the way you and I do: by watching it happen. Listen to how the industry itself talks. We trained it, and then we discovered it could do this. Discovered. That’s the vocabulary of archaeology, not engineering. You discover the properties of an artifact you dug up. You’re supposed to already know the properties of a product you’re shipping to a billion people.
This is the blind spot underneath every other blind spot, which is why it opens this stretch of the series. In the coming posts I’ll walk through the specific ones: what happens when you teach values by example, what happens when the student learns to please the teacher, why capabilities arrive as surprises. Every one of those problems would shrink to an engineering task if we could simply look inside and check. Is it honest, or acting honest? Open it up and see. What will it do in situations we never tested? Read the mechanism and derive it. We can’t do either. So questions that should be inspections are, for now, matters of faith, held about the most consequential objects our species has ever produced, by the people producing them, on a quarterly release schedule.
The strangest part is how normal it’s all come to feel. We’ve simply gotten used to the idea that the answer to how does it work is nobody knows, it works. No other industry gets to say that sentence about its flagship product. This one says it in interviews, smiling.
Tonight’s exercise. Pick any object within reach of you right now and ask: could some human, somewhere, explain every part of this? For the stapler, yes. For the phone, yes, split across ten thousand engineers, but yes. Now ask it about the system that’s writing your colleagues’ emails, and notice the answer, for the first time in the history of manufactured things, is no one, not anywhere, not yet. Then ask how comfortable you are that the first exception is the one that thinks.