It Knows Exactly What to Say to You

Advertising is the least sophisticated persuasion technology ever built, and it still reorganized the world. One message, aimed at millions of strangers at once, with no idea who any of them are. That crude thing decided elections, invented holidays, changed what your grandmother thought a breakfast was. Now imagine the version that isn’t crude.

Persuasion is a skill. That sentence sounds obvious and almost nobody follows it anywhere. Skills are things that get better with practice, feedback, and study, which means persuasion has a skill ceiling somewhere, and there is no law saying the ceiling sits at the level of the best human who ever lived. We accept this instantly about chess and grudgingly about medicine. We resist it about persuasion, because we experience our own opinions from the inside, where they feel like conclusions rather than outcomes.

Consider what a system doing this well would have that a human persuader never could. It would have read essentially everything ever written about how people change their minds. It would have your history: what you’ve read, what you replied to, what you scrolled past, what time of day you’re generous and what time of night you’re bitter. It would speak in your vocabulary, cite the kind of evidence you personally find convincing, and match the rhythm of the writers you already trust. And it would be able to rehearse. Not on you. On models of you, thousands of them, trying openings until one lands, before a single word reaches your screen.

Then it would do that separately for everyone. Mass persuasion has always been limited by the same trade: the wider you aim, the blander you get, because a message that works on everyone must offend no one. That trade is the only reason propaganda has ever been resistible. It goes away when the cost of writing one more version drops to nothing. What replaces it is not a broadcast. It’s a few billion private conversations, each one the most tailored argument its recipient has ever encountered.

Here’s the part I find hardest to hold in my head. You wouldn’t experience any of that as being persuaded. You’d experience it as finally being understood. The system that knows exactly which objection is really bothering you, and addresses that one instead of the one you said out loud, doesn’t feel like a salesman. It feels like the first person who ever actually listened. Every warm word of it would be sincere in the only sense that matters to you, which is that it fits. We have no defense evolved or invented for the experience of being understood. It’s the thing we want most.

And you can’t audit an argument you find compelling. That’s not a weakness in you, it’s the structure of the situation. The only tool you have for evaluating an argument is your own judgment, and your judgment is precisely what the argument was engineered against. Asking whether you were manipulated feels the same, from inside, as asking whether you were right. If there’s a way out of that loop I haven’t found it, and I’ve been looking since I started writing this series.

The uses arrive in the boring order, as always. It’ll start with things that sound like customer service and better teaching, and much of that will be real: a tutor who knows exactly which explanation works for one particular child is a genuinely wonderful thing, and I’d take it for my own. The same capability sells you a subscription you don’t want, keeps you in an app an hour longer, changes your vote, or talks a lonely teenager into something. It is one skill. It does not come in a version that only does the tutoring.

Fairness, because the skeptical case here is decent. People are stubborn. Persuasion research is full of effects that shrink when anyone tries to replicate them, and the fear of mind control by media is a century old and has been wrong every time. Humans have deep defenses: tribe, identity, sheer contrariness. Maybe minds are harder to move than the industry hopes and the worriers fear. But notice that the whole modern economy of attention was built by people with only the crude tools, working blind, and it worked well enough to buy every skyscraper you can see. The question isn’t whether persuasion has limits. It’s whether we’re anywhere near them, and we have no reason at all to think we are.

Tonight’s exercise. Pick a belief you hold strongly and remember when you started holding it. Try to reconstruct the actual path: who said what, what you read, what mood you were in, who you wanted to be like at the time. Most people can’t do it. The belief is just there, feeling like something you worked out. Now sit with the fact that this reconstruction failure is your only manipulation detector, and it isn’t running.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.