There’s a number that organizations watch obsessively.

It’s called adoption. Or engagement. Or sometimes just “users.” The dashboard shows it with a big upward trend line. The reports celebrate it. “Our AI platform adoption is at 35%, up from 12% last quarter!”

And then the business outcomes don’t improve.

The revenue stays flat. The process doesn’t get faster. The decision quality doesn’t shift. Employees are using the tool, but nothing actually changes.

This is what happens when you measure the wrong thing.

I was working with an organization that had deployed AI to help their sales team qualify leads. The system was built well. It had solid accuracy. The integration was clean. And they measured adoption by how many sales reps logged into the system each week.

Adoption climbed quickly. They celebrated. “67% of our sales team is using the system.”

But the lead closure rate didn’t move.

When they dug deeper, they found something interesting: The sales team was logging in, looking at the AI’s assessment of a lead, and then doing what they would have done anyway. The AI wasn’t changing any decisions. It was just there.

Usage was high. Impact was zero.

Here’s what I’ve learned about measuring AI adoption: The metric that matters isn’t whether people use the tool. It’s whether people make different decisions because of the tool.

That’s harder to measure. It requires connecting the AI’s recommendation to what actually happens. It requires understanding what the person would have done without the AI. It requires separating the signal from the noise.

But it’s the only metric that tells you whether adoption is real.

Let me be concrete about what this looks like. Imagine you’ve deployed an AI system to help with hiring decisions. A recruiter uses it to help screen candidates. The metric everyone watches is “percentage of candidates screened with AI assistance.”

That number says nothing about whether it matters.

What you actually want to know is: “Does the presence of the AI recommendation change which candidates the recruiter advances?”

That’s different. That requires you to compare the recruiter’s decisions with the AI present versus their historical decisions without it. Or to compare their decisions against a control group. Or to look at whether the candidates they advance (using the AI) have different outcomes than the candidates they would have advanced without it.

The recruiter might use the AI for every screening. But if they advance the same candidates they would have advanced anyway, adoption is an illusion.

This distinction matters because organizations often plateau at “everyone is using the tool” before they realize the tool isn’t changing anything.

I watched another organization deploy AI for customer service escalation routing. They wanted the system to recommend which customer issues should be escalated to senior specialists. Good idea. Smart use of AI.

Usage climbed. Adoption metrics looked great. Then someone asked: “Are we actually escalating the right issues?”

The answer was complicated. The system was making recommendations. The customer service reps were using those recommendations. But they were overriding them roughly 40% of the time. And they were also escalating issues the system didn’t recommend.

So the AI wasn’t actually driving the escalation decisions. The human judgment was. The AI was just providing additional information that the humans sometimes considered.

That’s not necessarily bad. But it’s very different from adoption that actually matters. If the system changes decisions, then 40% override rate is valuable. If the system isn’t changing decisions, then it’s just overhead.

Here’s the framework that separates real adoption from usage theater:

Ask: What decision does this AI affect?

The decision could be “should we hire this candidate?” Or “should we approve this loan?” Or “should we serve this ad?” Or “which support tickets should be escalated?” A specific decision.

Then ask: If we compare decisions made with the AI present versus without it, are they different?

This requires a baseline. What would have happened without the AI? Sometimes you can measure this with a holdout group. Sometimes you can look at historical data. Sometimes you can ask the human decision-maker what they would have done.

If the decisions are different, and the change is in the direction the AI was designed to improve, then adoption is real.

If the decisions are the same, then adoption is illusion.

The hard part is that real adoption often plateaus faster than usage adoption. You might get 70% of your organization using the tool, but only 30% of them actually changing decisions because of it. That’s frustrating to report, because it looks like only 30% adoption.

But it’s also the truth. And the truth is more useful than the appearance.

The organizations doing this well have shifted how they measure. They don’t report “percentage of employees using the tool.” They report “percentage of decisions that change because of the AI.” Or more specifically: “In situations where the AI made a different recommendation than the human would have made independently, how often did the AI recommendation win?”

That number is often surprising. It’s often lower than you’d expect. But it’s also the number that predicts business impact.

Here’s why this matters: If you’re measuring usage, you celebrate when the plateau is reached. If you’re measuring decision change, you keep investigating why the plateau exists.

Maybe the AI isn’t good enough yet. Maybe the recommendation isn’t presented in a way humans trust. Maybe the decision context is more nuanced than the AI understands. Maybe the organizational incentives don’t align with the AI’s recommendations.

Whatever the blocker is, you only find it if you’re measuring the thing that actually matters: Did the AI change the decision?

I’ve watched organizations spend millions deploying AI systems and then discover, two years in, that the system isn’t actually changing any decisions. But they have great adoption metrics. Everyone is using the tool.

That’s a very expensive way to learn that adoption and impact aren’t the same thing.

The metric to track is decision change. Not usage. Not logins. Not percentage of employees trained.

Decision change.

It’s harder to measure. It requires thinking carefully about what you’re trying to affect. It requires baseline data or holdout groups or counterfactual analysis.

But it’s the only metric that actually predicts whether the AI investment will pay off.

Everything else is just activity.