Your legal team has thought about AI liability. They’ve probably built a framework around it.

They probably haven’t thought hard enough about the scenario where the liability is clear and there’s nothing to sue.

This is the quiet liability that keeps me engaged with this space: not what happens when someone gets wronged by your AI system, but what happens when everyone knows someone got wronged and there’s nobody to hold accountable.

The accountability dissolution problem

Here’s the pattern: Your organization deploys an AI system. It makes a bad decision. Someone is harmed. The decision was technically made by the AI system, which is a tool.

Your company says: “We didn’t make the decision. The model did. We relied on the model’s recommendation because it had been validated and it was operating as designed.”

The individual who was harmed says: “I was harmed. Someone should fix this.”

Your company says: “We didn’t intend harm. We built reasonable controls. The model was performing as expected.”

The individual says: “I don’t care about your controls. I was denied credit, fired, or denied medical treatment. Someone should be accountable.”

At this point, the organization has an accountability void. The individual was harmed. The organization says it didn’t make the decision (the model did). So who made the decision?

Usually the answer is: nobody. The decision happened. Nobody made it. Nobody can be held accountable.

This is different from negligence liability, where you could argue the organization was careless. This is structural liability, where you could argue the organization constructed a system where decisions happen without anyone deciding.

Why this matters legally

You might think: “This is actually good for us. If nobody made the decision, nobody can be held liable.”

That’s backwards. Courts and regulators care about this differently than you’d expect.

The emerging pattern in how regulators and courts approach this is: If you’ve deployed a system that makes decisions affecting people, someone at your organization bears responsibility for those decisions.

The legal challenge is that companies are trying to distribute responsibility so widely that it disappears. The model recommends. A human reviews (but only if there’s time). The system flags concerns (but only for certain kinds of concerns). Business rules override the model (in certain cases).

At each step, someone is supposed to be responsible. In practice, nobody is.

A lending officer reviews a model recommendation for a loan that gets denied. If they just accept the recommendation without independent judgment, they haven’t made a decision. They’ve deferred to the model. If something goes wrong, can you hold them accountable for a decision they didn’t make? Not really.

A compliance officer reviews the model for bias. They check statistical metrics, see that the model is performing as expected overall, and approve deployment. Six months later, the model is producing disparate impact. The compliance officer says: “I reviewed it carefully and it was performing as expected.” The business says: “We followed the compliance officer’s assessment.”

Who’s accountable? Nobody really. Everybody did their job.

How regulators are approaching this

The regulatory approach is starting to shift. Instead of asking “who made the bad decision,” they’re asking “why did your system make a bad decision without anyone noticing?”

The focus is moving from individual accountability to system accountability.

Organizations that have been through regulatory reviews often discover a pattern. A regulator asks: “Your lending model denied this application. Walk me through how someone with decision authority reviewed this recommendation.”

The organization responds: “Our model is trained to be reliable. We monitor it quarterly. If it falls below performance thresholds, we retrain.”

The regulator pushes back: “That’s not what I asked. Someone made this specific decision. Tell me who. Tell me what they knew. Tell me what decision authority they had.”

The organization can’t answer, because nobody actually had decision authority for that decision. The model recommended it. If the recommendation fell outside certain parameters, it escalated to a human. But in this case, it didn’t. So the decision happened without anyone deciding.

The regulatory position is clear: “You can’t build a system where decisions happen without someone deciding. Either the model has authority (in which case you need to defend the model’s decision) or someone has authority (in which case you need to show they exercised it).”

This is the liability that most companies haven’t thought through. Not “the model was wrong,” but “nobody had the authority to decide this, yet a decision was made.”

Why traditional frameworks miss this

Most companies approach AI liability through the lens of traditional product liability. You build something, it fails, you’re liable if you were negligent. Your defense is: “We were careful. We tested it. We built reasonable controls.”

This works fine for physical products. A car has a brake failure, you’re liable if you failed to test the brakes adequately.

It doesn’t work for AI systems because the question isn’t just whether the system was tested, but whether anyone was actually responsible for using it.

A framework that addresses traditional product liability but doesn’t address decision authority liability is incomplete.

Organizations usually have the product liability framework: testing, monitoring, control standards. They rarely have the decision authority framework: who decides what, under what authority, with what information.

What actually protects you

The liability structure that actually protects an organization isn’t “we have no responsibility because the model decided.” It’s “someone in the organization had clear authority and responsibility, and they made a conscious decision.”

This sounds crazy until you realize what it means: You’re more protected legally if you have one person who decides “we’re going to use this model and here’s why I’m comfortable with that decision” than if you have elaborate controls but nobody clearly deciding.

The elaborate controls are good. But they don’t protect you if they’re controls without decision authority.

Organizations that handle this well usually:

1) Make decision authority explicit

There’s a person (not a committee, ideally not a meeting) who has the authority to decide “we use this model for this decision.”

2) Document the conscious choice

That person writes down (or records) why they made that choice. What did they know about the model? What risks were they accepting? What monitoring would they put in place?

3) Hold people accountable for that decision

If something goes wrong, the question is: Was that decision reasonable given what was known at the time? Not: Did the model fail? Did the model work properly?

4) Design escalation appropriately

If the model produces an unexpected outcome, there’s a clear escalation path. Someone with authority reviews it. They decide whether to override, whether to investigate further, whether the original decision authority still stands.

The liability protection comes from showing that someone thought about the risks and decided to proceed anyway. Not from showing that you had no responsibility.

What this means for governance

This fundamentally changes how you should think about AI governance.

It’s not just about frameworks and controls. It’s about clear decision authority and conscious choice.

If your governance structure is a matrix of approvals where nobody clearly decides, you’re building liability, not protecting against it.

If you have clear decision authority and clear documentation of conscious choice, you’re in a much better position.

This is why the accountability structure from Day 31 matters. If you don’t have clear decision authority and clear escalation processes, you’re not just failing at governance—you’re building legal exposure.

Where to start

If you have AI systems in production, start here:

For each system, answer: – Who has the authority to decide whether we deploy this? – What did they know about the model’s risks when they decided? – If something goes wrong, who reviews that outcome and decides what to do?

If you can’t answer these clearly, you have a liability problem. Not because your model is bad, but because nobody decided to use the model.

Fix that first, before you invest in more controls or more comprehensive frameworks.

Your SIEM team knows how to monitor systems. They monitor uptime, latency, error rates, resource utilization. They know what “the system is healthy” means.

They probably don’t know how to monitor whether an AI system is making good decisions.

These are different problems, and the confusion between them is creating a gap in enterprise AI governance that most organizations don’t realize they have.

Why system monitoring isn’t enough

Here’s the pattern I see: An organization deploys an ML model. They integrate it into their system. The operations team sets up monitoring. They monitor whether the model API is responding (yes), whether predictions are generating (yes), whether the system has adequate compute resources (yes).

Everything looks fine. Six months later, the model is systematically making worse decisions than it was when it deployed. But the operational monitoring shows no problems. Uptime is perfect. Latency is clean. Resources are healthy.

The model is failing invisibly.

This happens because system monitoring answers different questions than behavioral monitoring.

System monitoring asks: Is the system running? Is it available? Is it processing data? Are resources adequate?

Behavioral monitoring asks: Are the model’s decisions changing? Are they still making sense? Are there groups getting systematically different treatment? Is the model still producing what we designed it to?

A system can be perfectly healthy from a system perspective and producing degraded outputs from a behavioral perspective. These require different monitoring approaches.

What behavioral monitoring actually measures

Organizations that get this right usually monitor multiple layers:

1) Statistical drift

Is the model’s accuracy degrading? Are its predictions diverging from expectations? This is the most straightforward behavioral monitoring and most organizations try to do it.

It’s also insufficient. A model can maintain perfect accuracy overall while failing systematically in specific segments. A lending model might be as accurate as ever, but recommending worse terms to certain demographic groups than others. A hiring model might be as accurate overall but recommending different candidates for certain job categories.

2) Outcome monitoring

Are the decisions the model drives producing expected outcomes? If the model is recommending treatment for patients, are those patients getting better? If the model is recommending job candidates, are those candidates succeeding in the role?

This is harder to measure because outcomes take time. But it’s more important. A model can be statistically accurate while driving poor business outcomes because the decision context has changed.

3) Behavioral fairness

Are segments of the population getting systematically different treatment? Are predicted probabilities distributed differently across groups? Are decision rates or recommendation rates different?

This requires benchmarking. You need to know what “fair” means for your use case. Then you monitor whether the model is maintaining that standard.

Most organizations don’t do this routinely. They check for bias during validation, assume it stays stable, and move on.

4) Adversarial monitoring

Are there patterns in the data that the model hasn’t learned? Are there adversaries trying to game the system? Are there edge cases the model handles poorly?

This is the most sophisticated monitoring, and it requires building time and resources into the process. But it’s critical for systems that are working in adversarial environments (fraud detection, abuse prevention, security).

Where most organizations fail

The failure usually happens at the organizational boundary. Your operations team owns system monitoring. Your data science team might own model validation. Your business team owns outcomes. But nobody owns behavioral monitoring continuously.

So what happens is:

• Operational monitoring is excellent. You’d know instantly if the API goes down.
• Model validation happened once at deployment. It’s not continuously updated.
• Outcome monitoring happens in business reviews. It’s infrequent and high-level.
• Behavioral fairness monitoring is either missing or ad-hoc.

The system is set up to catch when a model fails to produce predictions. It’s not set up to catch when a model produces degraded predictions.

I watched a company deploy a model to help prioritize customer support tickets. The system monitoring showed perfect health. The model was generating predictions for 100% of tickets. The infrastructure was solid.

But six months in, the model was systematically de-prioritizing tickets from customers in certain regions. The accuracy was stable (it was still correctly predicting urgency). The decision pattern had shifted (urgent tickets from one region were being treated as less urgent). Nobody noticed because outcome monitoring was infrequent, and behavioral monitoring didn’t exist.

When they finally caught it through a customer complaint, they had months of bad decisions to remediate.

Building behavioral monitoring

If you’re thinking about how to set this up, the pattern that works usually involves:

Establish baselines

Before deploying the model, understand what “normal” behavior looks like. What’s the accuracy distribution? What are the outcome rates? What does the decision distribution look like across important segments?

This baseline becomes the reference point for monitoring.

Continuous measurement

Set up dashboards that track the metrics that matter: – Accuracy and its distribution across segments – Outcome rates (how often recommended actions succeed) – Decision distribution (is treatment/recommendation rate changing?) – Fairness metrics (is outcome distribution stable across protected attributes)

These should be tracked as continuously as your system health metrics.

Alerting on meaningful changes

Accuracy might vary by day or week. That’s fine. Accuracy systematically degrading is not fine. Set up alerts that trigger when meaningful changes occur.

The challenge is defining “meaningful.” A 1% accuracy drop in one segment might be noise. A 5% accuracy drop in another might be significant. It depends on your use case.

Postmortem culture

When something goes wrong, treat it like a system incident. What changed in the environment? What changed in the data? Was the model’s behavior appropriate given the change? Would monitoring have caught this earlier?

Use the postmortem to improve the monitoring for next time.

The hard part

The hard part isn’t the technology. You can build behavioral monitoring dashboards. This challenge exists across the observability landscape, regardless of tooling choice. The hard part is organizational. You have to decide:

Who owns monitoring? Not system health. Behavioral health. Who’s responsible for knowing whether the model is still making good decisions?

If the answer is “the data science team,” they need time and resources for continuous monitoring. They can’t just build the model and hand it off.

If the answer is “the operations team,” they need to understand that model monitoring is different from system monitoring, and they need to learn how to do it.

If the answer is “nobody,” you’re setting yourself up for the invisibly degrading model.

Most organizations avoid making this decision, which is why most organizations have gaps in their behavioral monitoring.

Where to start

Pick your highest-impact model. Pick 3-4 behavioral metrics that matter for that model: – Accuracy (or whatever your primary success metric is) – Something about equity (decision rates across groups, outcome distribution) – Something about the business (customer satisfaction, employee performance, whatever the model is trying to optimize for)

Set up a dashboard that shows these metrics over time. Look at it every week. When something changes, understand why.

That’s behavioral monitoring. It’s not complicated. But it requires someone to own it and actually do it.

The organizations that get this right know what their models are actually doing, not just that they’re running. The ones that don’t usually find out when things have already gone wrong.

A reasonable question to ask of any AI system in production: “What data did this model train on?”

Most large organizations cannot answer that question with specificity.

This is a bigger problem than it sounds. It’s not just an audit inconvenience. It’s a blocker to understanding your own model’s behavior, managing its risks, and defending its decisions.

Why this matters more than most organizations realize

Imagine a scenario that happens more often than anyone discusses publicly: Your lending model makes a recommendation that looks wrong for a specific demographic segment. The recommendation is mathematically sound. The model is performing as expected overall. But in this segment, something is off.

Your first question should be: What data did the model train on for this segment?

If you know the answer — “We trained on 47,000 approved loans in this geography from 2018-2020, and here’s the selection bias in that dataset” — you can investigate. Maybe the selection bias is the problem. Maybe the economic conditions of that time period are no longer representative. Maybe lending practices have changed.

If you don’t know the answer, you’re stuck. You can look at accuracy metrics (which look fine) and feature importance (which shows expected patterns) and conclude nothing is wrong. But something feels wrong. You just can’t diagnose it because you don’t know where the model’s learned patterns came from.

This happens more than you’d think. And when it does, the organization usually takes one of three paths:

Path 1: They remove the feature that seems problematic, retrain the model, and hope the problem goes away. They’ve changed the decision-making process without understanding why.

Path 2: They add a rule (don’t recommend this for this segment) that overrides the model when it hits the problematic pattern. They’ve built a patchwork system that’s harder to maintain and monitor.

Path 3: They accept the problem and let the model make the recommendation anyway, documenting that they’re aware of the concern. They’re betting that enough models get questioned that auditors won’t focus on this one.

None of these are good outcomes. All three are symptoms of not understanding your training data.

Where data provenance usually fails

Organizations typically get tripped up in three places:

1) Data collection and compilation

You’re training on multiple datasets. You’re using internal transaction history, third-party enrichment data, and purchased datasets. These come together in a data pipeline. Months or years later, someone asks: “What exactly went into the training set?”

The answer usually involves hunting through old notebooks, asking data engineers who’ve moved to other teams, or looking at Hadoop jobs that have been rerun dozens of times with different parameters. What you want to know is clear and simple. What you can actually discover is murky and partial.

Most organizations solve this by documenting it after the fact, which is fine until the documentation is incomplete or wrong. Better organizations build data versioning into the pipeline from the start. They tag versions, store lineage information, and treat training data as an artifact that can be retrieved.

2) Data transformation

The raw datasets get processed. They’re cleaned, features are engineered, outliers are handled. This transformation process is usually in code. That code evolves. Parameters change. One version of the code created the training data. A later version handles new data differently. Now you’ve got a production system where data is processed one way, and historical data was processed another way.

The question becomes: Is the model failing because the environment changed, or because we changed how we process the input?

This is fixable but requires discipline. You need to version the feature engineering code just like you version the model. You need to be able to say “this model trained on data processed by version 2.3 of the ETL pipeline.” Then when something goes wrong, you can reproduce that exact data processing.

3) Consent and licensing

More problematic: You’ve assembled training data from multiple sources, and some of those sources have licensing restrictions or data usage agreements. You trained the model with data you shouldn’t have had access to, or you used data in a way that violated the agreement.

This is harder to discover after the fact. The model is in production. The historical data is mixed with other data. You can’t untrain what you learned. You either acknowledge the problem (legal exposure), or you hope nobody notices.

The organizations that avoid this usually have data governance at the point of procurement. Before data comes into the pipeline, they know: Where did it come from? What can we use it for? Are there restrictions?

What working data provenance actually looks like

These are emerging industry best practices. Organizations that handle this well usually have systems that let them:

Answer the basic questions

“What data trained this model?” You should be able to specify a dataset version or date range and get back a manifest of what went into training.

“Has that data been modified?” You should be able to tell whether the data was processed differently at training time vs. now. You should be able to reproduce the historical processing if you need to.

“What are the licensing and consent restrictions?” You should know which parts of the training data have restrictions and what those restrictions are.

These aren’t complicated questions. They’re just hard to answer in systems that weren’t built with provenance in mind.

Link provenance to model behavior

“We know this model trained on 2018-2020 data from this geography. Here’s what economic conditions and lending practices looked like then. Here’s what they look like now. Here’s where we’d expect the model to diverge.”

This conversation requires you to know the training data. Then you can make an argument: “The model learned patterns from this time period and this context. If the context has changed, we should expect divergence.”

Establish audit trails

If data provenance is clear, auditing becomes more straightforward. You can trace decisions back to training data. If something looks wrong in a cohort, you can investigate whether that’s a modeling issue or a data bias issue.

Regulators are increasingly asking for this. “Your model recommended X for this segment. Tell us about the training data for that segment and why you think the recommendation is fair.” If you can’t answer, that’s a problem.

How to start fixing this

If you have models in production and you can’t confidently answer “what data trained this model,” you’re not alone. This is an endemic problem in large organizations.

Start with the highest-impact models. For each one, invest in understanding the training data:

Document it. Write down what datasets went in, where they came from, what processing happened. Even if it’s incomplete, it’s better than nothing.

Version it. From this point forward, treat training data and feature engineering as versioned artifacts. Tag them. Store them. Know what version produced what model.

Connect it to decisions. When something goes wrong, ask whether the decision-making divergence is because the model failed or because the environment changed. You can’t answer that without knowing the training data.

Link it to governance. Your model risk management should include understanding the training data and its limitations. “The model is statistically valid” isn’t enough if you don’t understand what it learned.

Data provenance takes effort. But it’s the foundation for understanding your own models and managing their risks. Organizations that have built it out are the ones that can confidently explain their model decisions. The ones that haven’t are the ones that eventually end up in uncomfortable regulatory meetings asking the question they can’t answer: “What was your model actually trained on?”

There’s a type of organizational problem that shows up in almost every large company that tries to scale AI decision-making. I call it the escrow problem because the decision gets stuck in a state where it’s technically made but functionally unmade.

Here’s how it usually starts: Your organization deploys a model. The model makes recommendations. The humans are supposed to review those recommendations before decisions are finalized. This is good risk management.

Then the model starts producing recommendations faster than the review process can handle them. And something breaks.

How the problem reveals itself

I watched it happen at a company with a lending model that ran daily. The model analyzed applications and generated recommendations. For each recommendation, there was an escalation process: If the model’s confidence was below a certain threshold, a human reviewer had to approve before the decision could be finalized.

This was a reasonable control. Humans would review uncertain cases and apply judgment.

The problem: The model was running faster than the humans could review. By the time a human looked at a recommendation, the business context had changed. A customer’s credit situation might have moved. Market conditions might have shifted. The recommendation that made sense when it was generated was now stale.

The company had two choices. They could slow down the model (run it less frequently). Or they could let decisions move forward without full review.

They did what most organizations do: They expanded the category of decisions the model could make without human review. They raised the confidence threshold. They created exception processes. They added more reviewers.

They never solved the fundamental problem: The decision authority couldn’t move as fast as the decision-generating process needed it to move.

Why this matters

The escrow problem has real consequences, and they’re usually invisible until something breaks.

First, it erodes decision quality. When humans are reviewing decisions that are already hours old, they’re not reviewing whether the decision was right then — they’re reviewing whether it’s still right now. These are different judgments. The review becomes reactive (the decision is already in flight) instead of deliberative (should we make this decision?).

Second, it creates decision drift. The organization sets out with human oversight for all AI recommendations. Gradually, as the volume overwhelms the review process, the definition of which decisions require oversight changes. But there’s no formal decision about this drift. It happens through accumulated exceptions and threshold adjustments. Six months in, the decision-making process is fundamentally different from what was designed, but nobody made a choice about it.

Third, it creates liability exposure. The organization built a control system (human review) and published that as their governance. When something goes wrong in a decision that bypassed the review, they have to explain why the published process wasn’t followed. Usually the answer is “our volume exceeded our capacity,” which is not a good explanation in a regulatory meeting.

What’s really happening

The escrow problem is usually a symptom of something deeper: The organization hasn’t actually decided who has the authority to make decisions.

It sounds like they have: “The model recommends, humans review, we finalize.” But what they’ve really said is: “The model recommends, humans might review if we have time, and then we do whatever happens next.”

That’s not decision authority. That’s hoping someone with judgment will be available.

In organizations that handle this well, decision authority is clear and it’s mismatched to speed:

Fast decisions — The model decides, using its learned judgment. Humans monitor whether the model’s decisions are working as expected, but they’re not involved in every decision. This is appropriate for high-volume, low-stakes decisions where the model’s track record of accuracy is proven.

Medium-speed decisions — The model recommends, and a human approves or overrides based on specific criteria. This requires that the human can make the decision in the timeframe the model generates recommendations. If they can’t, you have the escrow problem.

Slow decisions — Humans make the decision. The model provides input (analysis, risk assessment, comparable cases), but the human has the final authority and the time to exercise genuine judgment. This is appropriate for low-volume, high-stakes decisions.

The problem is that most organizations try to run medium-speed decisions through a slow decision process. They want the model’s speed, the human’s judgment, and the risk mitigation of oversight. They can’t have all three.

Choosing what to give up

Organizations that solve this problem usually make one of three choices:

Choice 1: Give up speed.

Slow down the model. Run it weekly instead of daily, or quarterly instead of monthly. Make sure the review process can keep up. This is the right choice if the business value of speed is low and the risk of escrow is high. It’s usually chosen by regulated organizations where decision documentation is critical, or where the stakes are high.

Choice 2: Give up human oversight.

Let the model decide for high-confidence cases. This is appropriate when the model’s track record is proven and the cost of wrong decisions is manageable. It requires confident monitoring (you actually know when the model fails) and a fallback process (if something goes wrong, there’s a clear escalation). It’s the choice many companies make when they realize their volume will never fit through human review.

Choice 3: Give up some decisions.

Don’t deploy the model for use cases where you can’t deliver fast enough human judgment. Keep it as an internal tool (gives humans better information) rather than a decision system (actually makes decisions). This is the right choice when you’re not sure about the model’s track record or when the stakes are high.

Most organizations don’t make any of these choices explicitly. They drift into Choice 2 (give up human oversight) while claiming they’re still running Choice 1 (humans review) and never actually doing Choice 3 (accepting limited scope).

How this connects to everything else

The escrow problem is actually a decision-rights problem. It shows up when you haven’t clearly assigned decision authority and you haven’t accepted the constraints that come with that assignment.

Organizations that have solved it clearly are usually clear about something else: Who actually owns the decision? Not “who reviews the recommendation,” but who has the power and the responsibility to say yes or no?

Once you know that, the speed problem becomes tractable. If the model owner has the authority to make decisions within certain guardrails, they can decide to expand those guardrails when volume exceeds human review capacity. It’s their call to make. If human reviewers have the authority, they can decide how much volume they can handle and what happens when volume exceeds capacity.

But most organizations have assigned the authority to someone in a governance role (a committee, a compliance team, a risk committee) who isn’t actually involved in day-to-day decision-making. So when volume increases, nobody has the authority to make the choice about what to do.

Where to start

If you’re building AI decision systems, ask this question early: What’s the slowest decision-making process you can tolerate for this use case?

If the answer is “we need to decide in minutes,” then human review isn’t your control mechanism. Your control mechanism is monitoring and fallback.

If the answer is “we can tolerate a few hours,” then you need enough human capacity to review decisions on that timeline. That’s a real constraint. Build for it.

If the answer is “we can tolerate a few days,” then you can probably build a human-review process that works.

But don’t answer with what you wish were true. Don’t say “humans review everything” if your volume makes that impossible. That’s how you end up in escrow.

The organizations that manage AI decision-making well know their constraints and design to them. They don’t promise speed, judgment, and oversight unless they can deliver all three. Most organizations promise all three and end up delivering none of them well.

Every financial services firm has a Model Risk Management framework. Most of them haven’t actually used it to catch anything.

This isn’t because the frameworks are poorly designed. It’s because they’re designed for a problem that doesn’t quite exist, deployed against risks that are invisible to the framework.

I’ll explain, because this is one of those areas where understanding the gap tells you a lot about how to think about AI risk more broadly.

What model risk frameworks are built to catch

The standard Model Risk Management framework came from Basel guidance and financial regulator requirements. It evolved around a specific category of problems: When you build a quantitative model that feeds decision-making, and the model is wrong in a systematic way, you make bad decisions at scale.

So the framework focuses on: Is the model valid? Is it accurate? Does it perform as expected? When something goes wrong, what’s the model’s role in the failure?

This is useful for catching models that are mathematically broken, statistically invalid, or performing worse than expected. It catches overfitting, concept drift, and data quality issues.

It does not catch most of the risks that actually matter.

What’s invisible to standard frameworks

Here’s a pattern I see repeatedly: An organization implements a model, validates it thoroughly, puts it into production, and monitors it diligently. The model performs exactly as expected. Accuracy is stable, validation metrics are healthy, everything looks fine.

Then something goes wrong. The model wasn’t wrong — the decision-making context around the model was wrong.

A lending model that accurately predicts default risk produces a disparate impact on protected classes. The model is statistically sound. It’s predicting correctly. But the business decision to use it for loan decisions in an underserved market was wrong because the training data came from a time when lending practices were different in that market.

A trading model that accurately captures market correlations suddenly produces an unexpected loss because correlation structure changed, and nobody had decided what to do if that happened. The model did what it was trained to do. The decision authority around how to handle model uncertainty had never been established.

A workforce optimization model that accurately predicts which positions will have retention problems is deployed to recommend layoffs, but nobody had decided whether a model designed to predict problems should drive who gets laid off. The model worked. The application was wrong.

In each case, the model risk framework shows green. The validation is sound. The monitoring is clean. The problem isn’t in the model — it’s in the decision system around the model.

Why frameworks miss this

Standard model risk frameworks are built to answer the question: “Is this model trustworthy as a mathematical instrument?”

The question that actually matters for most enterprise AI risk is different: “Is the decision we’re making with this model appropriate?”

These are correlated but not identical. You can have a trustworthy model used for an untrustworthy decision. And you can have a somewhat noisy model used thoughtfully where the human judgment around it is sophisticated enough to manage the risk.

Most frameworks check the first box and assume the second one is handled elsewhere. Spoiler: It’s usually not handled at all.

Organizations that handle model risk seriously usually do a second set of work that sits beside the traditional framework. It asks different questions:

What is this model actually being used for? Not in theory, but in practice. How is the decision actually being made? Who’s being affected? What would a bad outcome look like?

Why are we using a model for this decision at all? Sometimes the answer is “because it’s more accurate than human judgment.” Sometimes it’s “because it’s faster.” Sometimes it’s “because it removes the appearance of bias even though the input data might be biased.” These matter, because they change what your risk actually is.

What’s the decision being made if the model is wrong? If the model’s confidence is low, do you have a process for human override? If the model fails in a specific way, what’s the next decision? If you can’t describe what you do when the model is wrong, you don’t understand your own risk.

What would falsify our assumptions? You’re assuming the training data is representative of the population you’re applying the model to. What would prove that wrong? You’re assuming the decision context is stable. What would change it? You’re assuming the model’s accuracy is stable. How would you know if it wasn’t? If you can’t describe the conditions that would break your model’s usefulness, you’re building on faith, not on reasoning.

Is there a human in the loop who could reasonably override the model? Not should there be — could there be? If the model recommends something and a human thinks it’s wrong, can they say no? If not, you’ve built a system where the model is actually making the decision, and you should be honest about that. If yes, you need a process for when humans should override and what you learn from overrides.

The uncomfortable conversation

Organizations that separate themselves from the crowd usually do something harder than updating their framework. They have what I’d call the “uncomfortability conversation.”

Someone sits down and lists all the ways the model could produce a bad outcome that isn’t captured in the model risk framework:

• Accurate prediction of biased historical patterns (model correct, decision wrong)
• High accuracy but low coverage (model works for some populations, not others)
• Decision context change (model was trained in one business context, deployed in another)
• Correlation breakdown (model captures historical correlations that don’t predict future)
• Population drift (model trained on one population, applied to another with different characteristics)

For each, they ask: Do we have a process to catch this? Is this monitored? Would someone know if this happened?

Usually the answer is no. Usually the model risk framework is looking at the one thing that won’t happen (the model is mathematically broken) and not looking at the things that actually could happen.

Then they do hard work: They either redesign the model and decision system to make the actual risks visible and manageable, or they decide the risk is too high and don’t deploy the model.

Most organizations don’t do this work. They validate the model thoroughly, deploy it, monitor it religiously, and assume risk is managed because the monitoring is good.

What this means for building better systems

If your organization is evaluating whether to deploy a model, the conversation should include this piece: “Here are the ways this model could produce bad outcomes that our framework might not catch. Here’s how we’re going to manage each one.”

If the answer is “our framework handles it,” push: How exactly? Can you describe the monitoring alert that catches it? Can you describe the decision that gets made when it happens?

If the answer is “we’ll handle it with human judgment,” push again: What training do those humans have? How do you standardize that judgment? How do you know it’s working?

If you can’t get a satisfying answer to either question, the model isn’t ready to deploy. Not because it’s mathematically broken, but because you don’t understand the risks it creates in your actual business context.

This is the conversation that separates organizations that treat model risk as an audit requirement from organizations that treat it as a serious business problem.

The ones that do this work rarely end up with model risk surprises. The ones that don’t usually end up in regulatory meetings explaining why their perfectly validated models produced regrettable outcomes.

Your compliance team is probably preparing for the wrong thing.

This isn’t their fault. Regulation of AI doesn’t follow the template of financial regulation or healthcare regulation or environmental regulation. Which means the preparation that worked for Dodd-Frank or HIPAA or the Clean Air Act won’t work here. And most compliance teams are building their AI readiness using the only template they have.

The result: Organizations are investing in comprehensive governance frameworks, audit procedures, and documentation systems that regulators probably won’t ask for in the ways they’re expecting.

What regulators actually care about

I spoke with someone in the policy space at a financial regulator recently. I asked: “What’s the first thing you’re going to look at when you start auditing banks’ AI systems?”

The answer wasn’t “show me your governance framework” or “give me your audit logs” or “where’s your risk register?”

It was: “Show me the decisions this AI system is making. Now show me three examples of bad outcomes. Why did the system make those decisions, and why didn’t anyone catch them?”

That’s not a compliance question. That’s a comprehension question. Regulators want to understand what’s actually happening inside your systems, and they know most organizations can’t explain it.

This matters because it reorders the priority stack. The first thing regulators will assess isn’t whether you have governance — it’s whether you understand your system well enough to explain its behavior. Governance documents are evidence of understanding, not a substitute for it.

The explainability gap

Here’s where compliance teams usually get stuck: They build governance frameworks around the assumption that they need to demonstrate control and oversight. So they create approval workflows, audit logs, monitoring procedures. All legitimate components of a control system.

But they often skip the foundational work: Can anyone at the organization actually trace why the system made a specific decision in a specific case?

I worked with a company that had a beautiful AI governance structure. Quarterly reviews, decision approval sign-offs, dedicated model risk managers. When a regulator’s hypothetical became real — a model recommendation that produced a clearly unfavorable outcome for a protected class — they tried to trace it. Why did the system score that customer that way?

They couldn’t. They had governance, but no explainability. They had documentation, but no understanding. The model owner had moved to a different team. The training data pipeline had been updated. The feature engineering logic was lost in Jupyter notebooks nobody maintained. They had all the structural elements of control and almost none of the functional elements.

A governance structure built on this foundation is actually a liability. It demonstrates that you have processes, which proves you had accountability, which makes the poor outcome look worse — not better.

What preparation actually looks like

Organizations that are genuinely prepared for regulation are usually doing a different set of work right now. Some of it looks like governance, but most of it looks like understanding:

1) Explainability work — Can you pull a specific decision and explain it in a way that makes sense to someone who isn’t your data scientist? Not “the model scored it,” but “the model weighted these factors, which led to this result, and here’s why those weightings made sense for this use case.”

This requires actually maintaining that explainability. Some teams do this through SHAP values or other interpretability frameworks. Others just maintain clear decision trees or decision rules. The method matters less than the commitment: You’re building to be explainable, not building and hoping explanation comes later.

2) Outcome monitoring — Not just “is the model accurate,” but “is the model making decisions that we expect?” Are certain segments systematically getting different treatment? Are the outcomes diverging from what you designed for? Are there decisions that look right mathematically but feel wrong operationally?

Most monitoring systems focus on statistical drift (has the model’s accuracy changed). Regulators care about behavioral drift (are the decisions changing in ways we didn’t expect). These aren’t the same thing.

3) Decision documentation — Not in the governance sense (we approved this decision), but in the explainability sense. Why did we choose this architecture? What assumptions did we make? What would falsify those assumptions? If a bad outcome occurs, what would tell us whether the system failed or our assumptions about the system were wrong?

This sounds like it might take forever. Organizations that have done it well usually structure it around decision memoranda. One page per major choice: the choice, the alternatives considered, why you chose what you chose, what would tell you if you were wrong.

4) Traceability of inputs — This is where most organizations are weakest. Where did the training data come from? Has it been modified? Who had access to it? If an outcome looks discriminatory, can you reconstruct what data the model learned from?

This is partly a governance question, but it’s mostly a technical question. It requires data versioning, provenance tracking, and lineage documentation. It requires that somewhere in your system, you can reconstruct the training dataset from six months ago.

Why this matters more than compliance theater

The reason I’m focusing on this isn’t because regulators need explainability — though they do. It’s because having it actually makes you better at building AI systems.

Organizations that maintain explainability throughout development make better decisions about model architecture. They can’t just accept “the model says so.” They have to be able to defend the decision. This raises the bar.

Organizations that truly monitor behavioral drift catch problems earlier. They’re not looking at accuracy degradation in quarterly reports. They’re asking constantly: Are we still making the decisions we intended to make?

Organizations that maintain decision documentation learn faster. When something goes wrong, you can distinguish between “the system worked as designed but the design was wrong” and “the system didn’t work as designed.” Most post-mortems can’t make this distinction. That’s expensive.

Organizations that track data provenance get better at understanding their own bias. You can’t fix something you can’t trace. Once you know where your training data came from, which decisions it was used for, and which populations it was meant to represent, you can actually audit for bias.

Where to start

If your compliance team is currently building a governance framework without this foundational work, it’s not too late to reorder. Start here:

Can you pick a model that’s in production right now and explain in plain language why it made a specific decision in a specific case? If you can’t do that, you don’t have compliance readiness — you have documentation.

Once you can, build governance systems around the assumption that you can always be asked to explain. This means architecture that’s inherently explainable, data that’s inherently traceable, and decisions that are inherently defensible.

The governance will follow from there. And when regulators do arrive, they’ll be asking about the same things you’ve already learned to explain.

Most organizations approach AI accountability the way they approach aviation safety — which is exactly right in theory, but wrong in how they actually implement it.

They build systems designed to identify who made the error. Airlines built systems designed to prevent errors from ever happening. That distinction matters more than it seems.

I watched a financial services company spend six months designing an AI governance framework. The centerpiece was a “decision audit log” — comprehensive tracking of which stakeholder approved which model decision. They wanted accountability. What they actually built was a blaming machine.

Here’s what happened: When a model recommendation created problems (a credit decision that looked discriminatory, a trading algorithm with unexpected volatility), the leadership team’s first instinct was to trace the decision backwards and identify the failure point. Who signed off? Who didn’t ask the right questions? The accountability structure worked exactly as designed — it identified responsibility.

It destroyed the incentive to ever flag problems early.

Teams learned quickly that documenting concerns before decisions were made created a permanent record of doubt. If something went wrong later, you’d be asked in a meeting why you raised a concern and didn’t escalate it further. If you never documented it, you had plausible deniability. The audit log became a liability trap rather than a learning tool.

The framework inversion

The accountability that actually matters isn’t about tracing blame backwards. It’s about creating conditions where the system improves before things break.

Airlines don’t have accident review boards because they want to blame pilots. They have them because every incident — even minor ones — contains information about the system’s weaknesses. The system that made the incident possible is what they’re investigating, not the human who operated within it. The structure is designed to make reporting easier, not harder.

Organizations that handle AI accountability well operate the same way. They ask different questions:

What made this decision possible without someone catching it earlier? Not: Who should have caught it?

What information wasn’t available when the decision was made? Not: Who made a bad judgment call?

What would have to be different for someone to feel comfortable raising this issue before it happened? Not: Why didn’t anyone raise this issue?

These aren’t rhetorical differences. They’re structural. A blame-oriented accountability system makes the person who raises the concern a participant in the problem investigation. A system-oriented accountability structure makes them the resource that helped you avoid a bigger failure.

What actual accountability looks like

The companies I’ve seen get this right usually have three overlapping mechanisms, and they’re explicit about which mechanism answers which question:

1) Decision authority tracking — Who had the right to decide, and did they exercise it? This exists for clear decision boundaries, not for blame assignment. It answers one question: Is someone actually responsible for this decision, or did it happen by default? If nobody had authority to make it, the structure itself failed.

2) Escalation documentation — When someone raises a concern, does the system require response? Not agreement — response. An escalation that was raised and dismissed is different from an escalation that was never logged. The system needs to show that someone with decision authority received the information and consciously decided to proceed. This isn’t about blame; it’s about forcing conscious choice instead of default drift.

3) Post-decision learning — When something doesn’t work as expected, the organization conducts a structured review focused on system improvement, not individual performance. What assumptions turned out to be wrong? What monitoring signals did we miss? What would a smarter structure catch earlier next time? These findings feed back into the governance structure itself.

The third mechanism is where most organizations fail. They conduct the review, document the findings, and then… the governance structure doesn’t actually change. It gets updated in a PowerPoint. The people involved return to the same system that allowed the problem to happen.

The real cost of blame-oriented accountability

Here’s what actually happens in organizations built on blame-tracing: People become very good at decision concealment. Not dishonesty, exactly — more like distributed decision-making that creates plausible deniability. “The model just flagged that,” or “We always do it that way,” or “The system recommended it.” The decision becomes diffused across enough people and processes that nobody can be held accountable for it.

Which is actually worse than having accountability, because now nobody owns the decision and nobody has the power to change it.

I watched a company’s legal team flag a discrimination risk in a lending model. The risk was real, the concern was legitimate. But the escalation process required the model owner to respond, and if they responded “I understand the concern but I’m choosing to proceed,” they’d created a paper trail. So instead, the model owner requested “additional validation,” which meant another review, which meant more time, which meant the issue got absorbed into standard model refresh cycles. The discrimination risk got documented, then lost in process, then forgotten.

The accountability structure made the organization less safe, not more.

Building the structure that works

If you’re designing or redesigning accountability mechanisms for AI decisions, start here:

Clarity of authority — Someone actually owns each decision. Not a committee, not a consensus, not a matrix of shared responsibility. One person has the authority and the information to make it. That person’s role exists specifically to make these decisions, not to rubber-stamp them.

Escalation as information, not as stopping — Raising a concern doesn’t require agreement. It requires documented response. Someone with authority has to say “I understand this concern and I’m proceeding anyway” or “I understand this concern and we’re not proceeding.” Not “I’ll think about it,” not “Let’s form a subcommittee.” Clear choice.

Feedback loops that matter — When things don’t work as expected, the organization captures what was wrong and changes the governance structure itself. Not the policy document. The actual structure — who decides what, who escalates what, who reviews what. If the same risk appears twice, the governance structure failed.

The organizations that have gotten this right are usually smaller, or they’ve carved out a specific domain where accountability is treated as a system problem rather than a personnel problem. They’ve also accepted a harder truth: Accountability structures that actually work require letting people make decisions and then living with the consequences. If every decision requires protection from blame, you’ll never have clean decision-making.

The choice is whether you want accountability that looks good in audit meetings, or accountability that prevents problems from happening.

A month ago, we started somewhere obvious and necessary: the fundamentals.

We looked at how data actually flows through organizations and why most AI governance fails at the data layer, not the model layer. We asked what metrics boards should actually track instead of the ones they do. We examined resilience design and what it means to build systems that fail gracefully. We looked at the legal and IP liability that most enterprises don’t fully understand they’re carrying. We talked to foundation model vendors about why they’re unprepared for the questions they’re getting asked.

And then we shifted. The last few days weren’t about individual problems. They were about organizational structure: How responsibility actually distributes across decision-making, how it looks in production, how you avoid sunk cost traps, how you measure what matters.

There’s a pattern that connects all of this.

The pattern isn’t about AI capability gaps. Organizations are building capable AI systems. The models work. The integration is clean. The technical problems are getting solved.

The pattern is about governance. Not governance as a checkbox or a process. Governance as an organizational capability to see what you’ve built, make deliberate decisions about how to use it, and respond when those decisions need to change.

Here’s what thirty days of observation across early adopters has revealed:

Most organizations have built sophisticated systems inside of primitive governance structures.

This manifests in a specific way. You see a company with leading-edge machine learning, models that beat internal benchmarks, beautiful technical infrastructure. And then you see a decision-making process that would be unrecognizable in any other critical business domain.

When a high-stakes system breaks in traditional enterprise—when a payment processing system fails, when a trading system misbehaves, when supply chain optimization goes wrong—there’s a protocol. There are alarm systems. There are people whose job is monitoring. There’s accountability. There’s a decision structure for responding.

When an AI system breaks, many organizations are improvising.

Not because they lack commitment to responsibility. But because they built the technical capability much faster than they built the organizational capability to manage it.

This gap is the central problem in enterprise AI today. Not the models. The decision-making.

Let me try to map what this looks like, because understanding the pattern helps explain why some organizations are handling AI maturely and most aren’t.

Technical readiness is not governance readiness.

You can have sophisticated models and a data scientist team that understands how to train them, evaluate them, and deploy them. You can have good technical monitoring—dashboards that show accuracy, latency, performance. But none of that is governance.

Governance is answering different questions: Who decided this model was appropriate for this use case? What assumptions does that decision rest on? If those assumptions change, who notices and who decides what happens? If the model fails, who’s accountable? To whom? For what?

Most organizations have answers to the technical questions. They have answers to maybe 30% of the governance questions.

Responsibility doesn’t distribute automatically.

Organizations assume that because they’ve deployed a system and it’s running, someone is responsible for it. In practice, responsibility is often ambiguous. The business leader thinks the technical team owns it. The technical team thinks the business leader owns it. The governance team is in an advisory role with no binding authority.

When something needs to happen—investigation, remediation, decision change—no one is quite sure who initiates it. Weeks pass while the organization figures out the authority structure.

The organizations that don’t have this problem have done something simple: they’ve made responsibility explicit. Someone owns the decision. Someone owns the data. Someone has authority to pause it. Those are roles that already exist in the organization, but the accountability is explicit.

Observability is different from governance, but most organizations confuse them.

You can monitor a system. Dashboard. Alerts. Good technical visibility. That’s observability.

But observability without a decision-making structure just means you have good information about something you might not be able to react to.

Governance is what you do with the information the observability gives you. Who sees it? Who decides whether it matters? Who decides what to do about it? That’s the governance piece.

Organizations often invest in observability and under-invest in the decision structure that makes observability actionable.

Sunk costs are the decision-making killer.

This might be the most consistent observation. Organizations make AI investment decisions based on past spending rather than future expected value. They don’t have a framework for reassessing when assumptions change. So programs that should be redirected or stopped keep running because of what’s already been spent.

This isn’t irrational. It’s structural. There’s no protocol for “should we keep going?” There’s no scheduled moment to ask “if we started today, would we do this?” So the default is to keep going.

You can’t measure what you don’t define.

The most sophisticated AI organizations have made a deliberate choice about what they’re measuring for adoption. They’re not measuring usage. They’re measuring decision change. Did the presence of the AI recommendation change what the human decided?

This requires work—baseline data, control groups, careful attribution. But it’s the only number that predicts whether adoption is actually happening.

Most organizations measure usage and call it adoption. Two years later, they discover high usage with zero impact on decisions or business outcomes.

The real responsibility is operational discipline.

After all the governance frameworks, all the ethics reviews, all the fairness audits—the part that actually makes a system responsible is continuous operational visibility. Someone watching. Someone noticing when things change. Someone able to respond.

This requires people. It requires infrastructure. It requires ongoing investment. It’s unglamorous. It’s not conference-talk material. But it’s the actual work of responsibility.

The full picture looks like this:

Early-stage AI governance is trying to go from “we have capability” to “we have managed capability.” That transition requires changing how organizations make decisions, distribute responsibility, and invest resources.

The organizations that are handling this well aren’t the ones with the most advanced models. They’re the ones that built organizational structures that can absorb technical change. Who can see what they’ve built. Who have clear decision authority. Who can respond when things shift.

This isn’t automatic. It requires deliberate design.

And it requires something else too: cultural permission to ask hard questions. To say “should we still be doing this?” To admit that assumptions were wrong. To build tolerance for stopping programs that aren’t working out. To measure what matters rather than what’s easy to measure.

That’s harder than building the models.

But that’s where the actual work is.

The gap between AI capability and AI governance is real. You can build capable systems much faster than you can build the organizational discipline to manage them responsibly.

The month we’ve just completed was mapping that gap. The month ahead is about closing it.

The systems are ready. The question is whether the organizations are.

There’s a number that organizations watch obsessively.

It’s called adoption. Or engagement. Or sometimes just “users.” The dashboard shows it with a big upward trend line. The reports celebrate it. “Our AI platform adoption is at 35%, up from 12% last quarter!”

And then the business outcomes don’t improve.

The revenue stays flat. The process doesn’t get faster. The decision quality doesn’t shift. Employees are using the tool, but nothing actually changes.

This is what happens when you measure the wrong thing.

I was working with an organization that had deployed AI to help their sales team qualify leads. The system was built well. It had solid accuracy. The integration was clean. And they measured adoption by how many sales reps logged into the system each week.

Adoption climbed quickly. They celebrated. “67% of our sales team is using the system.”

But the lead closure rate didn’t move.

When they dug deeper, they found something interesting: The sales team was logging in, looking at the AI’s assessment of a lead, and then doing what they would have done anyway. The AI wasn’t changing any decisions. It was just there.

Usage was high. Impact was zero.

Here’s what I’ve learned about measuring AI adoption: The metric that matters isn’t whether people use the tool. It’s whether people make different decisions because of the tool.

That’s harder to measure. It requires connecting the AI’s recommendation to what actually happens. It requires understanding what the person would have done without the AI. It requires separating the signal from the noise.

But it’s the only metric that tells you whether adoption is real.

Let me be concrete about what this looks like. Imagine you’ve deployed an AI system to help with hiring decisions. A recruiter uses it to help screen candidates. The metric everyone watches is “percentage of candidates screened with AI assistance.”

That number says nothing about whether it matters.

What you actually want to know is: “Does the presence of the AI recommendation change which candidates the recruiter advances?”

That’s different. That requires you to compare the recruiter’s decisions with the AI present versus their historical decisions without it. Or to compare their decisions against a control group. Or to look at whether the candidates they advance (using the AI) have different outcomes than the candidates they would have advanced without it.

The recruiter might use the AI for every screening. But if they advance the same candidates they would have advanced anyway, adoption is an illusion.

This distinction matters because organizations often plateau at “everyone is using the tool” before they realize the tool isn’t changing anything.

I watched another organization deploy AI for customer service escalation routing. They wanted the system to recommend which customer issues should be escalated to senior specialists. Good idea. Smart use of AI.

Usage climbed. Adoption metrics looked great. Then someone asked: “Are we actually escalating the right issues?”

The answer was complicated. The system was making recommendations. The customer service reps were using those recommendations. But they were overriding them roughly 40% of the time. And they were also escalating issues the system didn’t recommend.

So the AI wasn’t actually driving the escalation decisions. The human judgment was. The AI was just providing additional information that the humans sometimes considered.

That’s not necessarily bad. But it’s very different from adoption that actually matters. If the system changes decisions, then 40% override rate is valuable. If the system isn’t changing decisions, then it’s just overhead.

Here’s the framework that separates real adoption from usage theater:

Ask: What decision does this AI affect?

The decision could be “should we hire this candidate?” Or “should we approve this loan?” Or “should we serve this ad?” Or “which support tickets should be escalated?” A specific decision.

Then ask: If we compare decisions made with the AI present versus without it, are they different?

This requires a baseline. What would have happened without the AI? Sometimes you can measure this with a holdout group. Sometimes you can look at historical data. Sometimes you can ask the human decision-maker what they would have done.

If the decisions are different, and the change is in the direction the AI was designed to improve, then adoption is real.

If the decisions are the same, then adoption is illusion.

The hard part is that real adoption often plateaus faster than usage adoption. You might get 70% of your organization using the tool, but only 30% of them actually changing decisions because of it. That’s frustrating to report, because it looks like only 30% adoption.

But it’s also the truth. And the truth is more useful than the appearance.

The organizations doing this well have shifted how they measure. They don’t report “percentage of employees using the tool.” They report “percentage of decisions that change because of the AI.” Or more specifically: “In situations where the AI made a different recommendation than the human would have made independently, how often did the AI recommendation win?”

That number is often surprising. It’s often lower than you’d expect. But it’s also the number that predicts business impact.

Here’s why this matters: If you’re measuring usage, you celebrate when the plateau is reached. If you’re measuring decision change, you keep investigating why the plateau exists.

Maybe the AI isn’t good enough yet. Maybe the recommendation isn’t presented in a way humans trust. Maybe the decision context is more nuanced than the AI understands. Maybe the organizational incentives don’t align with the AI’s recommendations.

Whatever the blocker is, you only find it if you’re measuring the thing that actually matters: Did the AI change the decision?

I’ve watched organizations spend millions deploying AI systems and then discover, two years in, that the system isn’t actually changing any decisions. But they have great adoption metrics. Everyone is using the tool.

That’s a very expensive way to learn that adoption and impact aren’t the same thing.

The metric to track is decision change. Not usage. Not logins. Not percentage of employees trained.

Decision change.

It’s harder to measure. It requires thinking carefully about what you’re trying to affect. It requires baseline data or holdout groups or counterfactual analysis.

But it’s the only metric that actually predicts whether the AI investment will pay off.

Everything else is just activity.

There’s a moment in most AI programs where someone needs to ask: Should we keep going?

Not whether it’s theoretically possible. Not whether the problem is interesting. Not whether the vendor says it’ll work.

Whether we should keep spending money on this.

And I’ve noticed something consistent: Organizations ask this question very late. After they’ve already spent more than they probably should. After they’ve built organizational muscle around the program. After it’s become politically difficult to stop.

By then, the conversation has already been decided by sunk costs.

“We’ve already invested $5 million. We can’t just walk away.”

“We’ve been working on this for eighteen months. We need to see it through.”

“The executive sponsor is committed. We can’t tell them it’s not working.”

These are all versions of the same underlying logic. We paid for it already. We can’t get that money back. So we might as well keep going.

This logic is backward.

Here’s what I mean. You’re a financial services company. You’ve invested in building an AI system to improve loan approval decisions. You spent months on requirements. You spent more months on data preparation. You spent more on model development. You’ve been building it for a year. The cost is significant. The team is big.

And then something becomes clear: The business actually cares more about speed than accuracy. The approval process needs to be faster, but your model is slow. Or the regulatory environment shifted and your model’s lack of interpretability is now a problem. Or the business priorities changed and what made sense eighteen months ago isn’t what matters now.

The question appears: Should we keep building this?

The wrong answer is “We’ve already spent $5 million. We’re too far to stop.”

The right answer is “What would we do if we could start over, knowing what we know now?”

If the answer to that question is “We wouldn’t build this,” then you probably shouldn’t keep building it.

I watched this play out with an organization that had committed to an enterprise-wide machine learning platform. The rationale was sound. They had many models. They wanted to manage them all through one system. The vendor claimed it could integrate with their infrastructure. The internal team was enthusiastic. The investment was made.

Eighteen months in, the integration was harder than expected. The vendor’s platform didn’t fit their stack the way they hoped. The maintenance overhead was larger than planned. The payoff wasn’t as clear. And more importantly, the business had moved on. Different problems seemed more urgent.

The team kept building anyway. They’d already spent enough that leadership hesitated to pull the plug.

Another year later, the project was finally cancelled. The system was never deployed. All that investment was gone.

That’s not a vendor failure. That’s not a technical failure. That’s a project management failure. More specifically, it’s a failure to ask the right question at the right time.

Here’s what should have happened at month twelve, when signs emerged that assumptions weren’t holding up:

The project manager should have asked: “If we could start over with what we know now, would we still build this?”

If the answer is no, the follow-up is: “What would we do instead?”

The critical question is: “How much more should we spend before we decide?”

This is where sunk costs become dangerous. Because the amount you’ve already spent doesn’t determine how much more you should spend. The only thing that determines that is future expected value.

The right framework is simple:

Every AI program should have a set of milestones with explicit decision points. Not just technical milestones. Business milestones. “At month six, we should know whether the business case holds.” “At month twelve, we should know whether the technical architecture works.” “At month eighteen, we should have evidence that the model actually improves the process.”

And at each milestone, the question shouldn’t be “Did we hit our technical targets?” It should be “Has anything changed about what makes this valuable?” And “If we started this project today with current knowledge, would we still do it?”

If the answer is no, or if the answer is “Maybe, but only if we change direction significantly,” then you have a decision to make.

Few organizations have this framework. So they drift. The sunk costs mount. The decision gets harder to make. The project becomes its own justification.

I’ve seen organizations rationalize spending millions more on projects they wouldn’t start if they could do it over. I’ve seen teams defend dying programs because the political cost of stopping was too high. I’ve seen executives avoid the conversation altogether because asking “Should we stop?” requires admitting something went wrong.

But here’s the thing: Something usually does go wrong. Not because the team is incompetent. But because the future is hard to predict. What seemed like a good use of capital eighteen months ago might not be today.

That’s not a failure of planning. That’s a fact of operating in uncertainty.

The failure is not adapting when that uncertainty resolves.

The organizations that handle this well have a pattern: They treat AI programs like venture capital investments. High risk. Frequent decision points. Clear go/no-go criteria. If something changes, you reassess. You don’t keep funding because of what you’ve already spent. You fund based on what you expect to get.

And critically, they build a culture where saying “This isn’t working anymore” is possible. Not as a personal failure. But as a normal part of learning what works and what doesn’t.

Because you will build things that don’t work out. You will make bets that don’t pay off. You will discover that assumptions were wrong. That’s how you learn what actually works.

The sunk cost fallacy isn’t just about money. It’s about how you treat uncertainty. Do you let past decisions trap you? Or do you use new information to make better decisions now?

The $5 million you already spent can’t be recovered. But the next $2 million might not need to be spent, if you ask the right question.

The question isn’t “Can we afford to stop?” The question is “Can we afford to keep going?”

Until you ask that clearly, sunk costs will keep winning.