Most boards have risk appetite statements. Most are useless.

They sit in governance documents, drafted by compliance and risk teams, worded so generically that any outcome could be called compliant. “We maintain a balanced approach to risk while pursuing innovation.” “We accept calculated risks that support strategic objectives.” When the problem arrives—a model fails, a bias issue goes public, a regulatory question gets serious—nobody points to that statement and says, “Ah, yes, this is exactly what we authorized.”

That’s not a risk appetite statement. That’s security theater.

A useful risk appetite statement for AI does something specific: it describes, in operational language, what kinds of failures the organization is willing to experience and what it isn’t. Not failure in general. Specific failure modes. Specific thresholds. Specific trade-offs.

The usual failure is that risk appetite statements are written as aspirations rather than constraints. They describe what the organization hopes to achieve, not what it’s willing to tolerate. A real risk appetite statement for AI should feel, on first read, uncomfortably concrete.

What You’re Actually Deciding

When you set a risk appetite for AI, you’re answering a smaller, sharper question than most boards realize: Under what conditions will we accept outcomes that we wouldn’t accept from traditional systems?

Because AI will produce outcomes that rule-based systems wouldn’t. It will make decisions that are accurate on average but wrong for specific people. It will fail in ways that are hard to predict. It will sometimes work better than legacy alternatives, sometimes worse. You have to decide, in advance, whether you’re okay with that trade-off.

That’s not a philosophical question. It’s an operational one.

Consider hiring. A traditional hiring system is a rubric: minimum GPA, degree from certain schools, years of experience. It’s crude. It misses talented people. But it’s consistent, and when it fails, you know why. An AI hiring system can be more accurate across the population, but for individual candidates, the reasons for decisions are opaque. And it will sometimes make mistakes that a human would catch.

What’s your appetite for that? Not in principle. In practice.

Do you accept a model that’s 5% more effective on average but makes occasional decisions you can’t explain to a candidate? Ten percent more effective? Twenty? And at what accuracy threshold do you switch back to rules? These aren’t theoretical questions. They’re the ones that will matter when the litigation starts.

A risk appetite statement that’s worth something says: “We accept hiring models that outperform our traditional rubric by at least 8% on overall placement success. We do not accept opaque rejections for protected class candidates. We conduct quarterly bias audits and halt model use if disparate impact on any protected class exceeds 3 percentage points.”

That’s specific. It’s defensible. It’s something your board is actually committing to.

The Categories That Matter

Most boards conflate risk appetite across different failure modes. They shouldn’t. An acceptable failure for a recommendation system (wrong suggestion, user ignores it) is not acceptable for a compliance system (false positive, audit cost explodes). An acceptable accuracy threshold for a back-office process (efficiency gains outweigh occasional rework) might be criminal negligence in a loan decisions system.

Your risk appetite statement should separate failure modes by consequence:

Direct customer/citizen impact. Model predictions affect someone directly. Loan decisions, hiring, benefits eligibility, medical triage. The failure mode is accuracy and bias. The question is: what accuracy level is required before we use this, and what disparities across populations do we tolerate? You need a numerical answer.

Operational efficiency. The model improves internal processes—contract review, log analysis, pattern detection. The failure mode is missed problems or rework. The question is: what percentage of errors will we tolerate before we switch back to human review? Not “we’ll monitor it.” What’s the actual threshold?

Strategic/regulatory exposure. The model touches compliance, audit, or regulatory reporting. The failure mode is misclassification with enforcement consequences. The question is: will we accept any error rate, or do we require 99.9% accuracy with human review of all flagged items? Do we get regulatory pre-approval before deploy?

Speed/convenience. The model is faster than the alternative but less certain. Chatbots instead of agents, automated triage instead of manual. The failure mode is a bad experience or escalation. The question is: what escalation rate is acceptable? How much productivity gain justifies how much friction?

Each of these has a different risk appetite. The mistake boards make is writing one statement and applying it to all four.

What Changes When You Do This

Setting a real risk appetite statement for AI changes three things:

First, it forces alignment before deployment, not after. Your board has said what trade-off they’re accepting. When the question comes later—“Should we keep using this despite the bias issue?”—you have a reference point.

Second, it creates clarity on what monitoring you actually need. If your appetite statement says “5% error rate is acceptable,” you build monitoring to catch when you hit 4.5%. You don’t build monitoring for perfection.

Third, it makes decisions delegatable. Once your board has set the appetite, your operations and compliance teams can evaluate models against it without coming back for every decision. You’ve moved from review-based governance to criteria-based governance.

The hard part isn’t writing the statement. The hard part is defending the numbers. Why 5% and not 3%? Why 99%? When someone on your board asks that question and you don’t have an answer, you’ve found where the real decision-making needs to happen.

That conversation should happen before you deploy the model. Not after something breaks.

The Conversation Worth Having

The best risk appetite statements come out of actual tension, not consensus. Someone in the room says, “I’m not comfortable deploying a model with an error rate above 2%.” Someone else says, “Then we can’t compete; the best model we can build is 3%.” That’s a real conversation. The number you land on—2.5%, maybe, with more aggressive monitoring—is worth something because it’s actually contested.

A risk appetite statement that arrived through consensus and compromise is the useless kind. The kind that sits in documents and means nothing when you need it.

Your board should sign off on risk appetite for AI. But make them sign off on something real—specific failure modes, specific thresholds, specific monitoring regimes. Not aspirations. Not hedging language that fits any outcome.

If they won’t commit to specific numbers, that’s information too. It means the organization isn’t actually ready to deploy the model yet. And that’s a decision worth making before you’ve already deployed it.

Your company is probably spending more on AI talent than you think.

Not in salary — though that’s part of it. In what you’re giving up while they’re maintaining the system you built.

This is the cost that doesn’t appear in budgets, but it explains why so many organizations that invested heavily in AI still feel like they’re not getting return on investment.

The context-switching tax

Here’s what usually happens: You hire a strong AI/ML engineer. They join at a critical moment. You have models to build, systems to set up. They dive in.

Six months in, the initial projects are done. The models are built. Things are working.

Now they’re managing the models. They’re monitoring. They’re updating. They’re responding when something breaks.

They’re also doing all the governance work that the previous ten sections have described:

• Maintaining data provenance
• Setting up behavioral monitoring
• Documenting decisions
• Mapping vendor dependencies
• Updating governance when something breaks

This isn’t one job. This is about five jobs. And you probably have one person trying to do it.

The best AI talent doesn’t want to be the everything person. They want to be building.

But they’re not building. They’re in maintenance mode. They’re reading dashboards. They’re writing retrospectives after incidents. They’re answering “why did the model do that” questions for business teams.

If they had capacity to build, they’d be building. They don’t have capacity because the infrastructure around the model requires constant care.

Why traditional staffing doesn’t work

Here’s where most organizations get the math wrong: They think “we need data scientists to build models, and we need one person to maintain them.”

What they don’t realize is that building an AI system that actually works — that is explainable, monitored, governed, accountable — requires way more than one person maintaining it.

You need:

• Someone understanding data provenance and maintaining lineage
• Someone building and maintaining behavioral monitoring
• Someone tracking governance (decisions, escalations, updates)
• Someone managing vendor dependencies and integration
• Someone responding when something goes wrong

These could be one person in a small organization. But they’re usually at least three people in any organization with more than a few models.

And you probably have them all under “one ML engineer” in your budget.

The reason AI projects feel understaffed is usually because they are.

But the reason they’re understaffed isn’t because AI is hard. It’s because organizations don’t realize what “AI working properly” requires in terms of infrastructure.

The burnout pattern

This is where talent costs become really high.

Your AI engineer is doing five jobs. They’re good at maybe three of them. The other two are just work that needs to get done. They’re smart, so they can do it, but it’s not what they want to do.

They’re smart enough to realize they could be doing something else. Something more interesting. Something that builds on their strengths.

So they look. And they find another opportunity. A company that will let them specialize. That will build proper infrastructure so they don’t have to be everything.

You lose the person. You hire someone new. You spend three months ramping them up. They’re immediately in context-switching mode because nobody invested in infrastructure while you had the first person.

The real cost of your AI talent isn’t their salary. It’s the turnover rate. It’s rebuilding the context every time someone leaves.

What you’re actually paying for

When you hire AI talent, you’re not just paying for the code they write. You’re paying for:

Understanding

This is usually 40% of someone’s time — understanding what the models do, what data they train on, what constraints they operate under.

If you don’t invest in explainability and data provenance, understanding becomes a constant burden. If you do, it becomes manageable and new hires can ramp faster.

Maintenance

This is usually 30% of someone’s time — keeping systems running, responding to failures, updating models when something goes wrong.

If you don’t invest in monitoring and governance structures, maintenance is reactive and constant. If you do, it’s more predictable and new people can take on pieces of it.

Context switching

This is usually 20% of someone’s time — explaining to other teams what the model does, answering questions, serving as the expert on call.

If you have documentation, explainability, and governance structures, context switching is reduced. If you don’t, every model decision flows through the person who understands it.

Building new things

This is usually 10% of someone’s time.

This is the part people think their AI talent should be doing. And it’s the part they’re doing least.

The companies that manage this well

The organizations that don’t have this problem usually have made investments that seem excessive for the number of models they have:

Data infrastructure

They’ve invested in versioning, lineage, provenance. It seems like a lot of infrastructure for what looks like it could be manual.

But it means new people understand what data trained models. It means understanding builds on itself.

Monitoring infrastructure

They’ve invested in behavioral dashboards, automated alerting, clear definitions of what “healthy” means.

It seems like overhead. But it means when something goes wrong, it’s detected automatically. It’s not “call the person who understands the model and ask them to look.”

Decision documentation

They maintain decision memoranda. Why was this model deployed? What were the constraints? What would change the decision?

This seems like busywork. But it means when something breaks, the diagnosis is much faster. And it means new people can understand the decisions without having to ask the original decision-maker.

Governance procedures

They have escalation paths, decision authority, update procedures.

These seem like bureaucracy. But they mean that governance doesn’t depend on one person’s understanding. It becomes embedded in how the organization operates.

Organizations that make these investments pay a premium for infrastructure. But they reduce the context-switching tax dramatically.

The AI engineer doesn’t have to be everything. They can specialize. They can focus on what they’re good at. They have space to build.

What this means for budgeting

If you’re budgeting for AI, budget for infrastructure as much as you budget for talent.

The traditional model: One data scientist builds the model. One engineer maintains it.

The realistic model: One person builds the model (or the team builds it). Three to five people work on the infrastructure that makes the model actually work: provenance, monitoring, governance, integration, incident response.

If you underinvest in infrastructure, you will pay for it in talent turnover. The people you hire will be doing five jobs poorly, and they’ll leave.

If you invest properly in infrastructure, you’ll spend more upfront, but your talent will last longer and be more productive.

The long-term math

Let’s say you have one AI person doing everything: building models, maintaining provenance, setting up monitoring, managing governance, responding to incidents.

Their salary is $200k. But you turn them over every 18 months. Onboarding costs $80k (including lost productivity). You actually spend $320k per year per person.

If you invest in infrastructure (engineering for provenance, monitoring, governance), you might spend $400k upfront. But now your AI person can focus on modeling. Their job is more interesting. They stay. You keep the same person for 4 years.

Total cost for 4 years: One person at $200k/year + infrastructure investment ($400k upfront spread over 4 years = $100k/year) = $300k/year.

With turnover: Three people * $320k/year (salary + turnover) = $960k.

The upfront infrastructure investment actually costs less.

Where to start

If you have AI engineers and they seem overstretched, the answer isn’t to hire more AI engineers.

The answer is to invest in the infrastructure that would let the people you have actually focus on their core work:

1. Invest in data provenance systems so understanding is documented
2. Invest in monitoring so maintenance is predictable
3. Invest in decision documentation so context doesn’t live in someone’s head
4. Invest in governance so the same decisions don’t have to be made again

This will cost money. But it will be less than what you’re spending on turnover of good people who got burned out doing five jobs.

The talent cost of AI isn’t the salary. It’s what you’re losing while they’re maintaining a system you didn’t invest in properly.

Most vendor relationships in enterprise AI are built backwards.

The company doing the evaluation has a gap. They need a model. They look at the available options, pick one, and then manage the dependency.

That’s backwards from how it should work. The question isn’t “which model should we buy.” It’s “if we depend on this vendor, what are the constraints that creates, and can we live with them?”

This matters because vendor dependency is actually a governance issue, not a procurement issue.

The hidden constraints of vendor dependency

Here’s a scenario that plays out regularly: A company licenses a foundation model from a vendor. They build applications on top of it. The model works well. Everything’s good for two years.

Then the vendor updates the model. It’s a better model. Stronger performance. But it has different behavior characteristics. It generates outputs that are slightly different. The applications that were built on top of the old model have to be updated.

Or the vendor changes their terms. They used to allow you to run the model internally. Now they want you to use their API. You have to redesign your applications.

Or the vendor stops supporting the version you’re using. They want you to upgrade. The upgrade comes with changes you’re not prepared for.

In each case, the company that built the dependency is now constrained. The vendor changed something. The company has to adapt.

This is fine if you have one model. This is a serious problem if you’ve built dozens of applications on top of the model and that model is embedded in your decision-making processes.

Why this becomes a governance problem

Most companies don’t think about vendor dependency until it’s a problem.

They’re focused on the model’s performance right now. Does it do what you need? Does it have good accuracy? Is it cost-effective?

They’re not focused on the second-order question: What happens if this vendor changes?

By the time that question becomes urgent, the dependency is already deep.

I watched a company build a suite of applications on top of an open-source model. They contributed improvements. They optimized for their use case. Six months in, they had 12 applications in production, all depending on that model.

Then the model’s maintainers made a change. Not a breaking change, exactly, but a change that affected how the model handled certain types of input.

The company had to choose: Spend weeks updating all 12 applications, or fork the model and maintain their own version.

They forked. Now they’re maintaining their own version of a model that’s being actively developed by a community. Their fork diverges. Updates from the community have to be manually merged. New bugs come out of the original model. Do they apply those fixes to the fork?

Within two years, maintaining the fork cost more than building their own model would have.

The company wasn’t paying attention to the governance implications of the vendor dependency. They were paying attention to whether the model worked.

What questions actually matter

The evaluation criteria for any AI platform partner aren’t the ones most companies focus on.

Most companies ask: What’s the model’s accuracy? How much does it cost? What’s the API latency?

These questions aren’t aimed at any specific category of provider. They apply to every AI platform relationship—cloud services, open-source deployments, and proprietary APIs alike. You should also ask:

1) How often does this model change?

And importantly: How do you define a change? Is it just updates to the weights? Does it include changes to architecture? Changes to inputs or outputs?

If you’ve built applications that depend on specific model outputs, and the model changes behavior, your applications might break.

You need to understand the cadence. Is the model updated quarterly? Whenever the vendor feels like it? When there’s a major improvement?

2) What’s your path if the vendor stops supporting this?

Not “if they go out of business,” but if they decide to deprecate this model or change their business model.

Can you download the model and run it yourself? Are there licensing restrictions? If you needed to migrate to a different model, how long would that take?

If the answer is “you’d have to rewrite everything,” you should factor that into your decision.

3) What changes are backwards compatible?

If the vendor updates the model, will your applications still work? Or will you have to retest and redeploy?

Some vendors are thoughtful about backwards compatibility. Others aren’t. If you depend on them, you need to know.

4) What’s the data retention policy?

If you’re using their API, are they storing your data? For how long? What are they using it for?

If they’re using your data to improve their models, that’s fine — but you should know it and be comfortable with it.

5) What happens to my applications if you change your terms?

If they say “we’re discontinuing the API and you have to use our cloud service,” what options do you have?

If they change pricing dramatically, what’s your recourse?

Most vendors won’t want to answer this question directly. But you should ask it anyway.

The vendor as governance decision

Here’s why this matters at the governance level: When you decide to use a vendor’s model, you’re not just deciding on that model’s performance. You’re deciding to accept a set of constraints on your system.

You’re deciding to accept that this vendor can change how your model behaves, and you’ll have to keep up.

You’re deciding to accept that if the vendor’s business changes, you might have to change yours.

You’re deciding to accept that you’re dependent on someone else’s roadmap.

All of these are fine — dependencies are inevitable. But they should be explicit decisions, not accidental ones.

Organizations that handle this well usually:

Map the dependencies

They know exactly which applications depend on which models, and what would happen if those dependencies changed.

Diversify where sensible

If they have applications that could work with multiple models, they build for flexibility. They’re not locked into one vendor.

Maintain fallback options

They have a plan if the vendor changes. Maybe it’s “we’ll migrate to this alternative,” or “we’ll build our own version,” or “we’ll redesign the application.” But they have a plan.

Make it a deliberate choice

When they decide to depend on a vendor, they’re making an explicit decision about the constraints that dependency creates.

What to do if you’re already dependent

If you’ve already built applications on vendor models and you’re realizing the dependency is deep, you can’t undo it quickly.

But you can start:

1. Map the dependencies. Which applications use which models? What would it cost to migrate?

2. Start reducing dependency where sensible. Can you build flexibility into your architecture so you’re not locked into one model?

3. Ask the hard questions. Especially the ones about what happens if the vendor changes.

4. Make explicit decisions about where to accept dependency and where to avoid it.

The companies that end up in bad situations usually didn’t plan for vendor risk. They just picked a model that worked and kept building on it.

The companies that manage vendor relationships well ask the governance questions upfront. They know what constraints they’re accepting. And when things change, they’re not surprised because they planned for the possibility.

I’m going to say something that will sound heretical: The best governance frameworks are the ones most organizations won’t recognize as governance frameworks.

They don’t look like frameworks. They look like a decision. Someone with authority decides whether to use a model. They document why. They monitor it. Something changes, they update the decision.

No committee sign-offs. No tiered approval processes. No model risk assessment templates. Just conscious choice.

The reason most governance frameworks don’t look like that is because they’re designed for the wrong problem.

What frameworks are actually designed for

Most organizations build AI governance frameworks for two reasons:

Reason 1: Audit readiness

They want to be able to show regulators that they have a system. They want processes. They want documentation. They want to demonstrate that they’re taking this seriously.

Reason 2: Risk distribution

They want to distribute risk across multiple people and functions so that nobody bears sole responsibility if something goes wrong. If something fails, there’s a paper trail showing that the model was reviewed, validated, and approved by multiple parties.

Both of these reasons are understandable. Neither of them produces governance that actually prevents problems.

You can tell because most frameworks look the same: A model is proposed. It goes through a validation process (multiple teams review it). It gets approved (often by committee). It’s deployed with monitoring. If something goes wrong, the organization can show the process and say “we did what we were supposed to do.”

The problem is that this process doesn’t actually prevent bad decisions. It just documents that a process existed.

The framework that doesn’t prevent problems

Here’s a concrete example: A company built an AI governance framework with five layers of review:

1. Data quality review (is the training data sound)
2. Model validation review (does the model perform as expected)
3. Fairness assessment (does the model produce disparate impact)
4. Business impact review (does deploying this make sense)
5. Executive approval (final sign-off)

On paper, this looks comprehensive. In practice, it was almost useless.

Here’s what happened: A model came through for approval. The data quality team reviewed it and approved it. The validation team checked the metrics and approved it. The fairness team looked at aggregate statistics and approved it. The business team saw revenue potential and approved it. The executive signed off.

The model got deployed. Within months, it was producing outcomes that everyone could see were problematic. The model was systematically making worse decisions for a specific population segment.

Then the conversation went:

• “The fairness team reviewed this.”
• “Yes, we looked at the aggregate statistics.”
• “Did you notice the disparate outcomes?”
• “No, the aggregate statistics looked fine.”
• “The model is producing disparate outcomes.”
• “Then the environment changed, or we made a wrong assumption.”
• “Your approval was supposed to prevent this.”
• “Our approval was that we followed the process.”

In the end, the governance structure had done exactly what it was designed to do: document that a process existed and distribute responsibility across multiple parties.

What it hadn’t done: actually prevent a bad outcome.

Why the framework didn’t work

The framework failed for a specific reason: Nobody at any step had the authority or responsibility to say “I’m uncomfortable with this, and we’re not deploying it until I understand why.”

Each reviewer had a specific domain: – Data quality: “Is the data clean?” Yes. Approve. – Validation: “Does the model perform?” Yes. Approve. – Fairness: “Do the aggregate statistics look ok?” Yes. Approve. – Business: “Is there business value?” Yes. Approve. – Executive: “Has everything been approved?” Yes. Approve.

At no point did anyone have to say: “I understand all the reviews and I’m choosing to take the risk of deploying this model given these constraints.”

It was a process without a decision. Everyone followed their part. The system reached approval. Nobody decided.

What actually works

The governance structures that actually prevent problems have a different pattern. Usually there’s a person (maybe a small group, but not a committee) with clear authority. Their job is to decide: Should we use this model?

When they decide, they’re deciding based on: – Understanding what the model does – Understanding the environment where it will be used – Understanding the constraints and risks – Deciding that deployment is acceptable given those constraints

This person doesn’t validate every detail. They rely on expert input. But they’re the one who synthesizes that input and makes the choice.

Then they document: “We are deploying this model. Here’s why I’m comfortable with that decision. Here are the constraints I’m putting around it. Here’s what would make me reverse this decision.”

Then when the environment changes or something unexpected happens, this person (or their successor) reviews and updates the decision.

This structure looks less “governance-like” than a matrix of committees and approval workflows. But it actually prevents problems.

Here’s why: The decision-maker has to be able to defend the decision. If the model fails in an unexpected way, the question isn’t “did we follow the process” but “would this decision still be defensible given what happened.”

This creates a different incentive structure. The decision-maker actually thinks about what could go wrong, not just whether the checklist is complete.

The problem with frameworks

Here’s the uncomfortable truth: Comprehensive frameworks usually create false confidence without creating better outcomes.

They create the appearance of control. The appearance matters for audits and regulatory meetings. But it doesn’t actually prevent bad decisions.

What prevents bad decisions is: – Someone with authority and accountability deciding whether to deploy – Enough expertise available to inform that decision – The decision-maker understanding and accepting the risks – Clear monitoring so that if something unexpected happens, it gets escalated back to the decision-maker

You could write this on a page. You don’t need a framework.

But the reason organizations build frameworks is because nobody believes that would survive an audit. If something goes wrong and your governance is “we had a thoughtful person decide,” that feels insufficient.

The regulatory environment hasn’t clarified that this is actually better than “we had a five-layer review process.” So organizations build the five-layer review process.

Starting over

If you were building AI governance from scratch — not trying to be audit-ready, just trying to prevent bad outcomes — here’s what you’d do:

Identify the person or small group with authority to make deployment decisions.

Make sure they have access to expert input: data scientists, compliance officers, business leads, whoever informs the decision.

Have them document the decision: “We’re deploying this model. Here’s why. Here are the constraints. Here’s what would change the decision.”

Set up monitoring. When something diverges from expectations, it escalates back to the decision-maker.

When the decision-maker would revert the decision if something unexpected happened, that’s governance.

When the framework is designed so that the decision-maker can never reverse course because they’re locked in by the approval process, that’s theater.

What to do if you already have a framework

If your organization already has a comprehensive governance framework (you probably do), you can’t just throw it out. But you can add a layer to it.

Add a decision layer. Someone with authority needs to actively decide “we deploy this given these constraints.” They’re not just approving at the end of the process. They’re deciding.

And they need to know that their job is to make a defensible decision, not to validate that everyone did their part.

The organizations that end up with the best outcomes usually have a framework (for audit readiness) and a decision (for actual risk management).

Most organizations have the framework and call it governance. They’re surprised later when the framework didn’t prevent a problem.

Your legal team has thought about AI liability. They’ve probably built a framework around it.

They probably haven’t thought hard enough about the scenario where the liability is clear and there’s nothing to sue.

This is the quiet liability that keeps me engaged with this space: not what happens when someone gets wronged by your AI system, but what happens when everyone knows someone got wronged and there’s nobody to hold accountable.

The accountability dissolution problem

Here’s the pattern: Your organization deploys an AI system. It makes a bad decision. Someone is harmed. The decision was technically made by the AI system, which is a tool.

Your company says: “We didn’t make the decision. The model did. We relied on the model’s recommendation because it had been validated and it was operating as designed.”

The individual who was harmed says: “I was harmed. Someone should fix this.”

Your company says: “We didn’t intend harm. We built reasonable controls. The model was performing as expected.”

The individual says: “I don’t care about your controls. I was denied credit, fired, or denied medical treatment. Someone should be accountable.”

At this point, the organization has an accountability void. The individual was harmed. The organization says it didn’t make the decision (the model did). So who made the decision?

Usually the answer is: nobody. The decision happened. Nobody made it. Nobody can be held accountable.

This is different from negligence liability, where you could argue the organization was careless. This is structural liability, where you could argue the organization constructed a system where decisions happen without anyone deciding.

Why this matters legally

You might think: “This is actually good for us. If nobody made the decision, nobody can be held liable.”

That’s backwards. Courts and regulators care about this differently than you’d expect.

The emerging pattern in how regulators and courts approach this is: If you’ve deployed a system that makes decisions affecting people, someone at your organization bears responsibility for those decisions.

The legal challenge is that companies are trying to distribute responsibility so widely that it disappears. The model recommends. A human reviews (but only if there’s time). The system flags concerns (but only for certain kinds of concerns). Business rules override the model (in certain cases).

At each step, someone is supposed to be responsible. In practice, nobody is.

A lending officer reviews a model recommendation for a loan that gets denied. If they just accept the recommendation without independent judgment, they haven’t made a decision. They’ve deferred to the model. If something goes wrong, can you hold them accountable for a decision they didn’t make? Not really.

A compliance officer reviews the model for bias. They check statistical metrics, see that the model is performing as expected overall, and approve deployment. Six months later, the model is producing disparate impact. The compliance officer says: “I reviewed it carefully and it was performing as expected.” The business says: “We followed the compliance officer’s assessment.”

Who’s accountable? Nobody really. Everybody did their job.

How regulators are approaching this

The regulatory approach is starting to shift. Instead of asking “who made the bad decision,” they’re asking “why did your system make a bad decision without anyone noticing?”

The focus is moving from individual accountability to system accountability.

Organizations that have been through regulatory reviews often discover a pattern. A regulator asks: “Your lending model denied this application. Walk me through how someone with decision authority reviewed this recommendation.”

The organization responds: “Our model is trained to be reliable. We monitor it quarterly. If it falls below performance thresholds, we retrain.”

The regulator pushes back: “That’s not what I asked. Someone made this specific decision. Tell me who. Tell me what they knew. Tell me what decision authority they had.”

The organization can’t answer, because nobody actually had decision authority for that decision. The model recommended it. If the recommendation fell outside certain parameters, it escalated to a human. But in this case, it didn’t. So the decision happened without anyone deciding.

The regulatory position is clear: “You can’t build a system where decisions happen without someone deciding. Either the model has authority (in which case you need to defend the model’s decision) or someone has authority (in which case you need to show they exercised it).”

This is the liability that most companies haven’t thought through. Not “the model was wrong,” but “nobody had the authority to decide this, yet a decision was made.”

Why traditional frameworks miss this

Most companies approach AI liability through the lens of traditional product liability. You build something, it fails, you’re liable if you were negligent. Your defense is: “We were careful. We tested it. We built reasonable controls.”

This works fine for physical products. A car has a brake failure, you’re liable if you failed to test the brakes adequately.

It doesn’t work for AI systems because the question isn’t just whether the system was tested, but whether anyone was actually responsible for using it.

A framework that addresses traditional product liability but doesn’t address decision authority liability is incomplete.

Organizations usually have the product liability framework: testing, monitoring, control standards. They rarely have the decision authority framework: who decides what, under what authority, with what information.

What actually protects you

The liability structure that actually protects an organization isn’t “we have no responsibility because the model decided.” It’s “someone in the organization had clear authority and responsibility, and they made a conscious decision.”

This sounds crazy until you realize what it means: You’re more protected legally if you have one person who decides “we’re going to use this model and here’s why I’m comfortable with that decision” than if you have elaborate controls but nobody clearly deciding.

The elaborate controls are good. But they don’t protect you if they’re controls without decision authority.

Organizations that handle this well usually:

1) Make decision authority explicit

There’s a person (not a committee, ideally not a meeting) who has the authority to decide “we use this model for this decision.”

2) Document the conscious choice

That person writes down (or records) why they made that choice. What did they know about the model? What risks were they accepting? What monitoring would they put in place?

3) Hold people accountable for that decision

If something goes wrong, the question is: Was that decision reasonable given what was known at the time? Not: Did the model fail? Did the model work properly?

4) Design escalation appropriately

If the model produces an unexpected outcome, there’s a clear escalation path. Someone with authority reviews it. They decide whether to override, whether to investigate further, whether the original decision authority still stands.

The liability protection comes from showing that someone thought about the risks and decided to proceed anyway. Not from showing that you had no responsibility.

What this means for governance

This fundamentally changes how you should think about AI governance.

It’s not just about frameworks and controls. It’s about clear decision authority and conscious choice.

If your governance structure is a matrix of approvals where nobody clearly decides, you’re building liability, not protecting against it.

If you have clear decision authority and clear documentation of conscious choice, you’re in a much better position.

This is why the accountability structure from Day 31 matters. If you don’t have clear decision authority and clear escalation processes, you’re not just failing at governance—you’re building legal exposure.

Where to start

If you have AI systems in production, start here:

For each system, answer: – Who has the authority to decide whether we deploy this? – What did they know about the model’s risks when they decided? – If something goes wrong, who reviews that outcome and decides what to do?

If you can’t answer these clearly, you have a liability problem. Not because your model is bad, but because nobody decided to use the model.

Fix that first, before you invest in more controls or more comprehensive frameworks.

Your SIEM team knows how to monitor systems. They monitor uptime, latency, error rates, resource utilization. They know what “the system is healthy” means.

They probably don’t know how to monitor whether an AI system is making good decisions.

These are different problems, and the confusion between them is creating a gap in enterprise AI governance that most organizations don’t realize they have.

Why system monitoring isn’t enough

Here’s the pattern I see: An organization deploys an ML model. They integrate it into their system. The operations team sets up monitoring. They monitor whether the model API is responding (yes), whether predictions are generating (yes), whether the system has adequate compute resources (yes).

Everything looks fine. Six months later, the model is systematically making worse decisions than it was when it deployed. But the operational monitoring shows no problems. Uptime is perfect. Latency is clean. Resources are healthy.

The model is failing invisibly.

This happens because system monitoring answers different questions than behavioral monitoring.

System monitoring asks: Is the system running? Is it available? Is it processing data? Are resources adequate?

Behavioral monitoring asks: Are the model’s decisions changing? Are they still making sense? Are there groups getting systematically different treatment? Is the model still producing what we designed it to?

A system can be perfectly healthy from a system perspective and producing degraded outputs from a behavioral perspective. These require different monitoring approaches.

What behavioral monitoring actually measures

Organizations that get this right usually monitor multiple layers:

1) Statistical drift

Is the model’s accuracy degrading? Are its predictions diverging from expectations? This is the most straightforward behavioral monitoring and most organizations try to do it.

It’s also insufficient. A model can maintain perfect accuracy overall while failing systematically in specific segments. A lending model might be as accurate as ever, but recommending worse terms to certain demographic groups than others. A hiring model might be as accurate overall but recommending different candidates for certain job categories.

2) Outcome monitoring

Are the decisions the model drives producing expected outcomes? If the model is recommending treatment for patients, are those patients getting better? If the model is recommending job candidates, are those candidates succeeding in the role?

This is harder to measure because outcomes take time. But it’s more important. A model can be statistically accurate while driving poor business outcomes because the decision context has changed.

3) Behavioral fairness

Are segments of the population getting systematically different treatment? Are predicted probabilities distributed differently across groups? Are decision rates or recommendation rates different?

This requires benchmarking. You need to know what “fair” means for your use case. Then you monitor whether the model is maintaining that standard.

Most organizations don’t do this routinely. They check for bias during validation, assume it stays stable, and move on.

4) Adversarial monitoring

Are there patterns in the data that the model hasn’t learned? Are there adversaries trying to game the system? Are there edge cases the model handles poorly?

This is the most sophisticated monitoring, and it requires building time and resources into the process. But it’s critical for systems that are working in adversarial environments (fraud detection, abuse prevention, security).

Where most organizations fail

The failure usually happens at the organizational boundary. Your operations team owns system monitoring. Your data science team might own model validation. Your business team owns outcomes. But nobody owns behavioral monitoring continuously.

So what happens is:

• Operational monitoring is excellent. You’d know instantly if the API goes down.
• Model validation happened once at deployment. It’s not continuously updated.
• Outcome monitoring happens in business reviews. It’s infrequent and high-level.
• Behavioral fairness monitoring is either missing or ad-hoc.

The system is set up to catch when a model fails to produce predictions. It’s not set up to catch when a model produces degraded predictions.

I watched a company deploy a model to help prioritize customer support tickets. The system monitoring showed perfect health. The model was generating predictions for 100% of tickets. The infrastructure was solid.

But six months in, the model was systematically de-prioritizing tickets from customers in certain regions. The accuracy was stable (it was still correctly predicting urgency). The decision pattern had shifted (urgent tickets from one region were being treated as less urgent). Nobody noticed because outcome monitoring was infrequent, and behavioral monitoring didn’t exist.

When they finally caught it through a customer complaint, they had months of bad decisions to remediate.

Building behavioral monitoring

If you’re thinking about how to set this up, the pattern that works usually involves:

Establish baselines

Before deploying the model, understand what “normal” behavior looks like. What’s the accuracy distribution? What are the outcome rates? What does the decision distribution look like across important segments?

This baseline becomes the reference point for monitoring.

Continuous measurement

Set up dashboards that track the metrics that matter: – Accuracy and its distribution across segments – Outcome rates (how often recommended actions succeed) – Decision distribution (is treatment/recommendation rate changing?) – Fairness metrics (is outcome distribution stable across protected attributes)

These should be tracked as continuously as your system health metrics.

Alerting on meaningful changes

Accuracy might vary by day or week. That’s fine. Accuracy systematically degrading is not fine. Set up alerts that trigger when meaningful changes occur.

The challenge is defining “meaningful.” A 1% accuracy drop in one segment might be noise. A 5% accuracy drop in another might be significant. It depends on your use case.

Postmortem culture

When something goes wrong, treat it like a system incident. What changed in the environment? What changed in the data? Was the model’s behavior appropriate given the change? Would monitoring have caught this earlier?

Use the postmortem to improve the monitoring for next time.

The hard part

The hard part isn’t the technology. You can build behavioral monitoring dashboards. This challenge exists across the observability landscape, regardless of tooling choice. The hard part is organizational. You have to decide:

Who owns monitoring? Not system health. Behavioral health. Who’s responsible for knowing whether the model is still making good decisions?

If the answer is “the data science team,” they need time and resources for continuous monitoring. They can’t just build the model and hand it off.

If the answer is “the operations team,” they need to understand that model monitoring is different from system monitoring, and they need to learn how to do it.

If the answer is “nobody,” you’re setting yourself up for the invisibly degrading model.

Most organizations avoid making this decision, which is why most organizations have gaps in their behavioral monitoring.

Where to start

Pick your highest-impact model. Pick 3-4 behavioral metrics that matter for that model: – Accuracy (or whatever your primary success metric is) – Something about equity (decision rates across groups, outcome distribution) – Something about the business (customer satisfaction, employee performance, whatever the model is trying to optimize for)

Set up a dashboard that shows these metrics over time. Look at it every week. When something changes, understand why.

That’s behavioral monitoring. It’s not complicated. But it requires someone to own it and actually do it.

The organizations that get this right know what their models are actually doing, not just that they’re running. The ones that don’t usually find out when things have already gone wrong.

A reasonable question to ask of any AI system in production: “What data did this model train on?”

Most large organizations cannot answer that question with specificity.

This is a bigger problem than it sounds. It’s not just an audit inconvenience. It’s a blocker to understanding your own model’s behavior, managing its risks, and defending its decisions.

Why this matters more than most organizations realize

Imagine a scenario that happens more often than anyone discusses publicly: Your lending model makes a recommendation that looks wrong for a specific demographic segment. The recommendation is mathematically sound. The model is performing as expected overall. But in this segment, something is off.

Your first question should be: What data did the model train on for this segment?

If you know the answer — “We trained on 47,000 approved loans in this geography from 2018-2020, and here’s the selection bias in that dataset” — you can investigate. Maybe the selection bias is the problem. Maybe the economic conditions of that time period are no longer representative. Maybe lending practices have changed.

If you don’t know the answer, you’re stuck. You can look at accuracy metrics (which look fine) and feature importance (which shows expected patterns) and conclude nothing is wrong. But something feels wrong. You just can’t diagnose it because you don’t know where the model’s learned patterns came from.

This happens more than you’d think. And when it does, the organization usually takes one of three paths:

Path 1: They remove the feature that seems problematic, retrain the model, and hope the problem goes away. They’ve changed the decision-making process without understanding why.

Path 2: They add a rule (don’t recommend this for this segment) that overrides the model when it hits the problematic pattern. They’ve built a patchwork system that’s harder to maintain and monitor.

Path 3: They accept the problem and let the model make the recommendation anyway, documenting that they’re aware of the concern. They’re betting that enough models get questioned that auditors won’t focus on this one.

None of these are good outcomes. All three are symptoms of not understanding your training data.

Where data provenance usually fails

Organizations typically get tripped up in three places:

1) Data collection and compilation

You’re training on multiple datasets. You’re using internal transaction history, third-party enrichment data, and purchased datasets. These come together in a data pipeline. Months or years later, someone asks: “What exactly went into the training set?”

The answer usually involves hunting through old notebooks, asking data engineers who’ve moved to other teams, or looking at Hadoop jobs that have been rerun dozens of times with different parameters. What you want to know is clear and simple. What you can actually discover is murky and partial.

Most organizations solve this by documenting it after the fact, which is fine until the documentation is incomplete or wrong. Better organizations build data versioning into the pipeline from the start. They tag versions, store lineage information, and treat training data as an artifact that can be retrieved.

2) Data transformation

The raw datasets get processed. They’re cleaned, features are engineered, outliers are handled. This transformation process is usually in code. That code evolves. Parameters change. One version of the code created the training data. A later version handles new data differently. Now you’ve got a production system where data is processed one way, and historical data was processed another way.

The question becomes: Is the model failing because the environment changed, or because we changed how we process the input?

This is fixable but requires discipline. You need to version the feature engineering code just like you version the model. You need to be able to say “this model trained on data processed by version 2.3 of the ETL pipeline.” Then when something goes wrong, you can reproduce that exact data processing.

3) Consent and licensing

More problematic: You’ve assembled training data from multiple sources, and some of those sources have licensing restrictions or data usage agreements. You trained the model with data you shouldn’t have had access to, or you used data in a way that violated the agreement.

This is harder to discover after the fact. The model is in production. The historical data is mixed with other data. You can’t untrain what you learned. You either acknowledge the problem (legal exposure), or you hope nobody notices.

The organizations that avoid this usually have data governance at the point of procurement. Before data comes into the pipeline, they know: Where did it come from? What can we use it for? Are there restrictions?

What working data provenance actually looks like

These are emerging industry best practices. Organizations that handle this well usually have systems that let them:

Answer the basic questions

“What data trained this model?” You should be able to specify a dataset version or date range and get back a manifest of what went into training.

“Has that data been modified?” You should be able to tell whether the data was processed differently at training time vs. now. You should be able to reproduce the historical processing if you need to.

“What are the licensing and consent restrictions?” You should know which parts of the training data have restrictions and what those restrictions are.

These aren’t complicated questions. They’re just hard to answer in systems that weren’t built with provenance in mind.

Link provenance to model behavior

“We know this model trained on 2018-2020 data from this geography. Here’s what economic conditions and lending practices looked like then. Here’s what they look like now. Here’s where we’d expect the model to diverge.”

This conversation requires you to know the training data. Then you can make an argument: “The model learned patterns from this time period and this context. If the context has changed, we should expect divergence.”

Establish audit trails

If data provenance is clear, auditing becomes more straightforward. You can trace decisions back to training data. If something looks wrong in a cohort, you can investigate whether that’s a modeling issue or a data bias issue.

Regulators are increasingly asking for this. “Your model recommended X for this segment. Tell us about the training data for that segment and why you think the recommendation is fair.” If you can’t answer, that’s a problem.

How to start fixing this

If you have models in production and you can’t confidently answer “what data trained this model,” you’re not alone. This is an endemic problem in large organizations.

Start with the highest-impact models. For each one, invest in understanding the training data:

Document it. Write down what datasets went in, where they came from, what processing happened. Even if it’s incomplete, it’s better than nothing.

Version it. From this point forward, treat training data and feature engineering as versioned artifacts. Tag them. Store them. Know what version produced what model.

Connect it to decisions. When something goes wrong, ask whether the decision-making divergence is because the model failed or because the environment changed. You can’t answer that without knowing the training data.

Link it to governance. Your model risk management should include understanding the training data and its limitations. “The model is statistically valid” isn’t enough if you don’t understand what it learned.

Data provenance takes effort. But it’s the foundation for understanding your own models and managing their risks. Organizations that have built it out are the ones that can confidently explain their model decisions. The ones that haven’t are the ones that eventually end up in uncomfortable regulatory meetings asking the question they can’t answer: “What was your model actually trained on?”

There’s a type of organizational problem that shows up in almost every large company that tries to scale AI decision-making. I call it the escrow problem because the decision gets stuck in a state where it’s technically made but functionally unmade.

Here’s how it usually starts: Your organization deploys a model. The model makes recommendations. The humans are supposed to review those recommendations before decisions are finalized. This is good risk management.

Then the model starts producing recommendations faster than the review process can handle them. And something breaks.

How the problem reveals itself

I watched it happen at a company with a lending model that ran daily. The model analyzed applications and generated recommendations. For each recommendation, there was an escalation process: If the model’s confidence was below a certain threshold, a human reviewer had to approve before the decision could be finalized.

This was a reasonable control. Humans would review uncertain cases and apply judgment.

The problem: The model was running faster than the humans could review. By the time a human looked at a recommendation, the business context had changed. A customer’s credit situation might have moved. Market conditions might have shifted. The recommendation that made sense when it was generated was now stale.

The company had two choices. They could slow down the model (run it less frequently). Or they could let decisions move forward without full review.

They did what most organizations do: They expanded the category of decisions the model could make without human review. They raised the confidence threshold. They created exception processes. They added more reviewers.

They never solved the fundamental problem: The decision authority couldn’t move as fast as the decision-generating process needed it to move.

Why this matters

The escrow problem has real consequences, and they’re usually invisible until something breaks.

First, it erodes decision quality. When humans are reviewing decisions that are already hours old, they’re not reviewing whether the decision was right then — they’re reviewing whether it’s still right now. These are different judgments. The review becomes reactive (the decision is already in flight) instead of deliberative (should we make this decision?).

Second, it creates decision drift. The organization sets out with human oversight for all AI recommendations. Gradually, as the volume overwhelms the review process, the definition of which decisions require oversight changes. But there’s no formal decision about this drift. It happens through accumulated exceptions and threshold adjustments. Six months in, the decision-making process is fundamentally different from what was designed, but nobody made a choice about it.

Third, it creates liability exposure. The organization built a control system (human review) and published that as their governance. When something goes wrong in a decision that bypassed the review, they have to explain why the published process wasn’t followed. Usually the answer is “our volume exceeded our capacity,” which is not a good explanation in a regulatory meeting.

What’s really happening

The escrow problem is usually a symptom of something deeper: The organization hasn’t actually decided who has the authority to make decisions.

It sounds like they have: “The model recommends, humans review, we finalize.” But what they’ve really said is: “The model recommends, humans might review if we have time, and then we do whatever happens next.”

That’s not decision authority. That’s hoping someone with judgment will be available.

In organizations that handle this well, decision authority is clear and it’s mismatched to speed:

Fast decisions — The model decides, using its learned judgment. Humans monitor whether the model’s decisions are working as expected, but they’re not involved in every decision. This is appropriate for high-volume, low-stakes decisions where the model’s track record of accuracy is proven.

Medium-speed decisions — The model recommends, and a human approves or overrides based on specific criteria. This requires that the human can make the decision in the timeframe the model generates recommendations. If they can’t, you have the escrow problem.

Slow decisions — Humans make the decision. The model provides input (analysis, risk assessment, comparable cases), but the human has the final authority and the time to exercise genuine judgment. This is appropriate for low-volume, high-stakes decisions.

The problem is that most organizations try to run medium-speed decisions through a slow decision process. They want the model’s speed, the human’s judgment, and the risk mitigation of oversight. They can’t have all three.

Choosing what to give up

Organizations that solve this problem usually make one of three choices:

Choice 1: Give up speed.

Slow down the model. Run it weekly instead of daily, or quarterly instead of monthly. Make sure the review process can keep up. This is the right choice if the business value of speed is low and the risk of escrow is high. It’s usually chosen by regulated organizations where decision documentation is critical, or where the stakes are high.

Choice 2: Give up human oversight.

Let the model decide for high-confidence cases. This is appropriate when the model’s track record is proven and the cost of wrong decisions is manageable. It requires confident monitoring (you actually know when the model fails) and a fallback process (if something goes wrong, there’s a clear escalation). It’s the choice many companies make when they realize their volume will never fit through human review.

Choice 3: Give up some decisions.

Don’t deploy the model for use cases where you can’t deliver fast enough human judgment. Keep it as an internal tool (gives humans better information) rather than a decision system (actually makes decisions). This is the right choice when you’re not sure about the model’s track record or when the stakes are high.

Most organizations don’t make any of these choices explicitly. They drift into Choice 2 (give up human oversight) while claiming they’re still running Choice 1 (humans review) and never actually doing Choice 3 (accepting limited scope).

How this connects to everything else

The escrow problem is actually a decision-rights problem. It shows up when you haven’t clearly assigned decision authority and you haven’t accepted the constraints that come with that assignment.

Organizations that have solved it clearly are usually clear about something else: Who actually owns the decision? Not “who reviews the recommendation,” but who has the power and the responsibility to say yes or no?

Once you know that, the speed problem becomes tractable. If the model owner has the authority to make decisions within certain guardrails, they can decide to expand those guardrails when volume exceeds human review capacity. It’s their call to make. If human reviewers have the authority, they can decide how much volume they can handle and what happens when volume exceeds capacity.

But most organizations have assigned the authority to someone in a governance role (a committee, a compliance team, a risk committee) who isn’t actually involved in day-to-day decision-making. So when volume increases, nobody has the authority to make the choice about what to do.

Where to start

If you’re building AI decision systems, ask this question early: What’s the slowest decision-making process you can tolerate for this use case?

If the answer is “we need to decide in minutes,” then human review isn’t your control mechanism. Your control mechanism is monitoring and fallback.

If the answer is “we can tolerate a few hours,” then you need enough human capacity to review decisions on that timeline. That’s a real constraint. Build for it.

If the answer is “we can tolerate a few days,” then you can probably build a human-review process that works.

But don’t answer with what you wish were true. Don’t say “humans review everything” if your volume makes that impossible. That’s how you end up in escrow.

The organizations that manage AI decision-making well know their constraints and design to them. They don’t promise speed, judgment, and oversight unless they can deliver all three. Most organizations promise all three and end up delivering none of them well.

Every financial services firm has a Model Risk Management framework. Most of them haven’t actually used it to catch anything.

This isn’t because the frameworks are poorly designed. It’s because they’re designed for a problem that doesn’t quite exist, deployed against risks that are invisible to the framework.

I’ll explain, because this is one of those areas where understanding the gap tells you a lot about how to think about AI risk more broadly.

What model risk frameworks are built to catch

The standard Model Risk Management framework came from Basel guidance and financial regulator requirements. It evolved around a specific category of problems: When you build a quantitative model that feeds decision-making, and the model is wrong in a systematic way, you make bad decisions at scale.

So the framework focuses on: Is the model valid? Is it accurate? Does it perform as expected? When something goes wrong, what’s the model’s role in the failure?

This is useful for catching models that are mathematically broken, statistically invalid, or performing worse than expected. It catches overfitting, concept drift, and data quality issues.

It does not catch most of the risks that actually matter.

What’s invisible to standard frameworks

Here’s a pattern I see repeatedly: An organization implements a model, validates it thoroughly, puts it into production, and monitors it diligently. The model performs exactly as expected. Accuracy is stable, validation metrics are healthy, everything looks fine.

Then something goes wrong. The model wasn’t wrong — the decision-making context around the model was wrong.

A lending model that accurately predicts default risk produces a disparate impact on protected classes. The model is statistically sound. It’s predicting correctly. But the business decision to use it for loan decisions in an underserved market was wrong because the training data came from a time when lending practices were different in that market.

A trading model that accurately captures market correlations suddenly produces an unexpected loss because correlation structure changed, and nobody had decided what to do if that happened. The model did what it was trained to do. The decision authority around how to handle model uncertainty had never been established.

A workforce optimization model that accurately predicts which positions will have retention problems is deployed to recommend layoffs, but nobody had decided whether a model designed to predict problems should drive who gets laid off. The model worked. The application was wrong.

In each case, the model risk framework shows green. The validation is sound. The monitoring is clean. The problem isn’t in the model — it’s in the decision system around the model.

Why frameworks miss this

Standard model risk frameworks are built to answer the question: “Is this model trustworthy as a mathematical instrument?”

The question that actually matters for most enterprise AI risk is different: “Is the decision we’re making with this model appropriate?”

These are correlated but not identical. You can have a trustworthy model used for an untrustworthy decision. And you can have a somewhat noisy model used thoughtfully where the human judgment around it is sophisticated enough to manage the risk.

Most frameworks check the first box and assume the second one is handled elsewhere. Spoiler: It’s usually not handled at all.

Organizations that handle model risk seriously usually do a second set of work that sits beside the traditional framework. It asks different questions:

What is this model actually being used for? Not in theory, but in practice. How is the decision actually being made? Who’s being affected? What would a bad outcome look like?

Why are we using a model for this decision at all? Sometimes the answer is “because it’s more accurate than human judgment.” Sometimes it’s “because it’s faster.” Sometimes it’s “because it removes the appearance of bias even though the input data might be biased.” These matter, because they change what your risk actually is.

What’s the decision being made if the model is wrong? If the model’s confidence is low, do you have a process for human override? If the model fails in a specific way, what’s the next decision? If you can’t describe what you do when the model is wrong, you don’t understand your own risk.

What would falsify our assumptions? You’re assuming the training data is representative of the population you’re applying the model to. What would prove that wrong? You’re assuming the decision context is stable. What would change it? You’re assuming the model’s accuracy is stable. How would you know if it wasn’t? If you can’t describe the conditions that would break your model’s usefulness, you’re building on faith, not on reasoning.

Is there a human in the loop who could reasonably override the model? Not should there be — could there be? If the model recommends something and a human thinks it’s wrong, can they say no? If not, you’ve built a system where the model is actually making the decision, and you should be honest about that. If yes, you need a process for when humans should override and what you learn from overrides.

The uncomfortable conversation

Organizations that separate themselves from the crowd usually do something harder than updating their framework. They have what I’d call the “uncomfortability conversation.”

Someone sits down and lists all the ways the model could produce a bad outcome that isn’t captured in the model risk framework:

• Accurate prediction of biased historical patterns (model correct, decision wrong)
• High accuracy but low coverage (model works for some populations, not others)
• Decision context change (model was trained in one business context, deployed in another)
• Correlation breakdown (model captures historical correlations that don’t predict future)
• Population drift (model trained on one population, applied to another with different characteristics)

For each, they ask: Do we have a process to catch this? Is this monitored? Would someone know if this happened?

Usually the answer is no. Usually the model risk framework is looking at the one thing that won’t happen (the model is mathematically broken) and not looking at the things that actually could happen.

Then they do hard work: They either redesign the model and decision system to make the actual risks visible and manageable, or they decide the risk is too high and don’t deploy the model.

Most organizations don’t do this work. They validate the model thoroughly, deploy it, monitor it religiously, and assume risk is managed because the monitoring is good.

What this means for building better systems

If your organization is evaluating whether to deploy a model, the conversation should include this piece: “Here are the ways this model could produce bad outcomes that our framework might not catch. Here’s how we’re going to manage each one.”

If the answer is “our framework handles it,” push: How exactly? Can you describe the monitoring alert that catches it? Can you describe the decision that gets made when it happens?

If the answer is “we’ll handle it with human judgment,” push again: What training do those humans have? How do you standardize that judgment? How do you know it’s working?

If you can’t get a satisfying answer to either question, the model isn’t ready to deploy. Not because it’s mathematically broken, but because you don’t understand the risks it creates in your actual business context.

This is the conversation that separates organizations that treat model risk as an audit requirement from organizations that treat it as a serious business problem.

The ones that do this work rarely end up with model risk surprises. The ones that don’t usually end up in regulatory meetings explaining why their perfectly validated models produced regrettable outcomes.

Your compliance team is probably preparing for the wrong thing.

This isn’t their fault. Regulation of AI doesn’t follow the template of financial regulation or healthcare regulation or environmental regulation. Which means the preparation that worked for Dodd-Frank or HIPAA or the Clean Air Act won’t work here. And most compliance teams are building their AI readiness using the only template they have.

The result: Organizations are investing in comprehensive governance frameworks, audit procedures, and documentation systems that regulators probably won’t ask for in the ways they’re expecting.

What regulators actually care about

I spoke with someone in the policy space at a financial regulator recently. I asked: “What’s the first thing you’re going to look at when you start auditing banks’ AI systems?”

The answer wasn’t “show me your governance framework” or “give me your audit logs” or “where’s your risk register?”

It was: “Show me the decisions this AI system is making. Now show me three examples of bad outcomes. Why did the system make those decisions, and why didn’t anyone catch them?”

That’s not a compliance question. That’s a comprehension question. Regulators want to understand what’s actually happening inside your systems, and they know most organizations can’t explain it.

This matters because it reorders the priority stack. The first thing regulators will assess isn’t whether you have governance — it’s whether you understand your system well enough to explain its behavior. Governance documents are evidence of understanding, not a substitute for it.

The explainability gap

Here’s where compliance teams usually get stuck: They build governance frameworks around the assumption that they need to demonstrate control and oversight. So they create approval workflows, audit logs, monitoring procedures. All legitimate components of a control system.

But they often skip the foundational work: Can anyone at the organization actually trace why the system made a specific decision in a specific case?

I worked with a company that had a beautiful AI governance structure. Quarterly reviews, decision approval sign-offs, dedicated model risk managers. When a regulator’s hypothetical became real — a model recommendation that produced a clearly unfavorable outcome for a protected class — they tried to trace it. Why did the system score that customer that way?

They couldn’t. They had governance, but no explainability. They had documentation, but no understanding. The model owner had moved to a different team. The training data pipeline had been updated. The feature engineering logic was lost in Jupyter notebooks nobody maintained. They had all the structural elements of control and almost none of the functional elements.

A governance structure built on this foundation is actually a liability. It demonstrates that you have processes, which proves you had accountability, which makes the poor outcome look worse — not better.

What preparation actually looks like

Organizations that are genuinely prepared for regulation are usually doing a different set of work right now. Some of it looks like governance, but most of it looks like understanding:

1) Explainability work — Can you pull a specific decision and explain it in a way that makes sense to someone who isn’t your data scientist? Not “the model scored it,” but “the model weighted these factors, which led to this result, and here’s why those weightings made sense for this use case.”

This requires actually maintaining that explainability. Some teams do this through SHAP values or other interpretability frameworks. Others just maintain clear decision trees or decision rules. The method matters less than the commitment: You’re building to be explainable, not building and hoping explanation comes later.

2) Outcome monitoring — Not just “is the model accurate,” but “is the model making decisions that we expect?” Are certain segments systematically getting different treatment? Are the outcomes diverging from what you designed for? Are there decisions that look right mathematically but feel wrong operationally?

Most monitoring systems focus on statistical drift (has the model’s accuracy changed). Regulators care about behavioral drift (are the decisions changing in ways we didn’t expect). These aren’t the same thing.

3) Decision documentation — Not in the governance sense (we approved this decision), but in the explainability sense. Why did we choose this architecture? What assumptions did we make? What would falsify those assumptions? If a bad outcome occurs, what would tell us whether the system failed or our assumptions about the system were wrong?

This sounds like it might take forever. Organizations that have done it well usually structure it around decision memoranda. One page per major choice: the choice, the alternatives considered, why you chose what you chose, what would tell you if you were wrong.

4) Traceability of inputs — This is where most organizations are weakest. Where did the training data come from? Has it been modified? Who had access to it? If an outcome looks discriminatory, can you reconstruct what data the model learned from?

This is partly a governance question, but it’s mostly a technical question. It requires data versioning, provenance tracking, and lineage documentation. It requires that somewhere in your system, you can reconstruct the training dataset from six months ago.

Why this matters more than compliance theater

The reason I’m focusing on this isn’t because regulators need explainability — though they do. It’s because having it actually makes you better at building AI systems.

Organizations that maintain explainability throughout development make better decisions about model architecture. They can’t just accept “the model says so.” They have to be able to defend the decision. This raises the bar.

Organizations that truly monitor behavioral drift catch problems earlier. They’re not looking at accuracy degradation in quarterly reports. They’re asking constantly: Are we still making the decisions we intended to make?

Organizations that maintain decision documentation learn faster. When something goes wrong, you can distinguish between “the system worked as designed but the design was wrong” and “the system didn’t work as designed.” Most post-mortems can’t make this distinction. That’s expensive.

Organizations that track data provenance get better at understanding their own bias. You can’t fix something you can’t trace. Once you know where your training data came from, which decisions it was used for, and which populations it was meant to represent, you can actually audit for bias.

Where to start

If your compliance team is currently building a governance framework without this foundational work, it’s not too late to reorder. Start here:

Can you pick a model that’s in production right now and explain in plain language why it made a specific decision in a specific case? If you can’t do that, you don’t have compliance readiness — you have documentation.

Once you can, build governance systems around the assumption that you can always be asked to explain. This means architecture that’s inherently explainable, data that’s inherently traceable, and decisions that are inherently defensible.

The governance will follow from there. And when regulators do arrive, they’ll be asking about the same things you’ve already learned to explain.