Model risk management in enterprise AI isn’t a solved problem. It’s barely understood.

Most organizations have a model risk framework inherited from financial services—approaches designed for regression models that predict loan default or market behavior. Those frameworks assumed you could hold the model still and understand it: extract coefficients, measure bias, validate performance, sign off, and deploy. The model’s behavior was relatively predictable within its operational envelope.

That world no longer exists.

The model risk governance you have today was built for a different problem. Foundation models introduce different considerations that challenge these assumptions. The model risk governance you’ll need in three years needs to be built now, because the complexity is arriving faster than most organizations realize.

What’s Actually Changing

The core shift is loss of transparency at scale. Your predecessors in model risk could, in principle, understand how their model worked. Someone—a data scientist, a mathematician—could crack it open and explain the causal path from input to output. This was tedious and expensive, but possible.

With foundation models and large language models, that option no longer exists at all. You’re deploying systems whose internal behavior you cannot fully explain, even if you built them. You’re introducing systems whose behavior changes with deployment context—different users, different use cases, different data—in ways that are difficult to predict and harder to measure consistently.

You’re also stacking models. An LLM calling an API that reads a database, transforms the data, feeds it back to another model. The risk surface isn’t one model anymore—it’s a web of interconnected systems, each with its own risk profile, each potentially amplifying or mitigating the others’ failures.

That’s not model risk. That’s system risk. And your governance framework is built for the wrong problem.

Three Dimensions of Complexity That Are Coming

Behavioral Drift: Your model performs differently in production than it did in test. This is a known problem. But with foundation models, behavioral drift becomes harder to detect and easier to miss. You’re monitoring for deviation from expected performance, but what’s “expected” when your model’s output distribution is continuously shifting based on user patterns, retrieval data, and context? The frameworks that detect drift in traditional ML break down when the thing drifting is too large to hold in your explanation budget.

Emergent Capabilities: You deploy a model for Task A. In production, users discover it’s useful for Task B, Task C, and Task D—sometimes better than your original use case. Those emergent capabilities were latent in the model. You didn’t design them. You didn’t test them. You don’t fully control when or how they appear. But you’re now accountable for them. Your risk governance needs to account for capabilities that emerge after deployment, not just the ones you architected.

Compounding Amplification: When you stack systems, failures don’t just accumulate—they amplify. A hallucination in one model becomes input to another, which makes a high-confidence wrong decision based on false data. Your risk governance for individual models doesn’t capture this compounding effect. You need frameworks that reason about system-level failure modes, not just model-level risk.

What’s Still Missing in Most Frameworks

The model risk programs I see in mature enterprises usually have three components: testing and validation, performance monitoring, and escalation procedures. They’re measuring the right things within their scope. But their scope is too narrow.

What’s missing is behavioral monitoring in context. You need to observe how your model behaves at the boundaries of its training data, with adversarial input, with out-of-distribution user populations. Not because you’re paranoid, but because that’s where real risk lives—at the edges, where assumptions break down.

What’s missing is feedback loop integration. Your production monitoring tells you the model is drifting. Your risk governance then escalates to data science for investigation. But the governance framework doesn’t close the loop: Did we fix it? Did we understand why it drifted? Did we update our assumptions about what environments this model can operate safely in? Without closed loops, you’re not learning—you’re just watching failures.

What’s missing is stack-level risk assessment. Your model is accurate. Your retrieval system is reliable. Your API is stable. But the combination of all three, deployed in production with human decision-makers relying on the output, has a different risk profile. You need frameworks that reason about system-level risk, not just component risk.

Building It Now

If you’re at a large enterprise with significant AI deployment, here’s what I’d prioritize starting today—not in three years:

First: Expand your definition of “model” from the algorithms to the system. Your governance framework should reason about the entire decision path: data ingestion, model inference, output formatting, integration with business systems, and human decision-making. A failure anywhere in that stack is your problem.

Second: Shift from static testing to continuous learning about model behavior. Deploy observability systems that understand model behavior across user populations, use cases, and contexts. Not just whether the model is accurate on average, but where it fails systematically. Where do certain user populations get different outputs than others? Where is your confidence high but accuracy is low?

Third: Separate risk tolerance by use case. A model that informs a business decision has different risk tolerance than a model that makes a business decision. A model that affects one customer has different requirements than a model that affects millions. Your governance framework should be granular enough to reason about these differences.

Fourth: Build escalation and learning into your governance loops. When something breaks, your escalation path should lead to understanding and prevention, not just remediation. If a model drifts, you should understand why and update your assumptions about safe operating conditions. That closed loop is where real governance lives.

Why This Matters

Model risk governance isn’t a compliance checkbox. It’s a decision-quality problem. If you don’t understand the risks your models are creating, you’re flying blind. And flying blind at enterprise scale—where decisions affect thousands of customers, millions of transactions, or critical business operations—is a risk your board should be skeptical of.

The organizations that get ahead of this are the ones that start now, while they still have time to build governance that actually works for their systems, not governance inherited from an older model-risk era.

The complexity is coming whether you’re ready or not. The question is whether you’re going to understand it when it arrives.