Every financial services firm has a Model Risk Management framework. Most of them haven’t actually used it to catch anything.

This isn’t because the frameworks are poorly designed. It’s because they’re designed for a problem that doesn’t quite exist, deployed against risks that are invisible to the framework.

I’ll explain, because this is one of those areas where understanding the gap tells you a lot about how to think about AI risk more broadly.

What model risk frameworks are built to catch

The standard Model Risk Management framework came from Basel guidance and financial regulator requirements. It evolved around a specific category of problems: When you build a quantitative model that feeds decision-making, and the model is wrong in a systematic way, you make bad decisions at scale.

So the framework focuses on: Is the model valid? Is it accurate? Does it perform as expected? When something goes wrong, what’s the model’s role in the failure?

This is useful for catching models that are mathematically broken, statistically invalid, or performing worse than expected. It catches overfitting, concept drift, and data quality issues.

It does not catch most of the risks that actually matter.

What’s invisible to standard frameworks

Here’s a pattern I see repeatedly: An organization implements a model, validates it thoroughly, puts it into production, and monitors it diligently. The model performs exactly as expected. Accuracy is stable, validation metrics are healthy, everything looks fine.

Then something goes wrong. The model wasn’t wrong — the decision-making context around the model was wrong.

A lending model that accurately predicts default risk produces a disparate impact on protected classes. The model is statistically sound. It’s predicting correctly. But the business decision to use it for loan decisions in an underserved market was wrong because the training data came from a time when lending practices were different in that market.

A trading model that accurately captures market correlations suddenly produces an unexpected loss because correlation structure changed, and nobody had decided what to do if that happened. The model did what it was trained to do. The decision authority around how to handle model uncertainty had never been established.

A workforce optimization model that accurately predicts which positions will have retention problems is deployed to recommend layoffs, but nobody had decided whether a model designed to predict problems should drive who gets laid off. The model worked. The application was wrong.

In each case, the model risk framework shows green. The validation is sound. The monitoring is clean. The problem isn’t in the model — it’s in the decision system around the model.

Why frameworks miss this

Standard model risk frameworks are built to answer the question: “Is this model trustworthy as a mathematical instrument?”

The question that actually matters for most enterprise AI risk is different: “Is the decision we’re making with this model appropriate?”

These are correlated but not identical. You can have a trustworthy model used for an untrustworthy decision. And you can have a somewhat noisy model used thoughtfully where the human judgment around it is sophisticated enough to manage the risk.

Most frameworks check the first box and assume the second one is handled elsewhere. Spoiler: It’s usually not handled at all.

Organizations that handle model risk seriously usually do a second set of work that sits beside the traditional framework. It asks different questions:

What is this model actually being used for? Not in theory, but in practice. How is the decision actually being made? Who’s being affected? What would a bad outcome look like?

Why are we using a model for this decision at all? Sometimes the answer is “because it’s more accurate than human judgment.” Sometimes it’s “because it’s faster.” Sometimes it’s “because it removes the appearance of bias even though the input data might be biased.” These matter, because they change what your risk actually is.

What’s the decision being made if the model is wrong? If the model’s confidence is low, do you have a process for human override? If the model fails in a specific way, what’s the next decision? If you can’t describe what you do when the model is wrong, you don’t understand your own risk.

What would falsify our assumptions? You’re assuming the training data is representative of the population you’re applying the model to. What would prove that wrong? You’re assuming the decision context is stable. What would change it? You’re assuming the model’s accuracy is stable. How would you know if it wasn’t? If you can’t describe the conditions that would break your model’s usefulness, you’re building on faith, not on reasoning.

Is there a human in the loop who could reasonably override the model? Not should there be — could there be? If the model recommends something and a human thinks it’s wrong, can they say no? If not, you’ve built a system where the model is actually making the decision, and you should be honest about that. If yes, you need a process for when humans should override and what you learn from overrides.

The uncomfortable conversation

Organizations that separate themselves from the crowd usually do something harder than updating their framework. They have what I’d call the “uncomfortability conversation.”

Someone sits down and lists all the ways the model could produce a bad outcome that isn’t captured in the model risk framework:

• Accurate prediction of biased historical patterns (model correct, decision wrong)
• High accuracy but low coverage (model works for some populations, not others)
• Decision context change (model was trained in one business context, deployed in another)
• Correlation breakdown (model captures historical correlations that don’t predict future)
• Population drift (model trained on one population, applied to another with different characteristics)

For each, they ask: Do we have a process to catch this? Is this monitored? Would someone know if this happened?

Usually the answer is no. Usually the model risk framework is looking at the one thing that won’t happen (the model is mathematically broken) and not looking at the things that actually could happen.

Then they do hard work: They either redesign the model and decision system to make the actual risks visible and manageable, or they decide the risk is too high and don’t deploy the model.

Most organizations don’t do this work. They validate the model thoroughly, deploy it, monitor it religiously, and assume risk is managed because the monitoring is good.

What this means for building better systems

If your organization is evaluating whether to deploy a model, the conversation should include this piece: “Here are the ways this model could produce bad outcomes that our framework might not catch. Here’s how we’re going to manage each one.”

If the answer is “our framework handles it,” push: How exactly? Can you describe the monitoring alert that catches it? Can you describe the decision that gets made when it happens?

If the answer is “we’ll handle it with human judgment,” push again: What training do those humans have? How do you standardize that judgment? How do you know it’s working?

If you can’t get a satisfying answer to either question, the model isn’t ready to deploy. Not because it’s mathematically broken, but because you don’t understand the risks it creates in your actual business context.

This is the conversation that separates organizations that treat model risk as an audit requirement from organizations that treat it as a serious business problem.

The ones that do this work rarely end up with model risk surprises. The ones that don’t usually end up in regulatory meetings explaining why their perfectly validated models produced regrettable outcomes.