Your compliance team is probably preparing for the wrong thing.

This isn’t their fault. Regulation of AI doesn’t follow the template of financial regulation or healthcare regulation or environmental regulation. Which means the preparation that worked for Dodd-Frank or HIPAA or the Clean Air Act won’t work here. And most compliance teams are building their AI readiness using the only template they have.

The result: Organizations are investing in comprehensive governance frameworks, audit procedures, and documentation systems that regulators probably won’t ask for in the ways they’re expecting.

What regulators actually care about

I spoke with someone in the policy space at a financial regulator recently. I asked: “What’s the first thing you’re going to look at when you start auditing banks’ AI systems?”

The answer wasn’t “show me your governance framework” or “give me your audit logs” or “where’s your risk register?”

It was: “Show me the decisions this AI system is making. Now show me three examples of bad outcomes. Why did the system make those decisions, and why didn’t anyone catch them?”

That’s not a compliance question. That’s a comprehension question. Regulators want to understand what’s actually happening inside your systems, and they know most organizations can’t explain it.

This matters because it reorders the priority stack. The first thing regulators will assess isn’t whether you have governance — it’s whether you understand your system well enough to explain its behavior. Governance documents are evidence of understanding, not a substitute for it.

The explainability gap

Here’s where compliance teams usually get stuck: They build governance frameworks around the assumption that they need to demonstrate control and oversight. So they create approval workflows, audit logs, monitoring procedures. All legitimate components of a control system.

But they often skip the foundational work: Can anyone at the organization actually trace why the system made a specific decision in a specific case?

I worked with a company that had a beautiful AI governance structure. Quarterly reviews, decision approval sign-offs, dedicated model risk managers. When a regulator’s hypothetical became real — a model recommendation that produced a clearly unfavorable outcome for a protected class — they tried to trace it. Why did the system score that customer that way?

They couldn’t. They had governance, but no explainability. They had documentation, but no understanding. The model owner had moved to a different team. The training data pipeline had been updated. The feature engineering logic was lost in Jupyter notebooks nobody maintained. They had all the structural elements of control and almost none of the functional elements.

A governance structure built on this foundation is actually a liability. It demonstrates that you have processes, which proves you had accountability, which makes the poor outcome look worse — not better.

What preparation actually looks like

Organizations that are genuinely prepared for regulation are usually doing a different set of work right now. Some of it looks like governance, but most of it looks like understanding:

1) Explainability work — Can you pull a specific decision and explain it in a way that makes sense to someone who isn’t your data scientist? Not “the model scored it,” but “the model weighted these factors, which led to this result, and here’s why those weightings made sense for this use case.”

This requires actually maintaining that explainability. Some teams do this through SHAP values or other interpretability frameworks. Others just maintain clear decision trees or decision rules. The method matters less than the commitment: You’re building to be explainable, not building and hoping explanation comes later.

2) Outcome monitoring — Not just “is the model accurate,” but “is the model making decisions that we expect?” Are certain segments systematically getting different treatment? Are the outcomes diverging from what you designed for? Are there decisions that look right mathematically but feel wrong operationally?

Most monitoring systems focus on statistical drift (has the model’s accuracy changed). Regulators care about behavioral drift (are the decisions changing in ways we didn’t expect). These aren’t the same thing.

3) Decision documentation — Not in the governance sense (we approved this decision), but in the explainability sense. Why did we choose this architecture? What assumptions did we make? What would falsify those assumptions? If a bad outcome occurs, what would tell us whether the system failed or our assumptions about the system were wrong?

This sounds like it might take forever. Organizations that have done it well usually structure it around decision memoranda. One page per major choice: the choice, the alternatives considered, why you chose what you chose, what would tell you if you were wrong.

4) Traceability of inputs — This is where most organizations are weakest. Where did the training data come from? Has it been modified? Who had access to it? If an outcome looks discriminatory, can you reconstruct what data the model learned from?

This is partly a governance question, but it’s mostly a technical question. It requires data versioning, provenance tracking, and lineage documentation. It requires that somewhere in your system, you can reconstruct the training dataset from six months ago.

Why this matters more than compliance theater

The reason I’m focusing on this isn’t because regulators need explainability — though they do. It’s because having it actually makes you better at building AI systems.

Organizations that maintain explainability throughout development make better decisions about model architecture. They can’t just accept “the model says so.” They have to be able to defend the decision. This raises the bar.

Organizations that truly monitor behavioral drift catch problems earlier. They’re not looking at accuracy degradation in quarterly reports. They’re asking constantly: Are we still making the decisions we intended to make?

Organizations that maintain decision documentation learn faster. When something goes wrong, you can distinguish between “the system worked as designed but the design was wrong” and “the system didn’t work as designed.” Most post-mortems can’t make this distinction. That’s expensive.

Organizations that track data provenance get better at understanding their own bias. You can’t fix something you can’t trace. Once you know where your training data came from, which decisions it was used for, and which populations it was meant to represent, you can actually audit for bias.

Where to start

If your compliance team is currently building a governance framework without this foundational work, it’s not too late to reorder. Start here:

Can you pick a model that’s in production right now and explain in plain language why it made a specific decision in a specific case? If you can’t do that, you don’t have compliance readiness — you have documentation.

Once you can, build governance systems around the assumption that you can always be asked to explain. This means architecture that’s inherently explainable, data that’s inherently traceable, and decisions that are inherently defensible.

The governance will follow from there. And when regulators do arrive, they’ll be asking about the same things you’ve already learned to explain.