A reasonable question to ask of any AI system in production: “What data did this model train on?”

Most large organizations cannot answer that question with specificity.

This is a bigger problem than it sounds. It’s not just an audit inconvenience. It’s a blocker to understanding your own model’s behavior, managing its risks, and defending its decisions.

Why this matters more than most organizations realize

Imagine a scenario that happens more often than anyone discusses publicly: Your lending model makes a recommendation that looks wrong for a specific demographic segment. The recommendation is mathematically sound. The model is performing as expected overall. But in this segment, something is off.

Your first question should be: What data did the model train on for this segment?

If you know the answer — “We trained on 47,000 approved loans in this geography from 2018-2020, and here’s the selection bias in that dataset” — you can investigate. Maybe the selection bias is the problem. Maybe the economic conditions of that time period are no longer representative. Maybe lending practices have changed.

If you don’t know the answer, you’re stuck. You can look at accuracy metrics (which look fine) and feature importance (which shows expected patterns) and conclude nothing is wrong. But something feels wrong. You just can’t diagnose it because you don’t know where the model’s learned patterns came from.

This happens more than you’d think. And when it does, the organization usually takes one of three paths:

Path 1: They remove the feature that seems problematic, retrain the model, and hope the problem goes away. They’ve changed the decision-making process without understanding why.

Path 2: They add a rule (don’t recommend this for this segment) that overrides the model when it hits the problematic pattern. They’ve built a patchwork system that’s harder to maintain and monitor.

Path 3: They accept the problem and let the model make the recommendation anyway, documenting that they’re aware of the concern. They’re betting that enough models get questioned that auditors won’t focus on this one.

None of these are good outcomes. All three are symptoms of not understanding your training data.

Where data provenance usually fails

Organizations typically get tripped up in three places:

1) Data collection and compilation

You’re training on multiple datasets. You’re using internal transaction history, third-party enrichment data, and purchased datasets. These come together in a data pipeline. Months or years later, someone asks: “What exactly went into the training set?”

The answer usually involves hunting through old notebooks, asking data engineers who’ve moved to other teams, or looking at Hadoop jobs that have been rerun dozens of times with different parameters. What you want to know is clear and simple. What you can actually discover is murky and partial.

Most organizations solve this by documenting it after the fact, which is fine until the documentation is incomplete or wrong. Better organizations build data versioning into the pipeline from the start. They tag versions, store lineage information, and treat training data as an artifact that can be retrieved.

2) Data transformation

The raw datasets get processed. They’re cleaned, features are engineered, outliers are handled. This transformation process is usually in code. That code evolves. Parameters change. One version of the code created the training data. A later version handles new data differently. Now you’ve got a production system where data is processed one way, and historical data was processed another way.

The question becomes: Is the model failing because the environment changed, or because we changed how we process the input?

This is fixable but requires discipline. You need to version the feature engineering code just like you version the model. You need to be able to say “this model trained on data processed by version 2.3 of the ETL pipeline.” Then when something goes wrong, you can reproduce that exact data processing.

3) Consent and licensing

More problematic: You’ve assembled training data from multiple sources, and some of those sources have licensing restrictions or data usage agreements. You trained the model with data you shouldn’t have had access to, or you used data in a way that violated the agreement.

This is harder to discover after the fact. The model is in production. The historical data is mixed with other data. You can’t untrain what you learned. You either acknowledge the problem (legal exposure), or you hope nobody notices.

The organizations that avoid this usually have data governance at the point of procurement. Before data comes into the pipeline, they know: Where did it come from? What can we use it for? Are there restrictions?

What working data provenance actually looks like

These are emerging industry best practices. Organizations that handle this well usually have systems that let them:

Answer the basic questions

“What data trained this model?” You should be able to specify a dataset version or date range and get back a manifest of what went into training.

“Has that data been modified?” You should be able to tell whether the data was processed differently at training time vs. now. You should be able to reproduce the historical processing if you need to.

“What are the licensing and consent restrictions?” You should know which parts of the training data have restrictions and what those restrictions are.

These aren’t complicated questions. They’re just hard to answer in systems that weren’t built with provenance in mind.

Link provenance to model behavior

“We know this model trained on 2018-2020 data from this geography. Here’s what economic conditions and lending practices looked like then. Here’s what they look like now. Here’s where we’d expect the model to diverge.”

This conversation requires you to know the training data. Then you can make an argument: “The model learned patterns from this time period and this context. If the context has changed, we should expect divergence.”

Establish audit trails

If data provenance is clear, auditing becomes more straightforward. You can trace decisions back to training data. If something looks wrong in a cohort, you can investigate whether that’s a modeling issue or a data bias issue.

Regulators are increasingly asking for this. “Your model recommended X for this segment. Tell us about the training data for that segment and why you think the recommendation is fair.” If you can’t answer, that’s a problem.

How to start fixing this

If you have models in production and you can’t confidently answer “what data trained this model,” you’re not alone. This is an endemic problem in large organizations.

Start with the highest-impact models. For each one, invest in understanding the training data:

Document it. Write down what datasets went in, where they came from, what processing happened. Even if it’s incomplete, it’s better than nothing.

Version it. From this point forward, treat training data and feature engineering as versioned artifacts. Tag them. Store them. Know what version produced what model.

Connect it to decisions. When something goes wrong, ask whether the decision-making divergence is because the model failed or because the environment changed. You can’t answer that without knowing the training data.

Link it to governance. Your model risk management should include understanding the training data and its limitations. “The model is statistically valid” isn’t enough if you don’t understand what it learned.

Data provenance takes effort. But it’s the foundation for understanding your own models and managing their risks. Organizations that have built it out are the ones that can confidently explain their model decisions. The ones that haven’t are the ones that eventually end up in uncomfortable regulatory meetings asking the question they can’t answer: “What was your model actually trained on?”