I was in a meeting with a general counsel last month. The company was deploying a large language model as an internal knowledge assistant. It was a reasonable project. The team had thought through data security. They’d built audit trails. They’d set usage policies.

The GC asked a question that stopped the room: “What happens if the model hallucinates and gives someone bad advice they act on?”

Not a weird question. A straightforward question. And it turned out nobody had a clear answer.

The conversation that followed was instructive. The data science team said: well, it’s a language model, hallucinations are a known limitation. The product team said: we have a disclaimer. The legal team said: disclaimers don’t protect us from liability.

And that’s where it usually breaks down.

Many organizations understand that AI systems can fail. What many are still developing their understanding of is the structure of liability when they do.

Here’s what I think you need before any AI system goes live in your organization:

A hallucination liability assessment.

This isn’t model documentation. This isn’t a risk register. This is a specific document that your legal team writes, not your data science team, that answers one question: if this system confidently says something that’s false, and someone acts on it, what is our legal exposure?

That’s the starting point. Because hallucination—when a language model generates confident falsehoods—is a specific kind of failure mode that most organizations haven’t built legal thinking around yet.

For decades, organizations have built legal frameworks for system failures: databases go down, processes fail, data gets lost. These are understood. There are precedents. There are insurance products.

But a system that confidently tells someone something false is a newer problem. And the legal structures around it are still settling.

Here’s why it matters:

If your system is purely informational (you’re using it to summarize documents internally), the risk profile is one thing. Someone reads the summary, notices it’s wrong, corrects it. Inconvenient. Not necessarily a legal problem.

But if your system is integrated into a workflow where a human is supposed to verify the output, but the human reasonably relies on the system’s apparent confidence, you have a different risk structure.

And if your system is trained on your proprietary knowledge and someone acts on its hallucinatory output in a way that damages a customer relationship, a business deal, or a compliance outcome, you have another risk structure entirely.

These aren’t hypothetical. I’ve watched organizations have exactly these moments. “The model told our sales team that we offered a service we don’t actually provide.” “The system generated a draft contract clause that contradicted our terms but the team didn’t catch it.” “The model synthesized customer data in a way that suggested we had capabilities we don’t have.”

None of these broke the system. All of them created liability.

So before you deploy, you need legal to answer:

First, context: What decisions or actions does this system enable or inform? What’s the chain from system output to human decision to real-world consequence?

Second, confidence: When this system is wrong, is it obviously wrong? Or does it confidently present falsehoods? (This matters because overconfident falsehoods create liability that uncertain outputs don’t.)

Third, verification: In the workflow, is there a verification step? And critically, is that step designed to catch hallucinations, or just to audit that the system ran?

Fourth, scope: Who can see the output? Is it constrained (internal team) or broad (customers, partners)? Scope multiplies liability.

Fifth, exposure: If the system hallucinates and someone acts on it, what’s the worst-case consequence? Client relationship? Compliance violation? Wrong regulatory filing? Dollar impact? This determines what you need to mitigate.

Once you’ve answered these, you’re in a position to actually assess whether you should deploy this system as-is, or whether you need to change something about how it’s used.

Maybe you need human verification at a specific step. Maybe you need to constrain who can access the output. Maybe you need specific disclaimers that are actually meaningful. Maybe you need to train people on the limitations. Maybe you need to not deploy it the way you initially planned.

Most organizations don’t do this. They get to deployment and they ask: does it work? And if it works, they deploy it. Then they’re surprised when the liability question comes up in month four because someone acted on something the system hallucinated.

Legal teams are increasingly asking for this, but they’re often asking late. The system is built. The deployment is planned. Now they’re trying to add guardrails to something that was designed without legal thinking.

That’s expensive and it’s slow. Better to do it upfront.

Here’s what I recommend: before you move a large language model (or any AI system with significant hallucination risk) from pilot to production, have your legal team write a one-page assessment. Not a binder of documentation. Not a risk matrix. One page. What’s the hallucination exposure? What are the paths from model output to liability? What mitigations do we need?

Then, once you have that clarity, the engineering team can decide: do we change the workflow? Do we add verification? Do we constrain access? Do we add specific training? Do we add disclaimers that actually mean something?

The companies that are handling this well aren’t the ones with perfect models. They’re the ones that are clear about the risk, and clear about what they’ve done to manage it.

And that clarity starts with a document that your legal team owns, not your data science team.

Get that document written before you go live. It will change how you deploy. And it might prevent you from being the organization that learns the liability lesson the hard way.