A financial services company’s recommendation engine went down at 2 AM. Not catastrophically—just an inference timeout that cascaded through their system. They had monitoring. They caught it within 10 minutes. Then they discovered they had no fallback strategy.

What should happen when the recommendation engine is down? Should the application show generic recommendations? Show yesterday’s recommendations? Hide recommendations entirely? The team had never discussed this. They had built a system that could fail but had no plan for what happens when it does.

It took them 45 minutes to decide, implement a fallback, and redeploy. For 35 minutes, their application was either broken or providing a degraded experience, and nobody had made an explicit choice about which.

This happens in most enterprises with AI systems. You plan for the system working. You don’t plan for the system not working. And when it fails—and it will—you discover that operational continuity was never actually planned.

What Fails First

Before thinking about fallback, it’s useful to understand what typically fails first in AI systems.

It’s rarely the model itself. Models are usually pretty robust once they’re in production. What fails is the infrastructure around the model: – The model serving platform goes down – The data pipeline that feeds the model stops producing fresh data – The API that returns model results experiences latency spikes – The inference process uses more resources than expected, causing memory or compute issues – The downstream system that consumes model output stops accepting it (usually because the format changed or the endpoint changed)

These are infrastructure failures, not model failures. But the effect is the same: your AI system is no longer functional. The application calling it still needs a response. What do you do?

What Most Enterprises Don’t Plan For

The typical enterprise approach is: 1. Build the AI system 2. Deploy it 3. Hope it doesn’t fail 4. When it fails, decide what to do in real time

This usually results in one of: – Cascading failure — The system fails, the application fails, everything stops working – Silent failure — The system fails but returns a null or default response that the application can’t distinguish from a real prediction – Slow failure — The system times out after waiting for a response for too long – Inconsistent failure — Different parts of the system fail in different ways, and different applications handle it differently

None of these are acceptable for systems that affect customers, operations, or compliance.

What Actually Needs to Happen

A system that can fail safely needs three components:

First: Detection. You need to know when the system is down or degraded. Not “the model didn’t return a prediction” (which could be normal if the model decided it wasn’t confident enough). But “the system is not functioning according to its SLA.”

This requires explicit health checks and monitoring. Not just “is the service running?” but “is the service returning predictions within [time window] with [confidence level]?” You need to define what healthy looks like and actively monitor for it.

Second: A fallback strategy. What should the application do when the AI system is not available? This is a business decision, not a technical one. Do you: – Return yesterday’s recommendations? – Return a generic/default response? – Return nothing and let the user experience a degraded interface? – Route to a human operator?

Different business contexts call for different fallbacks. A recommendation engine might fall back to “most popular” content. A fraud detection system might fall back to “hold the transaction pending manual review.” A pricing system might fall back to “yesterday’s prices” or “default pricing.”

The point is: someone needs to have made a conscious decision, not made this up at 2 AM.

Third: Activation and rollback. The fallback isn’t a plan document. It’s a runbook with explicit steps: – Who needs to be notified? – What command activates the fallback? – How do you monitor to know when the AI system is healthy again? – What does rollback from fallback look like? – How long are you willing to run on fallback?

This should be practiced. Not just documented. Teams should have run through the fallback procedure in a test environment and know that it works.

Why Enterprises Don’t Plan This

There are a few reasons fallback planning gets skipped:

First, it’s depressing. When you’re excited about deploying a new AI system, planning for its failure feels like pessimism. It’s more fun to plan for success.

Second, it’s not technically challenging. There’s no architecture question, no machine learning problem. It’s operational grunt work: write a runbook, test it, update it.

Third, there’s often a misunderstanding that if you build good monitoring, you don’t need fallback. The logic is: “if we monitor well, we’ll catch problems so fast that it won’t matter.” This is wrong. Good monitoring means you catch the problem in 2 minutes instead of 20. But if there’s no fallback, those 2 minutes are still 2 minutes of broken service.

How to Actually Do This

Start before you deploy. For every new AI system, someone should answer: – Failure modes: What are the most likely ways this system could fail? – Fallback strategy: For each failure mode, what should the system do? – Detection: What does healthy look like? What signals tell us it’s not healthy? – Runbook: Explicit steps for activating fallback, rolling back from fallback, and resuming normal operation.

Make the fallback strategy visible. Not a decision made in a design meeting and forgotten. Put it in the deployment documentation. Put it in the team’s operational handbook. Make it something the team reviews regularly (at least quarterly) and updates when the business context changes.

Practice it. Run a fire drill where you simulate the system being down and walk through the fallback activation. You’ll discover what you didn’t understand about how the fallback works.

Plan for degraded operation, not just failure. The most common scenario isn’t “the system is completely down.” It’s “the system is slow” or “the system is returning low-confidence predictions.” For these scenarios, you might not need a full fallback—you might just need to degrade performance (return fewer recommendations, adjust the scoring threshold, etc.) while keeping the system running.

The Practical Implication

The enterprises that handle system failures well aren’t the ones with the most advanced monitoring. They’re the ones that planned for failure before it happened. They have explicit fallback strategies. They’ve practiced them. They know how long they’re willing to run on fallback and what triggers going back to normal operation.

When their system fails, they don’t panic. They activate the documented fallback. The application keeps working, even if in a degraded way. They have time to fix the root cause without pressure.

Most enterprises treat fallback as optional. That works fine until the system fails. Then fallback becomes essential, and you’re making it up at 2 AM while users are waiting for the system to work again.

The cost of planning fallback is low—a few hours of thoughtful operational work before deployment. The cost of not planning it is high: a cascading failure, a degraded user experience, and decisions made in panic instead of policy.