The demo you’re watching was built, tested and rehearsed. It has its own backlog and its own maintainer. The person delivering it has given it more times than you have bought enterprise software in your career.

You are evaluating that artifact. You are not, yet, evaluating the product.

When every demo is excellent

Composite from a few casework evaluations: a public sector agency, permit and licensing, several hundred caseworkers, a document-heavy assessment process with a long backlog.

Three vendors demoed. All three were excellent. Genuinely, not sarcastically: polished, responsive, well-matched to the described problem. The evaluation team came out of three sessions with three sets of enthusiastic notes and no way to tell the products apart.

So they decided on price, which felt disciplined. Eleven months later the product was struggling with the agency’s actual document mix, a large share of which arrived as scans of forms that had been filled in by hand and then photocopied. None of the three demos had included a scan of anything.

Nobody misled anyone. The agency had never asked, and no vendor volunteers the input type their product handles worst.

What a demo selects for

A demo isn’t deceptive. It’s selective, and the selection happens along exactly the three axes that decide whether the thing works for you.

The path. It follows the route through the product where the product is strongest. Every product has one. Yours will not be it.

The data. Demo data is clean, representative, and often assembled specifically because the product handles it well. Your data has a decade of exceptions in it and four fields that mean different things depending on which system wrote them.

The scale. One user, a handful of documents, no concurrency. Latency at your volume is a different product experience and it never appears in a demo.

Underneath those sits the thing that matters most and is hardest to see: what it took to make it look like that. Setup effort is invisible in a demo by construction, and setup effort is most of your integration budget.

Three requests that break it

Three, not ten. A long list turns the session adversarial and you’ll get defensive answers to all of them. These three are enough, and you can make all three sound like curiosity rather than cross-examination, because that’s what they are.

One. Run it on our data, unprepared, now.

Not a curated sample sent two weeks ahead. A file pulled from your system that morning, exceptions included, in whatever state it’s actually in. Bring three.

You’re not testing accuracy. You’re testing what happens on input the vendor didn’t get to shape. If the honest answer is that this needs a preparation step first, you’ve just found part of your integration cost, in week three rather than month five. That’s a good outcome from the request, not a failed one.

Two. Show me it being wrong.

Ask them to produce a case where their system fails, and then to walk through how the user finds out.

This is the single most informative question in an evaluation. A vendor who has a failure case ready, describes it precisely, and can explain the detection path has thought hard about operating in the real world. A vendor who deflects has either not characterized their failure modes or has decided not to discuss them, and you want to know which before you’re a customer rather than after.

The follow-up matters as much: how does the user know. A system that’s wrong loudly is manageable. A system that’s wrong quietly, in a document somebody signs, is a different risk profile entirely and belongs in your business case rather than in a footnote.

Three. Who built this demo, and are they on our implementation?

Nearly always the answer is a solution engineer who is not. That’s not a scandal, it’s how the industry is staffed, and treating it as a gotcha wastes the question.

The value is in what comes next. Who does implement, how many of those people exist, how many concurrent implementations they’re running, and whether it’s the vendor’s own team or a partner. You’ll learn more about your delivery risk in that four-minute answer than in the entire capability section of the RFP response.

Who should be watching

Put two people in the room who will personally use the thing, and tell them beforehand that being difficult is their job today.

They will ask things you wouldn’t. A caseworker watching a demo of casework software notices in about ninety seconds that the screen assumes one applicant per file, and says so, while the evaluation team is still admiring the summarization. That observation is worth the entire session.

The reason this doesn’t happen by default is that demos get scheduled as executive sessions, and adding practitioners feels like it dilutes the audience. It’s the reverse. The executives can read the summary.

One thing to do differently

Ask for the failure case in the first demo rather than the third.

It changes the relationship immediately, in a way that’s mostly good. It signals you’re a buyer who’ll be dealing with the product in production, not a buyer being impressed. And it sorts your shortlist within ten minutes, because the vendors who’ve thought seriously about being wrong answer it well and are visibly pleased to be asked.