You will be given three references. All three are current customers, all three were asked whether they’d take the call before their names reached you, and all three have been told roughly what you’re evaluating.

That process is designed to produce a positive signal and it works. The interesting question is what you can get out of it anyway, and where the rest of your sample comes from.

Three happy customers, one region

Composite from a few infrastructure-heavy evaluations: a telecommunications operator, network operations and field dispatch, a workforce spread across three countries and several timezones.

They took the three references offered. All three were positive and none were dishonest. The buyer signed.

In month four they discovered that the vendor’s support model had real coverage in one region and a follow-the-sun arrangement everywhere else that mostly meant a ticket sat overnight. Their night shift was the shift that most needed it.

All three references had been in the well-covered region. Not chosen for that reason, as far as anyone could tell. That’s simply where the vendor’s happiest customers were, which is the same thing as saying that’s where their support was good.

The reference sample and the quality signal were the same variable, and nobody noticed because nobody asked what the three had in common.

Questions with factual answers

Do not ask a reference whether they’re happy. They were selected for being happy and they’ll confirm it accurately.

Ask questions with factual answers instead, ideally ones involving dates and artifacts, because those are hard to soften without lying and almost nobody wants to lie to a stranger on behalf of a vendor.

• How long from signature to your first real user? A date minus a date. If it’s four times the vendor’s stated implementation time, that’s your number, not theirs.
• What surprised you in month six? Month six is after the honeymoon and before the renewal, which makes it the most honest window in the relationship.
• What did you end up building that you thought you were buying? Every implementation has at least one. The answer is your integration estimate.
• Who on your team would have voted against this, and what was their argument? This gives a satisfied customer permission to voice the internal objection without owning it.
• If you were signing again, what would you put in the contract?

That last question is the best in the set and I’d trade the other four for it. It hands you contract terms you don’t yet know to ask for, from someone who learned them at their own expense and has no reason not to share.

The answers tend to be specific and unglamorous. Notice periods on model changes. A cap on the annual uplift. A named support contact. The right to add users mid-term at the original rate. Write every one of them into your contracting checklist as it arrives.

The rest of the sample

Three selected references are a sample of one population: satisfied customers, in the segment the vendor serves best, at their current maturity. You need at least one point outside that.

Former customers. The hardest to find and the most informative. Your own network is the usual route. Peer groups and industry associations are better than they sound, because someone in the room has always left someone.

Organizations that evaluated and chose otherwise. Often easier to reach, and they’ll tell you exactly what tipped it, which is usually one specific thing rather than an overall judgment.

Old case studies. Take the vendor’s published customer stories from three years ago and check whether those organizations are still customers. It costs an hour. A published reference who has since moved on is worth a call.

Illustratively, on one evaluation I’d put the three given calls at around eight useful minutes each. One conversation with a company that had left the vendor eighteen months earlier produced more than all three combined, largely because they had nothing to be careful about.

Weighting the unhappy one

A churned customer is also a biased sample, and it’s worth saying so plainly rather than treating the negative view as the true one.

They left for a reason. The reason may be specific to their data, their sector, their sponsor leaving, or a botched implementation by a partner the vendor no longer works with. Any of those can be irrelevant to you.

What you’re after isn’t the negative account. It’s the variance. Three positive calls tell you the product works somewhere. A fourth call from outside that population tells you how wide the band is between the best case and a bad one, and the width of that band is the actual risk you’re taking.

One practical thing: ask each of the three given references what they have in common with the others. Sometimes they know. Same industry, same region, same implementation partner, all onboarded by the same person. If all three share a trait you don’t share, you’ve learned something the calls themselves were never going to tell you.

One thing to do differently

Ask every reference what they’d put in the contract if they were signing again, and keep a running list across all the calls.

By the fourth conversation you’ll have a set of clauses assembled from other people’s expensive lessons, which is a considerably better starting position than a template your legal team last updated for a hosting agreement.