I’ve come to believe that customer validation interviews are detrimental to a pre-product startup for two reasons:
- You’ll wind up building something no one wants.
- You’ll wind up not building something everyone wants.
Building something no one wants
I’m going to share a painful story with you.
About a year ago, our three-person team began to build a software product. Two of us had been trained in “customer validation” methodology in business school, which essentially preaches that if you’re starting a company, you should talk to your customers before you build anything so that you know exactly what they want and don’t waste time or money building something no one will use.
Makes perfect sense. So much sense, in fact, that it seems like a risk-less way to start a company. If your prospective customers tell you exactly what they want, how much they’ll pay for it, and what features to include, and you build it, there’s almost no chance you won’t have a successful company, right?
But we didn’t know that. We had the initial idea for the product, which was going to be an eCommerce platform built for pre-sales, much like Kickstarter. It would allow companies to sell a product before they had inventory, then use the cash from the sales to make and ship the orders they had already sold. Kickstarter was basically doing that very successfully along with a couple others, and we were going to make an even better version.
We began interviewing prospective customers, asking them if they would use such a platform, what features they would want, how much they’d be willing to pay for it, etc.
We conducted about 100 interviews, many in person, driving all over the state to meet with people.
Of the 100 people we talked to, 98 said they would use it. Some were really excited and almost begged us to build the platform, so we did. We spent about 3 months and $10,000 doing so, and then we launched it.
We reached out to media, contacted over 5,000 potential users we had manually identified over the Summer, and reached out to the people we had interviewed.
Of the 98 people that said they would use it, when we finally finished the platform and sent them links to it, 98 didn’t use it. In fact, no one used it.
We were completely shocked. What in the world had happened?? We followed the recipe perfectly. We did exactly what business school had taught us, we killed ourselves finding people to interview, all the early signs indicated that we would be successful, and then we failed hard.
Once we began the postmortem, it didn’t take long to see why. When you’ve just bought someone lunch and you ask them earnestly, “do you think this is a good idea? Will you use it? Will you pay for it?” most people will say yes. Not because they mean yes, but because that’s what people do. It’s like a woman asking her husband if she looks fat. Sure, there are some who will tell you the truth (two out of 100 in our case), but most will tell you what you want to hear.
Interviews are a completely flawed mechanism – they don’t actually tell you whether or not people will pay for your product once it’s built, they only tell you what people will say if you ask them “will you pay for this?” in an interview.
Doing customer interviews or surveys to determine whether people will pay for something is like eating a french fry to find out what a hamburger tastes like. It’s absurd!
This is an experiment – you have a stimulus, subjects, and a response. You present a stimulus to the subjects in order to observe their responses.
The problem with customer validation interviews is that the stimulus changes dramatically once you actually build the product. During the customer interview phase, your stimulus is an interview or survey, and is likely presented by contacting people directly. Once you build the product, the stimulus is most likely the product itself, which customers find through normal life activities, like seeing an ad, or searching online.
Therefore, it’s likely that the results will be dramatically different.
Thinking that the customer interview experiment result will be an approximation of the true buying experiment is a fallacy. Interviews are nothing like normal buying behavior, so doing an interview tells you nothing about buying behavior, regardless of what the subjects say.
So, despite the promise of customer interviews, you still might wind up building something nobody wants.
Not building something everyone wants
I’ve got this new business idea. It’s really great. I’m kind of scared you’ll steal it, but I’ll tell you anyway, because I’m supposed to validate it through customer interviews. Come closer so no one else hears me.
[Whispering loudly] it’s a teleportation device. You wear it on your wrist. It can transport you to anywhere in the world instantly. It’s going to eliminate the need for cars, planes, gasoline, maybe even shoes! The military will totally buy it. We could even use it to ship products instantly. Imagine how excited Amazon will be! Think about it – you could go to Hawaii today, just for lunch, and be back to work before your break is over. No more traffic. So much time saved. I already built a prototype, check it out (shows you black wristband with circuit board and 3d-printed housing).
So, would you use this? Would you pay for it? Do you want it?
Yes yes yes, obviously.
Do you want to be able to pick what color it is, or should we just have a black one?
Ummm, I like red, so picking a color would be cool. Also, it should fit in my pocket. Or be on my phone. Or be an app on my phone, so I don’t have to carry another thing around.
Oh yeah, that’s good. It shouldn’t be too hard to add that feature. So if I make it, will you pay $10,000 for it?
Yeah, sure, it would replace my car, plane tickets, all transportation cost, so totally, yes. Consider it paid.
Congratulations, cowboy, you’ve just come up with a product everyone wants, and that consumers will pay $10,000 for! You’re rich! Hey, you could even pre-sell these on a site like Kickstarter!
So you do. You pre-sell $50 million dollars worth in just 30 days, the largest Kickstarter campaign ever.
But now your excited customers want their product, and you realize, you can’t build it. The physics is completely faulty. It’s been two months, where’s my package with the teleportation device? Where’s my app? This is beginning to look like a scam!
Sounds like an absurd scenario, but I can think of three companies without even trying who have done exactly this. Their products seemed more feasible than the teleportation device, of course, but they turned out to be equally impossible to build with their available resources.
So, because they did customer interviews, they were sure they were on to something. And they were. Consumers probably would buy their products. But they couldn’t build them. So they unknowingly endeavored on a death-march laden with beautiful sights along the way, like business competition winnings and accelerator acceptances, but a fatal result awaiting them at the end.
I’m not saying that customer interviews are always detrimental or useless. For people who have already built a product, even a minimal one, they seem to be very helpful. But for those who haven’t, I’ve seen nothing but pain and failure result. It’s time for business schools to quit teaching students to do them.