What to do when you can’t trust your customer feedback

Yesterday I read a great article called What You Should Never Ask Your Customers by David Bradly.

The article talks about how asking the wrong questions during customer interviews and user research leads to bad answers and a product catastrophe. It also mentions the moms test, a term I wasn’t familiar with, which is based on the book The Moms Test by Rob Fitzpatrick.

The moms test: 

“Everyone is innocently lying to you all the time, for a wide variety of reasons and the person who lies to you most is your mother, because she loves you and doesn’t want to hurt your feelings.”
(if you’re like me, you’re probably already falling in love with this brilliant premise…)

Failing the moms test is easy: all you need to do is ask your customers some really bad questions:
1. Ask generic questions such as “how often do you go to the gym?” 

A typical answer might be “twice a week”, which is probably a lie, but if you dig deep enough (like, for example: “when was the last time you went to the gym?”) you may get something like: “Last Monday” which is more than 10 days ago…

2. Ask for someone’s opinion about your product or idea: “What do you think about…?” 

A perfect recipe for getting biased answers you shouldn’t trust: when it comes to businesses and products, even seasoned experts are wrong most of the time. Just speak to any angel investor, or serial entrepreneur, or product manager about their success rate. Even experts with access to all the data in the world such as Gartner make false predictions such as this one: Gartner: Microsoft Will Beat Apple In Mobile By 2015 😂.

3. Ask hypothetical questions such as: “would you go to the gym more often if it had…?”

Hypothetical questions provide guesses and basing your product or business on the guesses of your unprofessional customers is professional suicide.

I just love the moms test! Let me throw in the friends test.

The friends test: 

The premise behind the friends test is that half of the things you want to do with your friends never happen because of a variety of reasons, but that doesn’t stop you from giving those empty statements and promises that never happen.

• “We should go out more often” 
• “We should play tennis sometimes, it’s been a while!” (20 years…)
• “We should arrange a weekend with the families!” 
• “Next time you are flying to London let me know and I’ll join you (leaving leave my 3 months baby…)”

Friends don’t lie to each other consciously. They mean what they say but their wishful thinking, combined with a total lack of self-awareness, are distorting their thoughts, making it impossible to trust what they say.

We all fail the friends test, which reminds me of a story:

The (rise and) fall of our local running group

A few months ago, in one of our much-unneeded neighborhood events, my wife was persuaded by her friends to start a running group on WhatsApp. The idea was simple: everyone we know runs every once in a while. In the cold winter it’s harder to find the energy to do it, and having an extra social push from someone in the neighborhood might help make it happen. We created a running group.
Everyone we know (well, besides a few anti-socials like myself…) asked to join this group: neighbors, parents from our kindergarten, parents from our kids’ school, etc.
In terms of early registration - the conversion rate was beyond comparison (almost everyone involved asked to join the group). We even started to believe a small app would be a good idea for this.

Reality check

3 weeks and 0 activity in the group later - it was obvious this product had serious engagement issues.

• In theory, we believed the running group had a strong product/market fit.
• In reality - we all failed the friends test, proving once again that the zombies are not coming.

Not only for new products

Getting customer feedback for an existing product can go wrong as well; some customers might give you the answers they think you are looking for while others might drag the discussion to specific feature requests or a checklist-style comparison between you and your competitors.

What to do when you can’t trust anyone

So if everybody is lying or just giving you the wrong answers, how can you get that feedback you need to build your product?

Before you have a product: 

1. Conduct your interviews carefully: talk less and listen more.
2. Collect facts and not opinions.
3. Understand what your potential customers do rather than what they say they do (or think they will do). 
4. Never ask: “how much would you pay for it?”, ask: “how much did you pay for... [something similar]”.
5. Collect problems & pains, not feature requests (see: problem roadmaps).

When you have a product direction: 

1. Create a low budget MVP using existing tools like email, facebook, WhatsApp groups.
2. Pro tip: get volunteers to test your potential competitors’ apps. See what works and what’s not. 
3. Cherry-pick your first users and early adopters. Wrong users will lead to poor decisions and poor product/market fit.
4. Make your first users commit to the MVP (by paying or committing to being active) - this will help you screen out bad candidates, even if they are your friends. 
5. Do not count on your customers to do the visionary work for you. Problem solving is an art. Your art. 

Which reminds me how 10 years ago our field service customers asked us to develop timesheets app running on Blackberry, the most popular business smartphone back at the time. Analyzing the future we rejected their request, and built our product strategy on iPhone and Android instead, resulting in hundreds of happy customers.

I use it every time someone comes to my room and throws a random feature request at me... 

When your product is up and running:

1. Measure every possible thing: key mobile metrics, user behavior in each session, user behavior over time, analytics based on individuals, cohorts, segments, everything you can possibly think of.
2. Store every piece of information possible - believe me, your database will not collapse, but this data will be needed someday. 
3. Use A/B test to prove or disprove your theories without asking your customers about it and while neutralizing biases and external events that might influence the results. 
4. Your customers are nothing but numbers. Get to know them, show empathy, but take cold decisions based on their behavior only.  
5. Ignore vocal minority and focus on the silent majority - always prefer the big picture over local escalations. 
6. Make data-driven decisions and analyze your decisions based on data.