5 Things People Get Wrong When Talking to Users

This post originally appeared on the Sliced Bread Design blog.

I was talking to an engineer the other day who was describing his startup's first experience in trying to get user feedback about their new product. Since it was a small company and the product didn't exist in production yet, their goals for gathering user feedback were:

• Get information about whether people thought the product was a good idea.
• Identify potential customer types, both for marketing and further research purposes.
• Talk to as many potential users as possible to get a broad range of feedback.
• Keep it as cheap as possible!

He had, unsurprisingly, a number of stories about mistakes they had made and lessons they'd learned during the process of talking to dozens of people. As he was sharing the stories with me, the thought that kept going through my head was, "OF COURSE that didn't work! Why didn't you [fill in the blank]?" Obviously, the reason he had to learn all this from scratch was because he hadn't moderated and viewed hundreds of usability sessions or had any training in appropriate user interview techniques. Many of things that user researchers take for granted were brand new to him. Having spoken with many other people at small companies with almost non-existent research budgets, I can tell you that this is not an isolated incident. While it's wonderful that more companies are taking user research seriously and understanding how valuable talking to users can be, it seems like people are relearning the same lessons over and over.

In order to help others who don't have a user experience background not make those same mistakes, I've compiled a list of 5 things you're almost certainly doing wrong if you're trying to get customer feedback without much experience. Even if you've been talking to users for years, you might still be doing these things, since I've seen these mistakes made by people who really should know better. Of course, this list is not exhaustive. You could be making dozens of other mistakes, for all I know! But just fixing these few small problems will dramatically increase the quality of your user feedback, regardless of the type of research you're doing.

Don't give a guided tour

One of the most common problems I've seen in customer interviews is inexperienced moderators wanting to give way too much information about the product up front. Whether they're trying to show off the product or trying to "help" the user not get lost, they start the test by launching into a long description of what the product is, who it's for, what problems it's trying to solve, and all the cool features it has. At the end of the tour, they wrap up with a question like, "So, do you think you would use this product to solve this exact problem that I told you about?" Is there any other possible answer than, "ummm...sure?"

Instead of the guided tour, start by letting the user explore a bit on his own. Then, give the user as little background information as possible to complete a task. For example, to test the cool new product we worked on for Superfish, I might give them a scenario they can relate to like, "You are shopping online for a new pair of pants to wear to work, and somebody tells you about this new product that might help. You install the product as a plug in to Firefox and start shopping. Show me what you'd do to find that pair of pants." The only information I've given the user is stuff they probably would have figured out if they'd found the product on their own and installed it themselves. I leave it up to them to figure out what Superfish is, how it works, and whether or not it solves a problem that they have.

A/B and Qualitative User Testing

Recently, I worked with a company devoted to A/B testing. For those of you who aren't familiar with the practice, A/B testing (sometimes called bucket testing or multivariate testing) is the practice of creating multiple versions of a screen or feature and showing each version to a different set of users in production in order to find out which version produces better metrics. These metrics may include things like "which version of a new feature makes the company more money" or "which landing screen positively affects conversion." Overall, the goal of A/B testing is to allow you to make better product decisions based on the things that are important to your business by using statistically significant data.

Qualitative user testing, on the other hand, involves showing a product or prototype to a small number of people while observing and interviewing them. It produces a different sort of information, but the goal is still to help you make better product decisions based on user feedback.

Now, a big part of my job involves talking to users about products in qualitative tests, so you might imagine that I would hate A/B testing. After all, wouldn't something like that put somebody like me out of a job? Absolutely not! I love A/B testing. It's a phenomenal tool for making decisions about products. It is not the only tool, however. In fact, qualitative user research combined with A/B testing creates the most powerful system for informing design that I have ever seen. If you're not doing it yet, you probably should be.

A/B Testing

What It Does Well

A/B testing on its own is fantastic for certain things. It can help you:

• Get statistically significant data on whether a proposed new feature or change significantly increases metrics that matter - numbers like revenue, retention, and customer acquisition
• Understand more about what your customers are actually doing on your site
• Make decisions about which features to cut and which to improve
• Validate design decisions
• See which small changes have surprisingly large effects on metrics
• Get user feedback without actually interacting with users