Qual vs. Quant: When to Listen and When to Measure

I have written about qualitative vs quantitative research before, but I still get a lot of questions about it. To answer some of those questions, I want to do a bit of a deeper dive here and give a few examples to help startups answer the key question.

To be clear, that key question is “when should I use qualitative research, and when should I use quantitative research for the best results?” Another way of looking at this is, “when should I be listening to users, and when should I just be shipping code and looking at the metrics?”

The real answer is that you should do both constantly, but there are times when one is significantly more helpful than the other.

I will continue to repeat my cardinal rule: Quantitative research tells you WHAT your problem is. Qualitative research tells you WHY you have that problem.

Now, let’s look at what that actually means to you when you’re making product decisions.

A One Variable Change

When you’re trying to decide between qualitative and quantitative testing for any given change or feature, you need to figure out how many variables you’re changing.

Here’s a simple example: You have a product page with a buy button on it. You want to see if the buy button performs better if it’s higher on the page without really changing anything else. Which do you do? Qualitative of quantitative?

That’s right, I said this one was simple. There’s absolutely no reason to qualitatively test this before shipping it. Just get this in front of users and measure their actual rate of clicking on the button.

The fact is, with a change this small, users in a testing session or discussion aren’t going to be able to give you any decent information. Hell, they probably won’t even notice the difference. Qualitative feedback here is not going to be worth the time and money it takes to set up interviews, talk to users, and analyze the data.

More importantly, since you are only changing one variable, if user behavior changes, you already have a really good idea WHY it changed. It changed because the CTA button was in a better place. There’s nothing mysterious going on here.

There’s an exception! In a few cases, you are going to ship a change that seems incredibly simple, and you are going to see an enormous and surprising change in your metrics (either positive or negative). If this happens, it’s worth running some observational tests with something like UserTesting.com where you just watch people using the feature both before and after the change to see if anything weird is happening. For example, you may have introduced a bug, or you may have made it so that the button is no longer visible to certain users.

What Does Your User Know?

You and your customer have very different sets of information. This shouldn’t come as a surprise to you, since by now it should be obvious that you are not your user.

Sometimes it can be very hard to distinguish what you know about your product or industry from what your user knows. But it’s important! Making assumptions about your user’s knowledge can lead to products that are impossible for normal people to use.

You Know...

The Details of How Your Product Works

You know all the technical and implementation details of your product. Your user doesn’t even know what those things mean.

One company I worked with had a feature that allowed users to mark off all the items they owned from a list. The engineers knew that, when a user made a selection, that selection was sent to the server in an AJAX request, so the account was kept constantly up to date.

The users didn’t know this. During testing, several users went through, marked off all their selections, and then searched in vain for a Save button. They assumed that they would have to save their work, since they had no idea about the AJAX request that was happening in the background.

The solutions to this were either to allow the users to explicitly save with a button or to give them a tiny amount of feedback while the item was being saved via a very brief wait spinner.

Once we implemented the latter solution, users immediately understood that each click was actually saving the item automatically, and they no longer looked for that Save button.

The take away: Don’t make any assumptions that your users understand what you’re doing for them unless you’re explicitly telling them in some way.

Every Single Feature and Its Purpose

You designed and built every feature in your product, so you know exactly what each of them does and where to find it in your product. Your user knows only what she finds during her time using your product.

One company I worked with had a very useful feature. When I interviewed users about new features they’d like to see, many of them requested the feature, even though it was already in the product! They simply had no idea that it even existed.

The take away: It’s not enough to create fabulous new features. You have to make sure that your features are discoverable by normal users.

Specialized Knowledge About the Product Space

Sometimes we build products to help people do hard things more easily.

Tax preparation software is a fantastic example of this. It is safe to say that most of us who use tax preparation software do not know nearly as much about tax preparation as the people who are building the software. At least, I hope they know more than I do!

The problem arises when we lose touch with exactly which parts of the space the users understand well. We can start to use jargon or terminology that makes perfect sense to us because we hear it all the time. We can design processes that seem completely reasonable if the user already knows the goal.

Unfortunately, this creates complicated, confusing products that assume a much higher level of understanding than the user has.

The take away: When you’re trying to help users accomplish a complicated goal, you need to work even harder to keep the interface simple.

Of course, your customer knows some stuff you don’t know...

How She’s Used to Doing Things

If you’re creating a product that is meant to help a users do something they already do (again, tax preparation or any business software), your goal is to create a generic experience that will satisfy as many people in your core demographic as possible.

But remember, each of those users already does things slightly differently. For example, if you’re helping people who sell things on eBay, you have to understand that each seller already has a process that she follows - from pricing to listing to shipping to dealing with customers.

Asking your users to change too many of their behaviors in order to use your product creates a huge barrier to acceptance.

If you only talk to a few potential customers (or, even worse, none at all), you run the risk of creating something that isn’t broadly usable by all sorts of different users.

The take away: Understanding the variations in user behavior will help you deliver something that is usable by a larger segment of your user base.