5 Mistakes People Make Analyzing Qualitative Data

My last blog post was about common mistakes that people make when analyzing quantitative data, such as you might get from multivariate testing or business metrics. Today I’d like to talk about the mistakes people make when analyzing and using qualitative data.

I’m a big proponent of using both qualitative and quantitative data, but I have to admit that qualitative feedback can be a challenge. Unlike a product funnel or a revenue graph, qualitative data can be messy and open ended, which makes it particularly tough to interpret.

For the purposes of this post, qualitative information is generated by the following types of activities:

• Usability tests
• Contextual Inquiries
• Customer interviews
• Open ended survey questions (ie. What do you like most/least about the product?)

Insisting on Too Large a Sample

With almost every new client, somebody questions how many people we need for a usability test “to get significant results.” Now, if you read my last post, you may be surprised to hear me say that you shouldn’t be going for statistical significance here. I prefer to run usability tests and contextual inquiries with around five participants. Of course, I prefer running tests iteratively, but that’s another blog post.

Analyzing the data from a qualitative test or even just reading through essay-type answers in surveys takes a lot longer per customer than running experiments in a funnel or looking at analytics and revenue graphs. You get severely diminishing returns from each extra hour you spend asking people the same questions and listening to their answers.

Here’s an example from a test I ran. The customer wanted to know all the different pain points in their product so that they could make one big sweep toward the end of the development cycle to fix all the problems. Against my better judgment, we spent a full two weeks running sessions, complete with a moderator, observers, a lab, and all the other attendant costs of running a big test. The problem was that we found a major problem in the first session that prevented the vast majority of participants from ever finding an entire section of the interface. Since this problem couldn’t be fixed before moving on to the rest of the sessions, we couldn’t actually test a huge portion of the product and had to come back to it later, anyway.

The Fix: Run small, iterative tests to generate a manageable amount of data. If you’re working on improving a particular part of your product or considering adding a new feature, do a quick batch of interviews with four or five people. Then, immediately address the biggest problems that you find. Once you’re done, run another test to find the problems that were being masked by the larger problems. Keep doing this until your product is perfect (ie. forever). It’s faster, cheaper, and more immediately actionable than giant, statistically significant qualitative tests, and you will eventually find more issues with the same amount of testing time.

It’s also MUCH easier to pick out a few major problems from five hours of testing than it is to find dozens of different problems from dozens of hours of testing. In the end though, you’ll find more problems with the iterative approach.

5 Big Mistakes People Make When Analyzing User Data

I was trying to write a blog post the other day about getting various different types of user feedback, when I realized that something important was missing. It doesn’t do any good for me to go on and on about all the ways you can gather critical data if people don’t know how to analyze that data once you have it.

I would have thought that a lot of this stuff was obvious, but, judging from my experience working with many different companies, it’s not. All of the examples here are real mistakes I’ve seen made by smart, reasonable, employed people. A few identifying characteristics have been changed to protect the innocent, but in general they were product owners, managers, or director level folks.

This post only covers mistakes made in analyzing quantitative data. At some point in the future, I’ll put together a similar list of mistakes people make when analyzing their qualitative data.

For the purposes of this post, the quantitative data to which I’m referring is typically generated by the following types of activities:

• Multivariate or A/B testing
• Site analytics
• Business metrics reports (sales, revenue, registration, etc.)
• Large scale surveys

Statistical Significance

I see this one all the time. It generally involves somebody saying something like, “We tested two different landing pages against each other. Out of six hundred views, one of them had three conversions and one had six. That means the second one is TWICE AS GOOD! We should switch to it immediately!”

Ok, I may be exaggerating a bit on the actual numbers, but too many people I’ve worked with just ignored the statistical significance of their data. They didn’t realize that even very large numbers can be statistically insignificant, depending on the sample size.

The problem here is that statistically insignificant metrics can completely reverse themselves, so it’s important not to make changes based on results until you are reasonably certain that those results are predictable and repeatable.

The Fix: I was going to go into a long description of statistical significance and how to calculate it, but then I realized that, if you don’t know what it is, you shouldn’t be trying to make decisions based on quantitative data. There are online calculators that will help you figure out if any particular test result is statistically significant, but make sure that whoever is looking at your data understands basic statistical concepts before accepting their interpretation of data.

Also, a word of warning: testing several branches of changes can take a LOT larger sample size than a simple A/B test. If you're running an A/B/C/D/E test, make sure you understand the mathematical implications.

Short Term vs. Long Term Effects

Again, this seems so obvious that I feel weird stating it, but I’ve seen people get so excited over short term changes that they totally ignore the effects of their changes in a week or a month or a year. The best, but not only, example of this is when people try to judge the effect of certain types of sales promotions on revenue.

For example, I've often heard something along these lines, “When we ran the 50% off sale, our revenue SKYROCKETED!” Sure it did. What happened to your revenue after the sale ended? My guess is that it plummeted, since people had already stocked up on your product at 50% off.

The Fix: Does this mean you should never run a short term promotion of any sort? Of course not. What it does mean is that, when you are looking at the results of any sort of experiment or change, you should look at how it affects your metrics over time.

An Agile Alternative: Embedded Design

Last week I wrote a blog post about a depressing example of non-agile design. This week, I'd like to show you an alternative method of design, along with some examples drawn from my experiences. This is not, by any means, a new concept, but hopefully I can convince a few people who still think that agile development and interaction design don't go together.

I've worked as a designer and user researcher with agile teams on several different projects. The trick is that, instead of doing all the research and design up front and then throwing the work over the wall to the engineering team, the designer stays embedded with the team throughout the entire development process.

How does an embedded interaction designer help at various stages of the development process?

Strategy and Product Definition

At this stage, your designer (or design and research team, if you’re well-funded) should be out talking to potential users about their problems. A designer at this point in the cycle can help you define who your customer is, learn how potential customers are currently dealing with the problem that your product solves, and come up with a small set of crucial features that will solve those problems better.

Could a great product manager also do some of these tasks? Sure, and they should definitely be involved or even leading the process. But, in my experience, being involved at this stage allows me, as a designer, to really understand the problems I’m going to be solving and the people for whom I’m going to be solving them. In other words, to have a successful design process, I will have to know all these things anyway, and participating in the earliest stage of research is the most efficient way for me to learn them.

Example:  One company with whom I was working wanted to add a new feature that would allow users to play music while interacting with the product. From our discussions with users of the product, we knew that people were already doing this, but in a way that caused them all sorts of issues and made many people unhappy. We interviewed current users about things like:

• How they were currently using music with the product?
• What benefits did they get from using music with the product?
• What major problems were they having with music? 

We also looked at the quantitative data that we had about what sort of music was being played, how much people were likely to pay for it, and  what percentage of people were active music users. The answers to all of these questions helped us come up with a new concept for music that would solve many of the problems and add some fun new benefits

A Depressing Example of Non-Agile Design

I recently had a really depressing experience. I was asked to work on a very cool application that focused on my favorite subject. That’s not the depressing part. That part was awesome.

Despite a lot of misgivings, I agreed to use the waterfall development process and design a lot of stuff up front, because that’s what the client wanted. Don’t get me wrong. They were doing some development work on the back end, but the goal was to figure out the entire application UX and feature set, wireframe it, and get a visual design done before starting work on anything customer-facing.

The application was quite large and had several different areas. I specifically designed it so that many of the features were modular, which meant that the development team could leave out whole chunks of the UI without ruining the application. They could then add more feature modules in as they finished them, which would theoretically allow them to launch as soon as they had just a few modules finished. Even though we were using a waterfall method, I didn’t want to design something that had to be absolutely 100% finished and built before it was usable. Basically, I was giving them the option to be agile in their development process, if they chose to, even if everything was designed up front.

After a few months of design, we had a great product roadmap, a list of features we ideally wanted in version 1.0, a list of features that would be acceptable for 1.0 (still FAR more than a MVP), lots of fully interactive wireframes, a gorgeous visual design (not done by me, of course), and a lot of data ready to go on the back end. We had also burned through our design budget, so I turned everything over to the developers and moved on to other things.

After several months, the very first version of the product shipped. This is the depressing part. It included exactly 0% of the original design. That’s right. Not a single element I’d designed made it into the first version of the product. Unfortunately, because all the design work had been done up front, I had no idea why the company made the decisions they had made. Of course, if I had been involved with the project all the way through, I could have stripped down the design in a way that at least tried to stay true to the original vision of the product. Instead, it came across as something that had never really been designed at all.

Oddly, there’s a part of me that’s happy that they did what they did. After all, they ended up releasing a MUCH smaller version of the product than we had originally envisioned. I’m not sure if it’s a Minimum Viable Product – only time will tell if it’s viable, after all – but it has many fewer features than we originally envisioned.

But, in another way, I’m sad. If I could have convinced them to go this small at the very beginning of the project, I could have designed their Minimum Viable Product in a lot less time, and, I think, with a lot better result than they got doing it on their own. More importantly, we could then have worked together to build out new features based on learning from the MVP, and we probably still would have had some design budget left to eventually implement that gorgeous visual design.

I guess my lesson from all of this is that I need to be stricter with companies who want a giant, fully fleshed-out product all designed for them at the beginning of the process. I need to be clearer that they’d be better off designing something very small and immediately implementable and then working from there. And I need to encourage them to spread the design budget out over a longer period of time so that they are never left without design resources. Either that, or I need to let go of my designs once they’ve been handed over to the client and acknowledge that they get to do (or not do) anything they want with them. I just find that really depressing.

Why Your Customer Feedback is Useless

Here’s the scenario: You have a minimum viable product. You’re talking to your users about it. You’re asking them questions, and they’re answering. But for some reason, it’s just not turning into usable information.

You wonder what’s going on. You imagine that perhapsyour users suck at giving good feedback or else they don’t have anything useful to say. Maybe, you think in a moment of hopeful delusion, your product is so perfect that it can’t be improved by customer feedback.

While these are all possibilities, the reality is that it’s probably not your customers’ fault. So, if you don’t seem to be getting any good data, what IS the problem? Probably one of the following things:

You’re asking the wrong questions

I’ve written before about asking customers the wrong questions, but in summary, customers are very good at giving you certain kinds of information and very bad at other kinds. For example, users are great at telling you about their problems. They can very easily tell you when something isn’t working or interesting or fun to use. What they suck at is telling you how to fix it.

Customers are great at:

• Complaining about problems
• Describing how they currently perform tasks
• Saying whether or not they like a product
• Showing you parts of a product that are particularly confusing
• Comparing one product to another similar product
• Explaining why they chose a particular method of doing something

Customers are bad at:

• Predicting their future behavior
• Predicting what other people will like
• Predicting whether they’ll pay for something
• Coming up with innovative solutions to their own or other people’s problems
• Coming up with brand new ideas for what would make a product more appealing

To take advantages of users’ strengths, have them describe things like, “Tell me about your most recent experience using the product and how that went for you.” Or ask them questions like, “What about [competitor product] do you particularly enjoy? What do you hate?” You can even ask questions like, “Of the following 5 features, which would you prefer?” You’ll want to be a lot more careful about listening to their answers to overly broad questions like, “What brand new feature would you like to see implemented?” or “What would make this product more fun to use?”

Seven More Ways People Suck at Customer Development

I’ve spent many years talking to users about how to improve products. It’s a big part of the job of any UX professional. With the recent emphasis on lean customer development and MVP, talking to customers is no longer limited to a couple of people in an organization. These days, everybody is learning from customers, and that’s a great thing for the usability of products.

Unfortunately, it does cause a few new problems. The main one is that most people who are new at talking to users kind of suck at it.

I’ve written a bit about the mistakes that inexperienced user researchers make when talking to users. However, there are several other serious problems that are far more likely to occur when the people doing the interviewing are deeply invested in the product. If you’re a founder, an engineer, a designer, a product owner, or anybody else with strong, emotional ties to your product, and you’re trying to learn by talking to users, you need to make sure you’re not falling into any of the following traps.

The Big Sale

Often, when you’re sitting down with a potential customer to talk them about your product, your first impulse is to make as good an impression as possible. This is perfectly natural. The problem is, it can really get in the way of getting good information from the potential customer about how to make your product better.

If your goal is to understand what’s not working about your product or what’s preventing people from buying it, stop trying to sell your product to the potential customer. Don’t read off a list of features that are in the product or that will be in the product. Don’t tell the customer how awesome the product is or how good it is at solving all their problems. And, for the love of God, don’t tell them about your Vision for the product.

What should you do? Shut up and let the customer tell you about their problems and how they’d like to solve them. Listen to the customer tell you about his perception of your product and what he sees as wrong with it. Your goal isn’t to convince the customer that your product is great. Your goal should be to let the customer help you make your product great.

How to Make Your Product as Awful as a Corporate Intranet

Recently I was dealing with intranets for a couple of largish companies, and I realized something. In every company where I've worked or contracted, intranets have been a problem. For some reason, the vast majority of intranets turn into huge, unwieldy, out-of-date, unusable link farms with bad search results. I’m told there are exceptions, but I’ve never seen them. Even small companies have big, awful intranets.

I started to figure out why intranets are so bad, and I came up with a list of problems that all of the ones I’ve seen seem to share. That’s when I noticed that these problems are in no way unique to intranets.

Many products I’ve used or worked on have had at least some of these problems.

None of the products I’ve used has ever managed to combine all of these problems into one enormous, unholy mess in quite the same way that a bad intranet can, but that doesn’t mean you shouldn’t give it a try! To help you out, here is a list of horrible things you can do to make your product just as bad as a typical intranet. Good luck!

Never throw anything away

The first key to really ruining your product’s usability is to never throw anything out.

Have a new feature, link, or module? Throw it onto the main screen along with everything else you’ve ever released! If even one person thinks it’s important or useful, you must be sure to support it until the end of time. This will ensure that your interface is so cluttered that nobody will be able to use the 10% of your product that is actually interesting to 90% of your customers.

Don’t put anybody in charge

It is very important to make sure that no single person is responsible for your overall product. If you want a really terrible product, make sure to enforce this total lack of responsibility and ownership.