Your Users Are Doing Something Surprising

This post is for all of you lucky enough to have a product with real users. Way back before you had users, or even a product, you probably went through a process to figure out what you should build. During that process you may have written user stories and work flows that described, in various levels of detail, how your users would perform each expected task. But you know who didn’t read your user stories? That’s right: your users.

The result? Somewhere out there, a whole lot of your users are doing something totally unexpected with your product.

Ok, maybe it’s obvious that sometimes users do the unexpected. Maybe you expected your SMS-based social network to help people find out where their friends are in real time, but instead celebrities started using it to market directly to their fans. Maybe you created a product designed to help bands promote themselves, but instead ended up with a social network where people hook up with their high school sweethearts. Or, although this seems extremely unlikely, maybe you had an MMO but people just used it to share photos.

Whatever they’re doing, it’s something you didn’t expect, but it’s very important that you learn what it is as soon as possible!

Why Should You Care?

There are three excellent reasons for you to know what your customers are actually doing with your product:

• So you know if you are missing an opportunity to pivot your product or marketing
• So you know if you are missing an important feature
• So you don’t accidentally destroy a commonly used workaround or "unplanned feature"

Pain Driven Design

I have a problem with both User Centered Design and Customer Driven Development. This may come as something of a shock to people who read my blog, since I’m constantly giving advice on better ways to talk to users, analyze data, and improve customer development efforts.

The problem I have with UCD and CDD is not the methods. The problem is that people so often misunderstand them. People hear “user centered” and think, for some insane reason, that we are encouraging them to turn their entire design process over to the user. They hear “listen to your customer” and think that we want them to blindly accept every ridiculous feature request made by somebody with a credit card and an opinion.

Guess what? I rarely speak for entire communities of people, but I think I can safely say that nobody in the user centered design or customer driven development business are asking that of anybody. If they are, they’re idiots and shouldn’t be listened to anyway.

Unfortunately, a lot of us have debunked this myth by explaining that we don’t actually think that customers can predict future behavior (even their own) or that they’re going to have some grand design vision for your product. We just think that customers are great at talking about their problems and pain points, and that those are good things for you and your designers to know when you create a new feature or product.

I’m suggesting that we come up with a new name that will be harder (or at least more fun) to misinterpret: Pain Driven Design.

What Is Pain Driven Design (PDD)?

The PDD methodology requires that, before you start design for a product or feature, you need to figure out what is causing pain for your users and potential users. The desired outcome of PDD is to make that pain go away by some brilliant method of your devising. You then check to make sure you made their pain go away without causing them any more pain.

5 Mistakes People Make Analyzing Qualitative Data

My last blog post was about common mistakes that people make when analyzing quantitative data, such as you might get from multivariate testing or business metrics. Today I’d like to talk about the mistakes people make when analyzing and using qualitative data.

I’m a big proponent of using both qualitative and quantitative data, but I have to admit that qualitative feedback can be a challenge. Unlike a product funnel or a revenue graph, qualitative data can be messy and open ended, which makes it particularly tough to interpret.

For the purposes of this post, qualitative information is generated by the following types of activities:

• Usability tests
• Contextual Inquiries
• Customer interviews
• Open ended survey questions (ie. What do you like most/least about the product?)

Insisting on Too Large a Sample

With almost every new client, somebody questions how many people we need for a usability test “to get significant results.” Now, if you read my last post, you may be surprised to hear me say that you shouldn’t be going for statistical significance here. I prefer to run usability tests and contextual inquiries with around five participants. Of course, I prefer running tests iteratively, but that’s another blog post.

Analyzing the data from a qualitative test or even just reading through essay-type answers in surveys takes a lot longer per customer than running experiments in a funnel or looking at analytics and revenue graphs. You get severely diminishing returns from each extra hour you spend asking people the same questions and listening to their answers.

Here’s an example from a test I ran. The customer wanted to know all the different pain points in their product so that they could make one big sweep toward the end of the development cycle to fix all the problems. Against my better judgment, we spent a full two weeks running sessions, complete with a moderator, observers, a lab, and all the other attendant costs of running a big test. The problem was that we found a major problem in the first session that prevented the vast majority of participants from ever finding an entire section of the interface. Since this problem couldn’t be fixed before moving on to the rest of the sessions, we couldn’t actually test a huge portion of the product and had to come back to it later, anyway.

The Fix: Run small, iterative tests to generate a manageable amount of data. If you’re working on improving a particular part of your product or considering adding a new feature, do a quick batch of interviews with four or five people. Then, immediately address the biggest problems that you find. Once you’re done, run another test to find the problems that were being masked by the larger problems. Keep doing this until your product is perfect (ie. forever). It’s faster, cheaper, and more immediately actionable than giant, statistically significant qualitative tests, and you will eventually find more issues with the same amount of testing time.

It’s also MUCH easier to pick out a few major problems from five hours of testing than it is to find dozens of different problems from dozens of hours of testing. In the end though, you’ll find more problems with the iterative approach.

5 Big Mistakes People Make When Analyzing User Data

I was trying to write a blog post the other day about getting various different types of user feedback, when I realized that something important was missing. It doesn’t do any good for me to go on and on about all the ways you can gather critical data if people don’t know how to analyze that data once you have it.

I would have thought that a lot of this stuff was obvious, but, judging from my experience working with many different companies, it’s not. All of the examples here are real mistakes I’ve seen made by smart, reasonable, employed people. A few identifying characteristics have been changed to protect the innocent, but in general they were product owners, managers, or director level folks.

This post only covers mistakes made in analyzing quantitative data. At some point in the future, I’ll put together a similar list of mistakes people make when analyzing their qualitative data.

For the purposes of this post, the quantitative data to which I’m referring is typically generated by the following types of activities:

• Multivariate or A/B testing
• Site analytics
• Business metrics reports (sales, revenue, registration, etc.)
• Large scale surveys

Statistical Significance

I see this one all the time. It generally involves somebody saying something like, “We tested two different landing pages against each other. Out of six hundred views, one of them had three conversions and one had six. That means the second one is TWICE AS GOOD! We should switch to it immediately!”

Ok, I may be exaggerating a bit on the actual numbers, but too many people I’ve worked with just ignored the statistical significance of their data. They didn’t realize that even very large numbers can be statistically insignificant, depending on the sample size.

The problem here is that statistically insignificant metrics can completely reverse themselves, so it’s important not to make changes based on results until you are reasonably certain that those results are predictable and repeatable.

The Fix: I was going to go into a long description of statistical significance and how to calculate it, but then I realized that, if you don’t know what it is, you shouldn’t be trying to make decisions based on quantitative data. There are online calculators that will help you figure out if any particular test result is statistically significant, but make sure that whoever is looking at your data understands basic statistical concepts before accepting their interpretation of data.

Also, a word of warning: testing several branches of changes can take a LOT larger sample size than a simple A/B test. If you're running an A/B/C/D/E test, make sure you understand the mathematical implications.

Short Term vs. Long Term Effects

Again, this seems so obvious that I feel weird stating it, but I’ve seen people get so excited over short term changes that they totally ignore the effects of their changes in a week or a month or a year. The best, but not only, example of this is when people try to judge the effect of certain types of sales promotions on revenue.

For example, I've often heard something along these lines, “When we ran the 50% off sale, our revenue SKYROCKETED!” Sure it did. What happened to your revenue after the sale ended? My guess is that it plummeted, since people had already stocked up on your product at 50% off.

The Fix: Does this mean you should never run a short term promotion of any sort? Of course not. What it does mean is that, when you are looking at the results of any sort of experiment or change, you should look at how it affects your metrics over time.

An Agile Alternative: Embedded Design

Last week I wrote a blog post about a depressing example of non-agile design. This week, I'd like to show you an alternative method of design, along with some examples drawn from my experiences. This is not, by any means, a new concept, but hopefully I can convince a few people who still think that agile development and interaction design don't go together.

I've worked as a designer and user researcher with agile teams on several different projects. The trick is that, instead of doing all the research and design up front and then throwing the work over the wall to the engineering team, the designer stays embedded with the team throughout the entire development process.

How does an embedded interaction designer help at various stages of the development process?

Strategy and Product Definition

At this stage, your designer (or design and research team, if you’re well-funded) should be out talking to potential users about their problems. A designer at this point in the cycle can help you define who your customer is, learn how potential customers are currently dealing with the problem that your product solves, and come up with a small set of crucial features that will solve those problems better.

Could a great product manager also do some of these tasks? Sure, and they should definitely be involved or even leading the process. But, in my experience, being involved at this stage allows me, as a designer, to really understand the problems I’m going to be solving and the people for whom I’m going to be solving them. In other words, to have a successful design process, I will have to know all these things anyway, and participating in the earliest stage of research is the most efficient way for me to learn them.

Example:  One company with whom I was working wanted to add a new feature that would allow users to play music while interacting with the product. From our discussions with users of the product, we knew that people were already doing this, but in a way that caused them all sorts of issues and made many people unhappy. We interviewed current users about things like:

• How they were currently using music with the product?
• What benefits did they get from using music with the product?
• What major problems were they having with music? 

We also looked at the quantitative data that we had about what sort of music was being played, how much people were likely to pay for it, and  what percentage of people were active music users. The answers to all of these questions helped us come up with a new concept for music that would solve many of the problems and add some fun new benefits

A Depressing Example of Non-Agile Design

I recently had a really depressing experience. I was asked to work on a very cool application that focused on my favorite subject. That’s not the depressing part. That part was awesome.

Despite a lot of misgivings, I agreed to use the waterfall development process and design a lot of stuff up front, because that’s what the client wanted. Don’t get me wrong. They were doing some development work on the back end, but the goal was to figure out the entire application UX and feature set, wireframe it, and get a visual design done before starting work on anything customer-facing.

The application was quite large and had several different areas. I specifically designed it so that many of the features were modular, which meant that the development team could leave out whole chunks of the UI without ruining the application. They could then add more feature modules in as they finished them, which would theoretically allow them to launch as soon as they had just a few modules finished. Even though we were using a waterfall method, I didn’t want to design something that had to be absolutely 100% finished and built before it was usable. Basically, I was giving them the option to be agile in their development process, if they chose to, even if everything was designed up front.

After a few months of design, we had a great product roadmap, a list of features we ideally wanted in version 1.0, a list of features that would be acceptable for 1.0 (still FAR more than a MVP), lots of fully interactive wireframes, a gorgeous visual design (not done by me, of course), and a lot of data ready to go on the back end. We had also burned through our design budget, so I turned everything over to the developers and moved on to other things.

After several months, the very first version of the product shipped. This is the depressing part. It included exactly 0% of the original design. That’s right. Not a single element I’d designed made it into the first version of the product. Unfortunately, because all the design work had been done up front, I had no idea why the company made the decisions they had made. Of course, if I had been involved with the project all the way through, I could have stripped down the design in a way that at least tried to stay true to the original vision of the product. Instead, it came across as something that had never really been designed at all.

Oddly, there’s a part of me that’s happy that they did what they did. After all, they ended up releasing a MUCH smaller version of the product than we had originally envisioned. I’m not sure if it’s a Minimum Viable Product – only time will tell if it’s viable, after all – but it has many fewer features than we originally envisioned.

But, in another way, I’m sad. If I could have convinced them to go this small at the very beginning of the project, I could have designed their Minimum Viable Product in a lot less time, and, I think, with a lot better result than they got doing it on their own. More importantly, we could then have worked together to build out new features based on learning from the MVP, and we probably still would have had some design budget left to eventually implement that gorgeous visual design.

I guess my lesson from all of this is that I need to be stricter with companies who want a giant, fully fleshed-out product all designed for them at the beginning of the process. I need to be clearer that they’d be better off designing something very small and immediately implementable and then working from there. And I need to encourage them to spread the design budget out over a longer period of time so that they are never left without design resources. Either that, or I need to let go of my designs once they’ve been handed over to the client and acknowledge that they get to do (or not do) anything they want with them. I just find that really depressing.

Why Your Customer Feedback is Useless

Here’s the scenario: You have a minimum viable product. You’re talking to your users about it. You’re asking them questions, and they’re answering. But for some reason, it’s just not turning into usable information.

You wonder what’s going on. You imagine that perhapsyour users suck at giving good feedback or else they don’t have anything useful to say. Maybe, you think in a moment of hopeful delusion, your product is so perfect that it can’t be improved by customer feedback.

While these are all possibilities, the reality is that it’s probably not your customers’ fault. So, if you don’t seem to be getting any good data, what IS the problem? Probably one of the following things:

You’re asking the wrong questions

I’ve written before about asking customers the wrong questions, but in summary, customers are very good at giving you certain kinds of information and very bad at other kinds. For example, users are great at telling you about their problems. They can very easily tell you when something isn’t working or interesting or fun to use. What they suck at is telling you how to fix it.

Customers are great at:

• Complaining about problems
• Describing how they currently perform tasks
• Saying whether or not they like a product
• Showing you parts of a product that are particularly confusing
• Comparing one product to another similar product
• Explaining why they chose a particular method of doing something

Customers are bad at:

• Predicting their future behavior
• Predicting what other people will like
• Predicting whether they’ll pay for something
• Coming up with innovative solutions to their own or other people’s problems
• Coming up with brand new ideas for what would make a product more appealing

To take advantages of users’ strengths, have them describe things like, “Tell me about your most recent experience using the product and how that went for you.” Or ask them questions like, “What about [competitor product] do you particularly enjoy? What do you hate?” You can even ask questions like, “Of the following 5 features, which would you prefer?” You’ll want to be a lot more careful about listening to their answers to overly broad questions like, “What brand new feature would you like to see implemented?” or “What would make this product more fun to use?”