7 User Research Myths and Mistakes

I wrote a post over at O'Reilly's Radar blog about a few of the myths and mistakes surrounding quantitative and qualitative research.

I can’t tell you how often I hear things from engineers like, “Oh, we don’t have to do user testing. We’ve got metrics.” Of course, you can almost forgive them when the designers are busy saying things like, “Why would we A/B test this new design? We know it’s better!”

In the debate over whether to use qualitative or quantitative research methods, there is plenty of wrong to go around. So, let’s look at some of the myths surrounding qualitative and quantitative research, and the most common mistakes people make when trying to use them.

Read More at O'Reilly >

Or sign up for the Webcast on March 11th, 2015 to learn about combining qualitative and quantitative research more effectively. 

Stop Accosting People in Coffee Shops - Use Guerrilla Research Wisely

Entrepreneurs, please stop accosting people in coffee shops.

I know that idiots like me have been telling you about the wonders of guerrilla user research. Some of us may even have included it in our books. Apparently, we were not clear enough about when testing something in a coffee shop is a reasonable option, since many of you have decided to do this to the exclusion of absolutely everything else.

Stop accosting people in coffee shops. Guerrilla research can hurt your product. — Tweet This

What Is Guerrilla Usability Testing?

Let’s do a very quick recap. Guerrilla Research is what we call a very specific form of usability testing. It involves, at a high level, hanging out in public places and asking people to use your product in exchange for coffee.

What’s It Good For?

Guerrilla Usability is a fast and cheap way of getting a certain type of feedback on one type of product.

It can be very effective if all of the following criteria are met:

• you have a consumer-oriented product that does not require any specialized knowledge to use
• you want to know if people who have never seen your product before can understand your value proposition immediately
• you want to know if brand new users can perform a single, very specific task using your product

What Does It Suck At?

Everything else.

But Why Can’t You Use It for…?

The worst abuse of this research methodology is when entrepreneurs use it to determine whether people WANT to use their product. This is horrific on so many levels.

Getting 20 people at a random Starbucks to smile politely at you while you pitch them your idea does not constitute validation. People are generally polite, and most of them will nod encouragingly and agree that your product is probably fantastic in exchange for a $5 Frappuccino. Even if they didn’t lie just to get you to go away, they’d still be incapable of telling you whether or not they would use your product, since people are really bad at telling the future.

A major problem with trying to figure out if people will love your product based on this sort of testing is that, even if your product is aimed at “everyone” (and that’s own problem, really), it’s not necessarily aimed at the people you’ll find in any given coffee shop. And even if it’s specifically for people in coffee shops, there’s no saying whether or not those people would have found or understood or used your product on their own, if you hadn’t been standing there offering it to them.

Just trust me on this. You will never find out if people like your product by talking to strangers in coffee shops. Stop it. You’re wasting your time.

You will never find out if people like your product by talking to strangers in coffee shops. — Tweet This

Also, you won’t find out if it’s usable or useful to anybody beyond the first five minutes of usage, since that’s all the time you’re likely to have. If you have a product that is only useful with repeated usage, coffee shop testing isn’t going to help you. All you get to see is the onboarding process, so if you’re ignoring other types of longer form research (diary studies, current user observations, etc.), you’re probably going to end up with a really well tested onboarding process and a really confusing product.

Finally, if your product is designed to be used by a specific type of user, or even a general consumer in a specific type of context, you’re screwed if you’re only doing guerrilla research. For example, if your product is for accountants, it’s very likely to have loads of content that wouldn’t be at all understandable to an average person on the street. That’s ok!

Not every product needs to be perfectly understandable to every person. We don’t design airplane cockpits so that just anybody can walk in and fly a plane. It’s not a reasonable expectation. But if you’re getting feedback from non-accountants or non-pilots on your accounting software or your cockpit design, you’re going to get suggestions that will probably make the product worse for your actual users.

The same goes for context of use. Something that might seem perfectly usable in a Starbucks with decent lighting, a good wifi connection, and a flat surface might not be that great if the product is meant to be used one-handed on a crowded bus or when sprinting through an airport or while driving a car. Context can have an enormous impact on the usability of a product.

What Can You Do Instead?

I’m not actually saying you can never use coffee shop testing. It’s a reasonable tool for a very limited set of research objectives.

But branch out a bit. Figure out what you want to learn, and then do the work to understand the best method for learning that. As an added bonus, you’ll stop getting kicked out of all those coffee shops.

Like the post? Read the book. UX for Lean Startups.

Your Job Is Not to Write Code

Dear Engineers,

Your job is not to write code.

I know. You think you were hired to write code. In fact, your entire interview process centered around how well you could write code. And I’m sure you do it really well.

But it’s not your job.

Your job is to improve our product for our users. If you want to get technical about it, your job is to improve our product for our users in a way that improves the key metrics of the company. But honestly, you don’t always have a lot of control over that second bit. You do, however, have an enormous amount of control over the first bit!

Of course, if you want to do your job well, it does mean that you may have to change some of your current behaviors.

For one thing, you need to make sure that the code you write (writing code is still one of the main things you will do when doing your job, by the way) runs the way it should, even on users’ machines.

Did you know that our users probably don’t have brand new MacBook Airs with giant Thunderbolt monitors set at the highest resolution possible and running the latest version of Chrome? I checked. A lot of them have Internet Explorer on 4 year old laptops, so sometimes things you build don’t work right on their machines. They’re still our users, and it’s still your job to improve the product for them, so please make sure that the code you wrote works in a reasonable number of environments.

In fact, you’re going to need to make sure that the code you wrote runs in production, in general. I don’t really care if your code runs locally. If your code just runs locally, then my only option is to sell your computer so that our users can use our software, and that really doesn’t scale.

So, to avoid that, you need to check your changes in production. Every time. Remember, your job is not just to ship something. It’s to ship something that improves our product for our customers. You can’t know it will do that unless you check that it runs in the way it’s supposed to.

Of course, in order to check your changes in production, you’re going to need to make sure that your code actually gets merged and pushed into production. I mean, you can’t really check your changes in production if you just let them sit unpushed for hours or days. Push your code. Get it into production. Then run it and check it.

This is obviously harder to do if you’re in an environment where you can’t do continuous deployment, but the theory still holds. When your code gets into production, whenever that is, you’re still responsible for it. Make sure that it’s doing what it ought to be doing - which is make the product better for users.

Another thing to remember is that sometimes users do surprising things, which means that it’s not enough just to test that your code works under perfect conditions. You need to make sure that it does something reasonable even in error cases and zero data states and when the user does something you might not expect, like use the back button or make two accounts by mistake.

This is hard. It means you’ll have to spend time thinking about the different things our users might do. But it’s an important part of your job, because it will vastly improve the product for our users if they aren’t constantly finding bugs or edge cases or dead ends.

There’s one more important part to your job. You need to make sure that we can measure whether we’re all doing our jobs well. That means adding metrics and analytics so that we can test the effects of our changes. If you expect the code you are writing to improve user engagement (by improving the user experience in some key way), then you need to have a way to learn whether or not you succeeded. How else will you know if your job is done? Because, as I’ve mentioned, your job isn’t done until you’ve improved the product for our users.

I know what you’re thinking. This will all take so long! I’ll be so much less effective!

This isn’t true. You’ll be far more effective because you will actually be doing your job. If you get hassled for writing less code, that’s a failure of management, and I apologize for it. We need to spend less time demanding that you write features and more time asking you to improve our product for our users. If we’re not doing that, I strongly suggest you ask us to. If we still refuse, you should leave and find an environment that lets you do your job. Which, not to beat a dead horse, is to make the product better for our users.

Please don’t feel like I’m picking on you. You’re not the only one who should be doing this job. It is all of our jobs to make the product better for our users. It is my job as a PM and UX Designer and Manager to understand our users well enough that I can help you know how to improve the product for them. It is the CEO’s job to find a strategy that allows us to make money by improving the product for our users.

No matter what our job titles, our jobs are all the same — to make the product better for our users. Every day. So let’s do that.

Thank you,

Your Product Manager

Dear Engineers, your job is not to write code! - Tweet This

*This post originally appeared on Medium*

What Happens Next?

Today I’d like to talk about why that feature you’re building has taken twice as long to build as you thought it would and why it will be hard to use once you’ve shipped it. The problem is that you didn’t really think it through before you started coding.

Before I jump in, I want to point out that this is not, in fact, an argument in favor of big, up front design, or an argument against Agile or Lean. Please believe that the technique I’m sharing here can very easily be used in any sort of environment.

When you create an interaction for a product, you have to design more than what it looks like. You even have to design more than what happens during the interaction. You have to design what happens after the initial user interaction. And then you have to keep going.

Design what happens after the initial interaction. Then keep going. — Tweet This

Let’s take a look at an example. Let’s imagine that you have an e-commerce product that sells children’s books. You encourage readers to post comments and ratings. It’s a new product, so everything is very bare bones. You didn’t spend a lot of time building a huge commenting system before you knew if anybody would leave comments. Instead, you just made something very simple where people can leave a comment or read a comment. Well done! That’s very Lean of you.

Then one day, you have to have this conversation with your designer:

You: Oh my God. People are awful!

Designer: Right?

You: Someone posted an ad for penis enhancement on the Where the Wild Things Are page, and I didn’t see it until someone sent a customer support email. It could have been there for days.

Designer: We should probably have an easier way for users to flag inappropriate content than sending us an email.

You: I guess we should. Can you add flagging to the comments?

Designer: Sure!

A little while later, the designer claims he’s added flagging to the design and is ready for engineering to implement it. The design looks like this.

“Well, that’s easy,” you think. “I’m just a few minutes away from users being able to tell me when a comment is inappropriate.”

Then, of course, the design goes to the engineer. And the conversation goes like this.

Engineer: What happens when a user clicks the flag link?

You: Well, the comment gets flagged.

Engineer: Ok, I’ll set a flag in the database whenever a user flags a comment.

Hopefully you can see where this is going. The upshot is that, when a user clicks the flag link, the database will record that fact and…well…nothing else. The user who clicked the flag link won’t get any feedback. Nobody will be notified that something was flagged. You can’t do anything with the user report about the comment.

In other words, nothing has been planned beyond the visual implementation and the initial user interaction. And this is all happening because you only designed the initial user interaction.

So, what really needs to happen for this feature to be considered usable and useful? What is the minimum set of subsequent interactions that need to be designed after the user clicks on the flag link?

Probably something like this:

• The user who flags a comment should get a confirmation that the comment was successfully flagged and be given some sort of information about what will happen next. 
• Someone at the company should be notified in some way that a comment has been flagged. 
• Someone at the company should be able to view the flagged comment and probably the identity of the person who flagged the comment. 
• Someone at the company should be able to remove the flagged comment. 
• The person who left the flagged comment should be notified that their comment was removed.

“My God!” you say. “My little, tiny feature has ballooned into an enormous undertaking! This is feature creep!”

What really needs to happen for a feature to be both usable and useful? — Tweet This

Well, no. It isn’t. You see, these are all things that would have to happen just to make the feature work at all. If you allow people to flag comments but don’t have any way for someone at your company to deal with the flagged comments, you’ve designed half a feature. Or, more to the point, you’ve designed a broken, unusable feature with a dead end.

Of course, that’s just the minimum set of things that need to be designed. There are all sorts of optional pieces like letting the flagger know when the issue is resolved or letting the flagger indicate why she’s flagging the comment or letting a customer service rep know how often people have been flagged so they can decide whether to ban the person. That last one would mean that you’d need some affordance for banning users and preventing them from returning.

Now, if you’re being very Lean Startup, you can always build the customer facing side first and decide not to build the admin side until people start flagging comments. In that case, when someone flagged a comment, you could simply have an engineer go into the database and remove the comment if you deemed it offensive.

That absolutely counts as designing a solution to the problem, by the way. It’s just a very manual design, and it’s probably one that won’t scale very well.

Why Should You Bother? 

Now you’re probably thinking that this all sounds exhausting and wondering why you would bother thinking through all of this stuff. That’s understandable. It can be exhausting. Do it anyway.

You see, thinking through the entire flow of interactions after the first one has a huge benefit to both your product and your product development process.

It helps your product by making sure you don’t have a lot of dead ends and awkward experiences. You don’t, for example, have a flagging feature that doesn’t result in flagged comments being pulled from your pages. This goes a long way toward making your overall user experience about a thousand percent better than the majority of products I use on a daily basis.

This process also improves your product development by helping you to understand the true complexity of a feature before you build it. You will never know exactly how long something will take to build, but you can certainly tell that building a flagging feature that works will take a lot longer than just putting a flag icon on each comment.

By fully exploring the complexity of features, you are better prepared to prioritize your backlog.

So, How Do You Do It? 

It’s pretty simple. Instead of thinking of your feature, think of the end user’s intent. Then, every time there’s some sort of interaction, you need to ask yourself, “has the user’s intent been satisfied?” If not, you’re not done, so you need to ask yourself, “What next?”

For example, your user isn’t going to flag something because she just loves flags. She’s going to flag something because she wants an offensive comment removed from a product page. Let’s go through the interaction sequence until that intent has been satisfied.

1. A user can click the flag icon. Does this result in the comment being removed? No? Ok, what next?
2. An email gets sent to someone at the company. Does this result in the comment being removed? No? Ok, what next?
3. The person at the company asks an engineer to remove the comment from the database (which the engineer has agreed to do). Does this result in the comment being removed? Yes? Great! You’re done (for now).

Just keep asking yourself what’s next until the user’s intent has been met. It’s the best way to guarantee that your features do what you think they do and what your users expect them to do.

Like the post? Follow me on Twitter. 

Also, check out the book for more product and UX tips — UX for Lean Startups.

Want Better UX? Change the Conversation.

If you’ve ever been a user experience designer, you’ve probably heard people say something like this when starting a new project:

• We want to make it delightful and easy to use.
• We need to do some user research.
• We want to improve our onboarding process.
• We think it needs a walkthrough for new users.
• We want a persona/photoshop mockup/wireframe/landing page/insert deliverable here.

All of these statements are absolutely useless. Why? Because none of them help you decide what to work on or how to improve a product.

So, the next time somebody introduces a UX project by asking for a specific deliverable or by giving vague instructions to “make it better,” you need to change the conversation.

You can do that by asking the following questions:

• Who is the target user for this product or feature?
• What problem are you trying to solve for those users?
• What business need are you trying to fulfill with this project?
• What metric are you trying to move?
• Why do you think that solving this particular user problem will move that metric?

I know you want it to be delightful. But what metric does it move? — Tweet This

How Do These Questions Help Users?

The most important thing about these questions is that they help you define three things that are critical to a successful design project:

1. The problem you are trying to solve
2. The reason you are trying to solve it
3. The way you will know if you’ve succeeded

Of course you want a better design. How will you know if you’ve succeeded? — Tweet This

Design without these elements isn’t really user experience design. It’s just drawing pictures. Design is about solving real problems, both for users and for the business. In fact, at its best, design is about solving problems for the business (for example, generating revenue or improving retention) by solving problems for the user (for example, offering something somebody wants to buy or helping make their lives better).

Design should solve problems for your business by solving problems for your user. — Tweet This 

By knowing the answer to these questions, you are far more likely to build a product that users want to use and that improves key metrics for your company.

How Do These Questions Help Designers?

Answering these questions can be incredibly helpful for individual designers. When we ask these questions, we reframe the project to give the designer far more freedom to solve problems, which is, after all, the fun part of the job.

Instead of being told “change the onboarding flow” or “create a tutorial walkthrough,” we get asked to “improve the 10 day activation metric for new users.”

As designers, we get to create our own hypotheses about how we will improve that metric rather than simply implementing someone else’s vision.

More importantly, we can understand when our designs were successful, because we have a specific metric against which we can measure our results. This kind of direct feedback can make us better designers.

How Do These Questions Help Engineers?

These questions can be incredibly useful for defining the scope of a project, which has a very real impact on engineering. For example, poorly defined projects are particularly susceptible to scope creep.

After all, if you don’t have a very solid idea of the problem you’re trying to solve or the metric you’re trying to move, it’s very easy to justify adding “just one more thing.” But when you have a clearly defined problem, it’s easy to push back on new feature requests that don’t contribute directly to solving that problem.

What to Do If They Can’t Answer Those Questions

The first few times you try to change the conversation, you may get push back. You’ll get clients or product managers or engineers who simply can’t answer these questions. Keep asking them.

If people can’t answer the questions, you need to help them get the answers before you start work on the project. Otherwise, you’re shortchanging your users, your company, your team, and yourself.

Like the post? Follow me on Twitter!

The Most Important User You're Not Talking To

Do you have a product? With users? 

If you answered “yes” to both of those questions, you have an amazing untapped source for product research. And I’m not talking about your users. 

I mean, sure, you should be listening to users and observing them. A lot. But there’s another group of people who can provide you with incredible insights into your product. 

You should be talking to people who used your product once and then abandoned it. Tweet This!

Specifically, you need to ask these people the following questions:

• What were you expecting when you tried this product?
• How did it not meet your expectations? 

This research will help you understand three things very clearly:

• What your messaging and acquisition strategy is telling people to expect.
• What problem the people you are acquiring are trying to solve.
• Why your product doesn’t solve this problem for the people you are acquiring. 

You’ll notice that I mentioned “acquisition” in each of the above points. This is intentional. You see, one of the things you are very likely to find out from this sort of research is that you are getting entirely the wrong group of people to try out your product. 

If you’ve been spending a lot of time optimizing your ads and your messaging for sign up conversion rather than for actual product usage and retention, it may turn out that you are acquiring a whole lot of the wrong sort of user for your product, which can be a costly mistake. This kind of research is fabulous for understanding if that’s true. 

The other thing that this research helps with is understanding whether or not you’re adequately solving the problem you think you’re solving in a way that users can understand. If new users can’t figure it out what your product does and how to do it in a few seconds, they’ll leave without ever knowing that your product was the solution to their problem. 

Of course, this isn’t the easiest group of people to interview. These folks can be tricky to track down and tough to schedule. But finding a way to interview people who thought they wanted to use your product and then changed their minds is something that will pay off hugely in the long run.

Making More UX Designers

Over the last few years, I’ve had an increasing number of people ask the same two questions. Specifically I get asked:

Where can I find a good UX designer?

How can I get into UX?

The best possible solution is, of course, to teach the people in the second group how to do the job and then introduce them to the people in the first group. The second best solution is to teach the people in the first group to do it for themselves. I’ve been experimenting lately with both of these approaches. 

This need for creating more UX designers is one of the biggest reasons I joined Tradecraft as an instructor at the beginning of the year. My co-teacher, the amazing Kate Rutter, and I each spend 3 days a week working with smart, motivated, tech-savvy students, teaching them UX design fundamentals like user research, task flows, personas, wireframing, and prototyping. Most importantly, we teach them to think like UX designers. 

Because it’s an intensive 12 week program, students have time to learn UX skills and apply them on real projects for real companies. They also learn from frequent guest mentors and speakers. 

It’s a competitive program. We don’t take many students. We’re not interested in churning out a lot of mediocre designers. We want to take a few people each quarter and turn them into great designers whom we’d be happy to recommend for jobs. We prefer people who have experience in some area of design, product management, user research, or engineering. 

So, if you’re someone who desperately wants to become a UX designer, and you want hands-on coaching from a couple of people who’ve been doing this for quite a few years, you should apply to Tradecraft for the next quarter. Or, if you’re a manager at a larger company, and you have a promising UX designer or Product Manager who needs some serious coaching to get to the next level, you should consider sponsoring that employee in the program. Lastly, if you’re looking for a newly minted UX designer, we’ve got a few of those graduating at the end of March. 

And if you want more information about any of it, you should email me at laura@usersknow.com and ask.

By the way, Tradecraft also has programs for people who want to be Growth Hackers and Sales People.