7 User Research Myths and Mistakes

I wrote a post over at O'Reilly's Radar blog about a few of the myths and mistakes surrounding quantitative and qualitative research.

I can’t tell you how often I hear things from engineers like, “Oh, we don’t have to do user testing. We’ve got metrics.” Of course, you can almost forgive them when the designers are busy saying things like, “Why would we A/B test this new design? We know it’s better!”

In the debate over whether to use qualitative or quantitative research methods, there is plenty of wrong to go around. So, let’s look at some of the myths surrounding qualitative and quantitative research, and the most common mistakes people make when trying to use them.

Read More at O'Reilly >

Or sign up for the Webcast on March 11th, 2015 to learn about combining qualitative and quantitative research more effectively. 

Stop Accosting People in Coffee Shops - Use Guerrilla Research Wisely

Entrepreneurs, please stop accosting people in coffee shops.

I know that idiots like me have been telling you about the wonders of guerrilla user research. Some of us may even have included it in our books. Apparently, we were not clear enough about when testing something in a coffee shop is a reasonable option, since many of you have decided to do this to the exclusion of absolutely everything else.

Stop accosting people in coffee shops. Guerrilla research can hurt your product. — Tweet This

What Is Guerrilla Usability Testing?

Let’s do a very quick recap. Guerrilla Research is what we call a very specific form of usability testing. It involves, at a high level, hanging out in public places and asking people to use your product in exchange for coffee.

What’s It Good For?

Guerrilla Usability is a fast and cheap way of getting a certain type of feedback on one type of product.

It can be very effective if all of the following criteria are met:

• you have a consumer-oriented product that does not require any specialized knowledge to use
• you want to know if people who have never seen your product before can understand your value proposition immediately
• you want to know if brand new users can perform a single, very specific task using your product

What Does It Suck At?

Everything else.

But Why Can’t You Use It for…?

The worst abuse of this research methodology is when entrepreneurs use it to determine whether people WANT to use their product. This is horrific on so many levels.

Getting 20 people at a random Starbucks to smile politely at you while you pitch them your idea does not constitute validation. People are generally polite, and most of them will nod encouragingly and agree that your product is probably fantastic in exchange for a $5 Frappuccino. Even if they didn’t lie just to get you to go away, they’d still be incapable of telling you whether or not they would use your product, since people are really bad at telling the future.

A major problem with trying to figure out if people will love your product based on this sort of testing is that, even if your product is aimed at “everyone” (and that’s own problem, really), it’s not necessarily aimed at the people you’ll find in any given coffee shop. And even if it’s specifically for people in coffee shops, there’s no saying whether or not those people would have found or understood or used your product on their own, if you hadn’t been standing there offering it to them.

Just trust me on this. You will never find out if people like your product by talking to strangers in coffee shops. Stop it. You’re wasting your time.

You will never find out if people like your product by talking to strangers in coffee shops. — Tweet This

Also, you won’t find out if it’s usable or useful to anybody beyond the first five minutes of usage, since that’s all the time you’re likely to have. If you have a product that is only useful with repeated usage, coffee shop testing isn’t going to help you. All you get to see is the onboarding process, so if you’re ignoring other types of longer form research (diary studies, current user observations, etc.), you’re probably going to end up with a really well tested onboarding process and a really confusing product.

Finally, if your product is designed to be used by a specific type of user, or even a general consumer in a specific type of context, you’re screwed if you’re only doing guerrilla research. For example, if your product is for accountants, it’s very likely to have loads of content that wouldn’t be at all understandable to an average person on the street. That’s ok!

Not every product needs to be perfectly understandable to every person. We don’t design airplane cockpits so that just anybody can walk in and fly a plane. It’s not a reasonable expectation. But if you’re getting feedback from non-accountants or non-pilots on your accounting software or your cockpit design, you’re going to get suggestions that will probably make the product worse for your actual users.

The same goes for context of use. Something that might seem perfectly usable in a Starbucks with decent lighting, a good wifi connection, and a flat surface might not be that great if the product is meant to be used one-handed on a crowded bus or when sprinting through an airport or while driving a car. Context can have an enormous impact on the usability of a product.

What Can You Do Instead?

I’m not actually saying you can never use coffee shop testing. It’s a reasonable tool for a very limited set of research objectives.

But branch out a bit. Figure out what you want to learn, and then do the work to understand the best method for learning that. As an added bonus, you’ll stop getting kicked out of all those coffee shops.

Like the post? Read the book. UX for Lean Startups.

Your Job Is Not to Write Code

Dear Engineers,

Your job is not to write code.

I know. You think you were hired to write code. In fact, your entire interview process centered around how well you could write code. And I’m sure you do it really well.

But it’s not your job.

Your job is to improve our product for our users. If you want to get technical about it, your job is to improve our product for our users in a way that improves the key metrics of the company. But honestly, you don’t always have a lot of control over that second bit. You do, however, have an enormous amount of control over the first bit!

Of course, if you want to do your job well, it does mean that you may have to change some of your current behaviors.

For one thing, you need to make sure that the code you write (writing code is still one of the main things you will do when doing your job, by the way) runs the way it should, even on users’ machines.

Did you know that our users probably don’t have brand new MacBook Airs with giant Thunderbolt monitors set at the highest resolution possible and running the latest version of Chrome? I checked. A lot of them have Internet Explorer on 4 year old laptops, so sometimes things you build don’t work right on their machines. They’re still our users, and it’s still your job to improve the product for them, so please make sure that the code you wrote works in a reasonable number of environments.

In fact, you’re going to need to make sure that the code you wrote runs in production, in general. I don’t really care if your code runs locally. If your code just runs locally, then my only option is to sell your computer so that our users can use our software, and that really doesn’t scale.

So, to avoid that, you need to check your changes in production. Every time. Remember, your job is not just to ship something. It’s to ship something that improves our product for our customers. You can’t know it will do that unless you check that it runs in the way it’s supposed to.

Of course, in order to check your changes in production, you’re going to need to make sure that your code actually gets merged and pushed into production. I mean, you can’t really check your changes in production if you just let them sit unpushed for hours or days. Push your code. Get it into production. Then run it and check it.

This is obviously harder to do if you’re in an environment where you can’t do continuous deployment, but the theory still holds. When your code gets into production, whenever that is, you’re still responsible for it. Make sure that it’s doing what it ought to be doing - which is make the product better for users.

Another thing to remember is that sometimes users do surprising things, which means that it’s not enough just to test that your code works under perfect conditions. You need to make sure that it does something reasonable even in error cases and zero data states and when the user does something you might not expect, like use the back button or make two accounts by mistake.

This is hard. It means you’ll have to spend time thinking about the different things our users might do. But it’s an important part of your job, because it will vastly improve the product for our users if they aren’t constantly finding bugs or edge cases or dead ends.

There’s one more important part to your job. You need to make sure that we can measure whether we’re all doing our jobs well. That means adding metrics and analytics so that we can test the effects of our changes. If you expect the code you are writing to improve user engagement (by improving the user experience in some key way), then you need to have a way to learn whether or not you succeeded. How else will you know if your job is done? Because, as I’ve mentioned, your job isn’t done until you’ve improved the product for our users.

I know what you’re thinking. This will all take so long! I’ll be so much less effective!

This isn’t true. You’ll be far more effective because you will actually be doing your job. If you get hassled for writing less code, that’s a failure of management, and I apologize for it. We need to spend less time demanding that you write features and more time asking you to improve our product for our users. If we’re not doing that, I strongly suggest you ask us to. If we still refuse, you should leave and find an environment that lets you do your job. Which, not to beat a dead horse, is to make the product better for our users.

Please don’t feel like I’m picking on you. You’re not the only one who should be doing this job. It is all of our jobs to make the product better for our users. It is my job as a PM and UX Designer and Manager to understand our users well enough that I can help you know how to improve the product for them. It is the CEO’s job to find a strategy that allows us to make money by improving the product for our users.

No matter what our job titles, our jobs are all the same — to make the product better for our users. Every day. So let’s do that.

Thank you,

Your Product Manager

Dear Engineers, your job is not to write code! - Tweet This

*This post originally appeared on Medium*