Entrepreneurs, please stop accosting people in coffee shops.
I know that idiots like me have been telling you about the wonders of guerrilla user research. Some of us may even have included it in our books. Apparently, we were not clear enough about when testing something in a coffee shop is a reasonable option, since many of you have decided to do this to the exclusion of absolutely everything else.
Stop accosting people in coffee shops. Guerrilla research can hurt your product. — Tweet This
What Is Guerrilla Usability Testing?
Let’s do a very quick recap. Guerrilla Research is what we call a very specific form of usability testing. It involves, at a high level, hanging out in public places and asking people to use your product in exchange for coffee.
What’s It Good For?
Guerrilla Usability is a fast and cheap way of getting a certain type of feedback on one type of product.
It can be very effective if all of the following criteria are met:
- you have a consumer-oriented product that does not require any specialized knowledge to use
- you want to know if people who have never seen your product before can understand your value proposition immediately
- you want to know if brand new users can perform a single, very specific task using your product
What Does It Suck At?
Everything else.
But Why Can’t You Use It for…?
The worst abuse of this research methodology is when entrepreneurs use it to determine whether people WANT to use their product. This is horrific on so many levels.
Getting 20 people at a random Starbucks to smile politely at you while you pitch them your idea does not constitute validation. People are generally polite, and most of them will nod encouragingly and agree that your product is probably fantastic in exchange for a $5 Frappuccino. Even if they didn’t lie just to get you to go away, they’d still be incapable of telling you whether or not they would use your product, since people are really bad at telling the future.
A major problem with trying to figure out if people will love your product based on this sort of testing is that, even if your product is aimed at “everyone” (and that’s own problem, really), it’s not necessarily aimed at the people you’ll find in any given coffee shop. And even if it’s specifically for people in coffee shops, there’s no saying whether or not those people would have found or understood or used your product on their own, if you hadn’t been standing there offering it to them.
Just trust me on this. You will never find out if people like your product by talking to strangers in coffee shops. Stop it. You’re wasting your time.
You will never find out if people like your product by talking to strangers in coffee shops. — Tweet This
Also, you won’t find out if it’s usable or useful to anybody beyond the first five minutes of usage, since that’s all the time you’re likely to have. If you have a product that is only useful with repeated usage, coffee shop testing isn’t going to help you. All you get to see is the onboarding process, so if you’re ignoring other types of longer form research (diary studies, current user observations, etc.), you’re probably going to end up with a really well tested onboarding process and a really confusing product.
Finally, if your product is designed to be used by a specific type of user, or even a general consumer in a specific type of context, you’re screwed if you’re only doing guerrilla research. For example, if your product is for accountants, it’s very likely to have loads of content that wouldn’t be at all understandable to an average person on the street. That’s ok!
Not every product needs to be perfectly understandable to every person. We don’t design airplane cockpits so that just anybody can walk in and fly a plane. It’s not a reasonable expectation. But if you’re getting feedback from non-accountants or non-pilots on your accounting software or your cockpit design, you’re going to get suggestions that will probably make the product worse for your actual users.
The same goes for context of use. Something that might seem perfectly usable in a Starbucks with decent lighting, a good wifi connection, and a flat surface might not be that great if the product is meant to be used one-handed on a crowded bus or when sprinting through an airport or while driving a car. Context can have an enormous impact on the usability of a product.
What Can You Do Instead?
I’m not actually saying you can never use coffee shop testing. It’s a reasonable tool for a very limited set of research objectives.
But branch out a bit. Figure out what you want to learn, and then do the work to understand the best method for learning that. As an added bonus, you’ll stop getting kicked out of all those coffee shops.
Like the post? Read the book. UX for Lean Startups.