Improving the ROI for Your User Research

This post originally appeared on the Sliced Bread Design blog.

So, you decided to do some user research in order to find out where you can make improvements. After a few hours of user interviews, you ended up with a notebook full of scribbled information that all seemed really critical. How in the world do you figure out what to do with all that information?

If your answer is “talk about it all abstractly with everybody in the company or write a huge paper that nobody will read and then go on with business as usual,” you're in good (bad?) company.

But you have to DO something with all that data. You have to analyze it and turn it into actionable items that your engineering department can use to fix your product. It's not always easy, but I'm going to give you an approach that should make it a little easier. This isn't the only way to do your test analysis, but it's one of the quickest and easiest that I've found when you are concerned with key metrics.

When to use this method:

• You have an existing product with a way to measure key metrics, and you’re interested in improving in particular areas related to your bottom line
• You have a limited research and development budget and want to target your changes specifically to move key metrics
• You are looking for the “low hanging fruit” that is getting in the way of your users performing important tasks with your product
• You are working in an agile development environment that is constantly tweaking and improving your product and then testing the changes

When not to use this method:

• You have an existing product that you are planning to completely overhaul, and you want to understand all of the major problems before you do your redesign
• You are trying to create an overall awesome, irresistible user experience that is not related to a specific metric
• You are designing a new product or feature and are observing people using other products to identify opportunities for innovation

If you fall into the first bucket, read on…

The Five Basic Steps:

• Identify key metrics you'd like to improve
• Identify the tasks on your site that correlate with improvement in those metrics
• Observe people performing the appropriate tasks
• Identify the barriers preventing people from completing or repeating the tasks
• Develop recommendations that address each specific barrier to task completion

Why I Hate Paper Prototypes

This post originally appeared on the Sliced Bread Design blog.

Ok, maybe hate is a little strong. Paper prototypes and sketches have their place in interaction design. For example, they're great for helping to quickly brainstorm various different approaches to a problem at the beginning of a design process. They're also a very fast and cheap way to illustrate a new idea, since most people can draw boxes faster than they can build interactive prototypes. But, in my opinion, they have several serious drawbacks.

Before I get too far into this, let me define what I mean by a paper prototype, since I've heard people use the term to refer to everything from sketches on actual pieces of paper (or cocktail napkins in a couple of cases) to full color printed mockups with a polished visual design. In this instance, I'm referring to a totally non-interactive screen, mockup, or sketch of any sort of application that is meant to be shared with customers, test participants, or team members. It can be printed on actual paper or shown on a computer screen, but whatever the viewer does to it, a paper prototype is not interactive.

So, what don't I like about them?

Screen vs. Paper

This first couple of peeves apply to screens that are actually printed out or drawn directly on paper. With a few exceptions that I've listed below, I've found this approach to be really counterproductive.

Iterating On a Design

One of the biggest problems with hand drawn sketches on paper has less to do with user interactions and more to do with my work flow as a designer. Sure, sketching something quickly on a piece of paper can be quick, but what happens when I realize that I want to swap two sections of the screen? I can draw arrows and lines all over it, but that gets messy pretty fast. Whenever I want to make any changes to my design, I need to create a whole new sketch. This can mean redrawing the entire screen quite a few times.

If I'm creating a design in HTML or any other prototyping tool, the very first version might take a little longer than a quick sketch, but the second through nth iterations are a whole lot faster. And, as a bonus, I can check them into source control, which means I'm a lot less likely to lose my work than if I have dozens of pieces of paper scattered all over my office.

Interacting With Paper

Whether they're sketched out by hand or printed out on paper, people interact with paper screens differently than they do with computer screens. They view them at a different angle. They focus on different parts of the screen. They use their hands to interact with them rather than a mouse and keyboard. Any feedback that you get on a printed design will be colored by the fact that people are fundamentally interacting with it differently than they would if it were on a computer screen.

Given all of these drawbacks, there are a few situations when designs printed on paper can be used effectively:

• You are at the very beginning of the design process, and you want to explore a bunch of different possible directions with other team members very quickly before committing yourself to fleshing out one or two specific options.
• You're designing material that is meant to be printed, like brochures, user manuals, books, etc. In this case, you want to know how people will interact with the printed media.
• Your product is an interface for some sort of embedded or small screen device that would be very difficult to replicate in a quick interactive prototype. For example, a screen for certain mobile devices or the heads-up display for a car dashboard might be hard to show interactively in the appropriate context.
• You have several different visual designs, and you'd like to show them all to users at the same time in order to see which one is the most attention-getting. You’ll still need to show the designs on screen, of course, since colors can vary so much between screen and print, but it can be helpful to lay out several pieces of paper so that the various options can easily be compared.
• You need to share screens with people in an environment with absolutely no access to a computer whatsoever. You know, maybe you’re in the middle of a meeting and need to sketch something quickly, or the rest of your design team is Amish, or you are designing in a post-apocalyptic wasteland where the computers are trying destroy humanity.

On the other hand, if you're designing desktop or web applications for standard computers, at the very least, show your prototypes on a computer, even if they are not interactive!

6 Stupid Excuses for Not Getting Feedback

This post originally appeared on the Sliced Bread Design blog.

Almost every company I talk to wants to test their products, get customer feedback, and iterate based on real user metrics, but all too often they have some excuse for why they just never get around to it. Despite people's best intentions, products constantly get released with little to no customer feedback until it's too late.

I'm not trying to promote any specific methodology for testing your products or getting customer feedback. Whether you're doing formal usability testing, contextual inquiries, surveys, a/b testing, or just calling up users to chat, you should be staying in contact with customers and potential customers throughout the entire design and development process.

To help get you to stop avoiding it, I've explored six of the most common stupid excuses for not testing your designs and getting feedback early.

Excuse 1: It's a design standard

You can't test every little change you make, right? Can't you sometimes just rely on good design practices and standards? Maybe you moved a button or changed some text. But the problem is, sometimes design standards can get in the way of accomplishing your business goals.

For example, a few months ago at a talk given by Bill Scott, he talked about a developer who had a/b tested the text on a link. One option read, "I'm now on Twitter." The second read, "Follow me on Twitter." The third read, "Click here to follow me on Twitter." Now, anybody familiar with "good design practices" will tell you that you should never, ever use the words "click here" to get somebody to click here. It's SO Web 1.0. But guess which link converted best in the a/b test? That's right. "Click here" generated significantly more Twitter followers than the other two. If that was the business goal, the bad design principle won hands down.

Does this mean that you have to do a full scale usability test every time you change link text? Of course not. Does it mean you have to use the dreaded words "click here" in all your links? Nope. What it does mean is that you should have some way to keep an eye on the metrics you care about for your site, and you should be testing how your design changes affect customer behavior, even when your changes adhere to all the best practices of good design. So, to put it simply: prioritize what you care about and then make sure you test your top priorities.

Excuse 2: Company X does it this way

I can't tell you how many times I've heard, "Oh, we know that will work. Google/Facebook/Apple does it that way." This is the worst kind of cargo cult mentality. While it's true that Google, Facebook, and Apple are all very successful companies, you aren't solving exactly the same problem that those companies are, you don't have exactly the same customers that they do, and you don’t know if they have tested their designs or even care about design in that particular area. You are, hopefully, building an entirely different product, even if it may have some of the same features or a similar set of users.

Is it ok to get design ideas from successful companies? Of course it is. But you still need to make sure your solutions work for your customers.

I previously worked with a company that had a social networking product. Before I joined them, the company decided that, since other companies had had good luck with showing friend updates, they would implement a similar feature, alerting users when their friends updated their profiles or bought products. Unfortunately, the company's users weren't very interested in the updates feature as it was implemented. When we finally asked them why they weren't using the feature, the users told us that they would have been very interested in receiving an entirely different type of update. Of course, if the company had connected with users earlier in the process, they would have rolled the feature out with the right information and gotten a much more positive reaction on launch.

Another thing to remember is that just because a company is successful and has a particular feature doesn't mean it's that exact feature that makes them successful. Google has admitted that the "I'm Feeling Lucky" button loses them page views, but they keep it because they, and their customers, like the feature. That doesn't mean it's a good business plan for your budding search engine startup to adopt a strategy of only providing people with the equivalent of the "I'm Feeling Lucky" button. In fact, this is a great example of why you might need to employ multiple testing methods: qualitative (usability, contextual inquiry, surveys), to find out if users find the feature compelling and usable, and quantitative (a/b, analytics), to make sure that the feature doesn't bankrupt you.

The bottom line is, it doesn't matter if something works for another company. If it’s a core interaction that might impact your business or customer behavior, you need to test new features and designs with your customers to make sure that they work for you. Obviously, you also need to make sure that you’re not violating anybody’s IP, but that’s another blog post.