You Need to Redesign Your Product

Iterate, iterate, iterate. If there is something that I hear from lean startups more than “pivot” and “fail” it’s that iterating on your product is the way to improve it. And I absolutely believe that iteration is critical to making great features and products.

The problem is, sometimes just improving what you’ve got isn’t enough. Every so often, from a UX perspective anyway, you just need to throw everything out and start from scratch.

I’m not necessarily talking about reskinning the site with a new visual design, although that sometimes has to be done. Sometimes you also need to completely reorganize and refocus everything about your product’s user experience.

Of course, this can be incredibly expensive and time consuming, so it’s not something that you want to do unless it’s really necessary.

Here are a few signs that you may need to do a complete product redesign:

There’s No Room for Your New Feature

A big part of lean startups is coming up with lots of new feature ideas and throwing them in front of users to see what sticks. Another big part is killing the features that don’t make the cut, but that can be hard to do.

There comes a time in the life of every lean startup where they have a new feature that doesn’t quite fit within the navigation and structure of the rest of the product.

Often this new feature is not quite a pivot, but it may be the first step in that direction. Maybe you’re adding a social component to an ecommerce application or you’re adding games or a marketplace to a social site.

Or maybe you’ve just run out of room on your front page, and you simply can’t add another widget.

Whatever the reason, when you have a new feature that you can’t logically fit anywhere into your product, it’s probably time to do an overhaul, or at least a reorganization. It’s probably also a great time to go through and kill some of those underperforming features in order to make room for the new stuff.

You’ve Added a “Miscellaneous” Section to Your Navigation

Ever been tempted to add a section to your product navigation called “Misc.” or “Other” or “Stuff”? Yeah, we’ve all wanted to do it. DON’T.

The Importance of Predictability in Design

If you watch a few user tests, there’s an excellent chance that, at some point, the user will point at some element of the product and ask, “What does this do?” If I’m moderating the test, it is almost guaranteed that I will respond with “What would you expect it to do?”

I’m not trying to be difficult when I ask that question. I’m trying to learn the user’s expectations, because this is an important thing to understand when evaluating the usability of a product.

In my many years of watching and running usability tests, I’ve noticed a pattern of behavior for users encountering a new screen:

1. Scan quickly
2. Click the first call to action that seems remotely relevant to the task at hand
3. Declare the product confusing if that call to action doesn’t do what they expect
4. Leave

There are three very simple things that you can do right now to improve this experience in your product.

Calls to Action Should Be Explicit

Users should always know, or at least have a good idea of, what will happen if they click on a call to action. One way to do that is to make the copy or icon associated with the call to action really obvious and descriptive while still keeping it concise enough that people have a chance of reading it.

Recently, a company was testing landing pages. There were two pages, and the ONLY difference between the two was the copy on a button. One said “Login” and the other said “Login with Facebook.” Both buttons popped open the Facebook Connect dialog asking them to enter their Facebook information.

The percentage of people who clicked on the button was not statistically different. However, nearly twice as many people who clicked on the “Login with Facebook” button actually completed the step and ended up logging in.

Why is that? Because they weren’t surprised by the Facebook dialog. The people who simply expected to log in to the site were thrown by suddenly being asked to enter their Facebook credentials. The other group understood what was about to be asked of them, and they continued on.

Ironically, some people who might have been happy to login in with Facebook most likely didn’t want to click the plain Login button at all, since they didn’t want to create a whole new account or weren’t sure they could log in if they didn’t already have an account.

Things That Look the Same Should Act the Same

Have a More Info link in a couple of places in your product? Make sure that every time somebody clicks it, the same thing happens. If it’s an inline popup with some help text in one place, make sure it isn’t a link to a different screen in another place.

Can Something Be Too Easy?

I spend a huge amount of my time trying to make things easier to use. Most of the time, the problem with products isn’t the idea, it’s that the people who are meant to use them just get horribly confused.

If you’d asked me last year, I would have said there’s no way to make something too easy.

However, I’ve recently seen a couple of examples that reminded me that there are some surprising dangers to simplification.

It Can Piss Off Certain Users

I’ve worked with companies who have a small but passionate group of power users who are deeply involved with the product and the community. The products that gather this particular sort of following tend to have a lot of user generated content.

Now, when you first start out, your tools for users may be…well, a polite word would be “rough.” A better way of putting it would probably be “pieces of crap.” And yet, if your offering is compelling enough, you will end up with a group of people who care deeply enough to put up with all of your bugs and failings and complicated interfaces, and they will create something awesome in spite of all that.

Of course, they will bitch and whine about how hard they have it, but when you actually take steps to make your product too easy to use – for example, if noobs can all of a sudden use templates to create something that used to take power users hours or days – the early adopters who put all that time into your product will scream. By making it possible for anybody to recreate their hard work, you’ve devalued their contribution.

This is especially true in companies where users can make money or gain points from user generated content. You see, your early adopters want all those hours they put into learning the tools to be worth something. They still want to be the highest ranked on the leaderboard and to make the most cash from selling their products.

What do you do about it? Honestly? This is a great problem to have. It means that you’ve taken something that’s been proven to be fun to do and made it accessible to a whole lot more people. The most important thing is to try to get your community onboard with the changes before they happen so that you reduce the outcry.

You also want to make sure that the old guard still feels valued and useful. For example, enlist their help in creating templates. Get their feedback on adding power user features to the new tools. Give them credit for creating content without using the tools.

The Dangers of Metrics (Only) Driven Product Development

When I first started designing, it was a lot harder to know what I got right. Sure, we ran usability tests, and we looked generally at things like page counts and revenue before and after big redesigns, but it was still tough to know exactly what design changes were making the biggest difference. Everything changed once I started working with companies that made small, iterative design changes and a/b tested the results against specific metrics.

To be clear, not all the designers I know like working in this manner. After all, it's no fun being told that your big change was a failure because it didn't result in a statistically significant increase in revenue or retention. In fact, if you're a designer or a product owner and are required to improve certain metrics, it can sometimes be tempting to cheat a little.

This leads to a problem that I don't think we talk about enough: Metrics (Only) Driven Product Development.

What Is Metrics (Only) Driven Product Development?

Imagine that you work at a store, and your manager has noticed that when the store is busy, the store makes more money. The manager then tells you that your job is to make the store busier - that's your metric that you need to improve.

You have several options for improving your metric. You could:

• Improve the quality of the shopping experience so that people who are already in the store want to stay longer
• Offer more merchandise so that people find more things they want to buy
• Advertise widely to try to attract more people into the store
• Sell everything at half off
• Remove several cash registers in order to make checking out take longer, which should increase the number of people in the store at a time, since it will take people longer to get out
• Hire people to come hang out in the store

As you can see, all of the above would very likely improve the metric you were supposed to improve. They would all ensure that, for awhile at least, the store was quite busy. However, some are significantly better for the overall health of the store than others.

A Review of UserTesting.com

A lot of people recently have asked me about the new crop of user testing tools available on the internet. One specific tool that comes up a lot is usertesting.com, and I’d like to talk a little bit about my experience with it.

Frankly, when I first saw the site, my initial reaction was, “Well, that’s not going to be very useful.” I’ve heard a similar gut reaction from several user researchers. Once I’d given it a try, my reaction changed to, “Oh, shit. This could seriously cut into my income.”

Having used it several times now, I can happily say that neither of these reactions was correct.

The Cons:

Let’s start with the things that I originally noted about usertesting.com that made me think it wouldn’t be very useful.

There is no moderator.

Not having a moderator  means that there isn’t a human being running the test who can ask follow up questions and delve deeply into issues that come up naturally during a session.  Good moderators don’t just follow a script; they ask the right questions to really understand why users are doing what they’re doing.

It’s less useful for testing incomplete interactive prototypes.

I have not yet found a good way to test prototypes with usertesting.com. When I design, I create very sketchy, incomplete mockups of products. Typically, these only run in one browser, they have no visual design, and large parts of them won’t really work or will use fake data.

I can deal with all of these issues in a one-on-one session by explaining that the prototypes are not real, giving the participant some help in areas where the prototype isn’t perfect, or gently reminding them not to fixate on the visual design. This isn’t really possible without a human moderator. 

Pie-Jacking and Other Tips for Making Engineers Tolerate UX

Obviously, I find UX to be incredibly important. And these days, I’m finding more and more people who agree with me. Unfortunately, in many organizations, there are people who still feel that user research and interaction design slow things down or aren’t really necessary.

Sometimes, those people are engineers. Not always, of course. I’ve worked with lots of engineers who are very excited about the idea of good design and getting user feedback. But occasionally you run into groups of engineers who have yet to be convinced.

In the interest of achieving harmony between the Engineering and UX departments at your company, here are a few tips for convincing people (especially engineers) of the value of user research and design.

To be clear, I am in no way suggesting that you trick engineers or lie to them or manipulate them in any way. I’m working on the assumption that engineers are frequently extremely logical people who just need evidence that things are useful before they buy in.

Involve Them in the User Research

The number one way to get anybody excited about user research and UX is to get them involved with it. Too frequently, the only thing about user research that engineers get to see is a thirty page paper detailing all of the problems with their product. This is boring, painful, and easily ignored.

In my experience working with dozens of companies, the single most powerful tool for getting engineers in touch with users was having them sit in on a few usability sessions. The sessions didn’t have to be formal. I’d just put the engineers in chairs in the back of a small conference room and have a participant use the product in front of them.

Without fail, the engineers started to understand the pain that their users were feeling. And, since engineers are not monsters, they wanted to help those people. They would fix bugs they observed during the sessions. They would ask for advice on how to improve screens that they had previously thought were just fine.

Most importantly, the engineers would suddenly understand how different they were from their users! Suddenly, it became much harder for the engineers to believe that they had all the answers to their customers’ problems, since they had seen that they were really quite different.

6 Incredibly Important Lean UX Lessons

A project I’ve been working on recently should really be featured in all books on agile and lean design. I’ve found that many projects and clients have similar issues with their design processes, but it’s rare that a single project clearly demonstrates so many incredibly important tenets of lean UX. 

So far, over the course of about a month, this one has driven home all of the following lessons:

• You shouldn’t be a slave to your metrics, especially when they’re telling you something surprising
• Qualitative testing is necessary for understanding why your metrics are what they are
• Iteration in the design process is critical
• Very small changes can make a huge difference
• You need to understand the actual experience your user is having with your product
• Don’t give up on your vision too early!

So, what happened?

I’m working with a very cool startup that is absolutely devoted to metrics. They A/B test and measure everything. When they first brought me on, the team went over some history of the product.

They originally had a particular flow for the product, but when they did qualitative testing, it became clear to them that users expected things to behave differently. The team designed a new version of the product that they felt addressed the issues that testers were having. They then built and released the new version.

It bombed. Well, perhaps that’s a bit strong, but it performed significantly worse than the original design in an A/B test.

User Research Tips

After my talk at Web 2.0 Expo on combining qualitative research, quantitative metrics, and design vision for better products, there were some questions from the audience. Interestingly, the large majority of questions were about the qualitative research part of the talk.

And that makes sense. Qualitative research can be tough to incorporate into your development process. Until fairly recently, it's been a big, expensive, time-consuming endeavor. Often it required having outside consultants come in to run tests in a rented lab behind a one way mirror. Additionally, a lot of product folks assumed that it would slow down the development process, since it would often add a step between design and engineering.

Now, if you read my blog or listen to me speak, you know that I advocate quick and cheap testing over large, formal studies, and I like taking advantage of tools that let me run remote usability studies. I also feel that testing and research speeds up your development process, since it tends to catch problems early, when they're easier to fix.

That said, user research is easy to get wrong. It takes some practice to be good at things like moderating sessions and analyzing data. For those of you who are interested in learning more about these things, I've compiled a list of resources to get you started.

My Blog Posts

These older posts should help you fix some of the common problems people have with user research:

• 5 Mistakes People Make Analyzing Qualitative Data
• Why Your Customer Feedback is Useless
• Pain Driven Design
• Shut up and Show Me Something
• Love Thy User: 5 Tips for Dealing with Tough Customers
• 6 Stupid Excuses For Not Getting Feedback
• 5 Things People Get Wrong When Talking to Users
• 6 Ways You May Be Failing at Customer Development
• 7 More Ways People Suck at Customer Development
• A Faster Horse - When Not To Listen To Users
• When Talking to Users Isn't Enough
• Improving the ROI for Your User Research
• A/B and Qualitative User Testing
• Why I Hate Paper Prototypes
• When To Get Help With User Research

Books

There are a million books about user research. These are two very good ones. Let me know in the comments if you've read any other particularly helpful ones. 

• Observing the User Experience: A Practitioner's Guide to User Research  
• Remote Research

Online Tools

These tools do NOT eliminate the need to actually interact with your users in person, but they can be extremely valuable additions to your user research process. 

• Skype, GoToMeeting, WebEx, etc. - Allow you to screen share so that you can observe how your users are interacting with your product. 
• usertesting.com - Very fast & cheap way to test your new user experience. 
• NavFlow - Lets you test your site navigation using mockups, which allows you to get feedback before you build the product. 
• Five Second Test - Great for testing things like landing pages or whether your calls to action are obvious enough. 
• Ethnio - Helps recruit session participants who are currently using your product. 
• Revelation - Helps you run longer term studies with current users.

Please Stop Annoying Your Users

Once upon a time, I worked with a company that was addicted to interstitials. Interstitials, for those of you who don’t know the term, are web pages or advertisements that show up before an expected  content page. For example, the user clicks a link or button and expects to be taken to a news article or to take some action, and instead she is shown a web page selling her something.

Like many damaging addictions, this one started out innocently enough. You see, the company had a freemium product, so they were constantly looking for ways to share the benefits of upgrading to the premium version in a way that flowed naturally within the product.

They had good luck with one interstitial that informed users of a useful new feature that required the user to upgrade. They had more good luck with another that asked the user to consider inviting some friends before continuing on with the product.

Then things got ugly.

Customers could no longer use the product for more than a few minutes without getting asked for money or to invite a friend or to view a video to earn points. Brand new users who didn’t even understand the value proposition of the free version were getting hassled to sign up for a monthly subscription.

Every time I tried to explain that this was driving users away, management explained, “But people buy things from these interstitials! They make us money! Besides, if people don’t want to see them, they can dismiss them.”

How This Affects Metrics

Of course, you know how this goes. Just looking at the metrics from each individual interstitial, it was pretty clear that people did buy things or invite friends or watch videos. Each interstitial did, in fact, make us some money. The problem was that overall the interstitials lost us customers and potential customers by driving away people who became annoyed.

The fact that the users could simply skip the interstitials didn’t seem to matter much. Sure people could click the cleverly hidden “skip” button – provided they could find it – but they had already been annoyed. Maybe just a little. Maybe only momentarily. But it was there. The product had annoyed them, and now they had a slightly more negative view of the company.

Here’s the important thing that the company had to learn: a mildly annoyed user does not necessarily leave immediately. She doesn’t typically call customer service to complain. She doesn’t write a nasty email. She just gets a little bit unhappy with the service. And the next time you do something to annoy her, she gets a little more unhappy with the service. And if you annoy her enough, THEN she leaves.

The real problem is that this problem is often tricky to identify with metrics. It’s a combination of a lot of little things, not one big thing, that makes the user move on, so it doesn’t show up as a giant drop off in a particular place. It’s just a slow, gradual attrition of formerly happy customers as they get more and more pissed off and decide to go elsewhere.

If you fix each annoyance and A/B test it individually, you might not see a very impressive lift, because, of course, you still have dozens of other things that are annoying the user. But over time, when you’ve identified and fixed most of the annoyances, what you will see is higher retention and better word of mouth as your product stops vaguely irritating your users.

Some Key Offenders

I can’t tell you exactly what you’re doing that is slightly annoying your customers, but here are a few things that I’ve seen irritate people pretty consistently over the years:

• Slowness
• Too many interstitials
• Not remembering information - for example, not maintaining items in a shopping cart or deleting the information that a user typed into a form if there is an error
• Confusing or constantly changing navigation
• Inconsistent look and feel, which can make it harder for users to quickly identify similar items on different screens
• Hard to find or inappropriately placed call to action buttons
• Bad or unresponsive customer service

It’s frankly not easy to fix all of these things, and it can be a leap of faith for companies who want every single change to show a measurable improvement in key metrics. But by making your product less annoying overall, you will end up with happier customers who stick around.

Like the post? Follow me on Twitter!

Also, come hear me speak on Wednesday, Sept. 29^th, at Web 2.0 Expo New York. I’ll be talking about how to effectively combine qualitative research, quantitative analytics, and design vision in order to improve your products. 

Everything In Its Place

I talk to a lot of designers. We’re all different kinds of designers: visual, interaction, user experience, information, blah blah blah, but many of us take the same things for granted. Because of this, designers will probably be bored to tears by this post, while non-designers may learn something that can make it much easier to build products that people can use.

But first, a story. A designer friend of mine had a baby. She asked her husband to put up a note asking people to please not ring the doorbell, since the baby was sleeping. Later, after somebody rang the doorbell and the baby woke up and she was contemplating divorce, she wondered why her husband hadn’t put up the damn note like she had asked.

The thing is, he HAD put up the note. He had put it right on the door at eye level so anybody could see it. What he hadn’t done was associated the call to action with the actual action.

What the hell does that mean? 

A big part of any user experience design is figuring out where to put stuff. This may sound obvious, but it’s best to put stuff where people are most likely to use it. That means associating calls to action with the thing that is being acted upon.

Here’s an example you may have considered. Where do you put a buy button on a page? Well, when a user is trying to decide whether or not to buy something, which pieces of information is the user most likely to need? He definitely needs to know how much he’s paying for the item. He might need pictures of the item. He almost certainly needs to know the name of the item and perhaps a short description.

Considering those needs, the Buy button should probably go near those things on the page. It should even go in a defined visual area with just those things. Here’s the hard part: it needs to go near those things EVEN IF IT LOOKS BETTER SOMEPLACE ELSE.

What's with all the screaming?

I’m all for having a nice visual design. I believe that a page should be balanced and pretty and have a reasonable amount of white space and all that. But if one element of your gorgeous visual design has separated your Buy button from the information that your user needs in order to decide to buy, then your gorgeous visual design is costing you more money than you think.

This isn’t just true for Buy buttons; it’s true any time the user has to make a decision. The call to action to make the decision must be visually associated with any information that the user needs to make that decision. Additionally, any information that is NOT related to the decision should be visually separate.

This also applies to things that aren't calls to action, of course. Related information should all be grouped together while unrelated information should be somewhere else. It's just that simple. Oh, and bonus points if you keep all similar items in the same place on every screen of your product so people always know where to look.

So, where should my friend’s husband have put the note? He should have put it within inches of the doorbell itself. Why? Because the decision the user was making was whether or not to ring the doorbell. The husband needed to put the information about the sleeping baby right at the point where the user was making the decision, not in a completely different part of the interface (the door) where the user might or might not even notice it.

The next time you're deciding where something goes, remember, this strategy is not only important for creating a usable product, it just might save your marriage!

Like the post? I'm a veritable font of useful information, I swear. Follow me on Twitter!

Your User's Computer Sucks

I often tell people that “you are not your user.” I’m an interaction designer. It’s part of my job description. What I should probably also be reminding people is that your computer is not your user’s computer. Nor is your internet connection, monitor, environment, or a lot of other things.

Why is this important? These things mean that, aside from making your product usable in a lab setting or in your office, it’s also got to work well in the standard environment of your user.

So, ok, if you build tools for lean startups, you can probably ignore most of this post. Your users all have dual core machines (probably more than one), fiber internet connections, 24 inch flat screen monitors, and a cubicle or office in which to use those things. They probably also have a smart phone that never leaves their hand and a comparable set up at home so that they are never more than a few inches from enough computer power to get them burned as witches in some parts of the world.

But the rest of you probably build products for teens or busy families or doctors or construction workers or business travelers or, you know, the vast majority of humanity that doesn’t use a computer for a living. And you need to not only understand your users; you need to understand your user’s computing environment.

I do a lot of in-home/office studies as well as remote usability testing. This means that, not only do I get to see real users with my products, I get to see them use them on their computers, and I’ve seen this over and over.

Here are a few examples of how your user’s environment really matters:

Your Computer is Faster than Your User’s

Interactions that take a split second on my machine sitting right next to the server may take two or three seconds on an old computer half way around the world. This doesn’t seem like much, but it has a pretty big impact.

When an interaction takes a split second, I don’t need to plan for any intermediate feedback (a spinner, a progress bar, a disabled button, etc.). When an interaction takes three seconds, I really, really do need one of those things, or else I’m going to get repeated button presses and confused users who don’t know whether anything is happening. Of course, interactions that are annoyingly slow on my computer are going to be completely intolerable to a lot of my users.

You Have More Computers than Your User Does

I have a laptop and a smartphone at home that are all my own, and I use them constantly.  The other day, I asked a user why she chose to use a product late at night. She explained that she had school aged kids who needed the computer in the afternoons, and during the day she was typically out of the house. Her only computer time was a few minutes after the family was all in bed.

Many products are used by multiple people during the day on the same computer, sometimes at the same time. Having limited time on a shared computer creates all sorts of design implications for things like privacy and the need to be able to interrupt and resume tasks.

Your Monitor is Bigger than Your User’s Monitor

Of course, then there’s monitor size. Many people have declared the death of “the fold” because people don’t mind scrolling a bit for interesting information, but I still see products with really important calls to action that don’t show up on screens smaller than a bay window.

Guess what? I’ll scroll to read the second half of a blog post, but I’m not going on a damn treasure hunt for your Buy button. If I don’t find it quickly, there are a whole lot of other sites that will sell me what I want. If your target audience accesses your site on laptops or smartphones or Etch A Sketches, figure it out and design accordingly.

Look, “getting out of the building” isn’t just another way of saying “chatting with customers.” It means understanding customers and how they use your product. A big part of how people use your product is dictated by the environment in which they use it. So make sure that you don’t only understand who is using your product and how they are using it. Learn WHERE they’re using it and on what sort of equipment. It can make a huge difference.

Like the post? Follow me on Twitter!

The Best Visual Design in the World

As I’ve mentioned before on this blog, I don’t do visual design. It’s not that I don’t think it’s important. I’m just not very good at it.

Even though I can’t produce gorgeous visual designs, like every other person on the planet, I know what sorts of visual design I prefer. I don’t have one particular style that I’m in love with, but I have pretty strong reactions, both positive and negative, to different “looks.”

Recently, I worked with a company that had a visual design I didn’t like. Now, since I’m not a visual designer, I’m not going to speculate on whether it was badly designed or just not to my taste, but I will tell you that when I showed it to many people in Silicon Valley, they didn’t like it either.  

In fact, enough people reacted negatively that I stopped showing it to people in the Valley. I even found myself apologizing for it, despite the fact that I didn’t design it, and I don’t love it.

And then I did some user testing on the site. And do you know what? The users love it. They LOVE it. It is absolutely fantastic for this particular demographic, which, by the way, has nothing to do with the Silicon Valley CEOs and designers who were horrified by it.

I was showing some wireframes, with the usual disclaimers of “this isn’t how it will look; these are just black and white mockups of the final site; we’re not losing the other color scheme; blah blah blah.” Despite repeated statements to this effect, users would periodically interrupt the test to volunteer how much they love the visual design of the current site and how they really don’t want it to change.  

Why is this important? It’s a great example of the fact that your visual design should reflect the aesthetic of your target market and not necessarily you. Say it with me, “You are not your user.”

Designing a beautiful, elegant, slick site that will appeal to designers, Silicon Valley executives, and Apple users is fantastic…if you’re selling to people like designers, Silicon Valley executives, or Apple users. That’s not the market for this company, so they’re smart not to build a product that appeals aesthetically to that market.

Is there such a thing as bad visual design? Sure. I’ve certainly seen visual designs that interfered with usability. Buttons can be too small; calls to action can be de-emphasized; screens can be too cluttered; navigation can be hard to find. But just because something isn’t visually appealing to you, doesn’t make it a bad visual design. The only people who have to like it are your users.

In your next design meeting, remember this: the best visual design in the world is the one your users love. 

Love Thy User: 5 Tips for Dealing with Tough Customers

Sometimes we build products for ourselves, but most of the time our target market is somebody completely different. It can cause all sorts of problems when we’re asking questions and observing people completely different from ourselves. Sometimes, and this can be tough to admit, we don’t really like our users very much.

Maybe you’re not like this. Maybe you’ve never had a difficult set of users who constantly yell and scream about their needs and how they’re not being met regardless of what you do for them. Maybe you’ve never spent time building a brand new feature designed to make your users happy only to have them shrug and say, “oh, that’s not what we wanted at all.” Maybe you’ve never had a passionate community of early adopters all grumpy because their favorite suggestions aren’t being followed to the letter. But trust me, the rest of us have.

The problem is, because most of our users are so different from us, their behavior can be extremely hard to understand or predict. On many occasions, this has led people to ignore their customers or neglect to include them in the development process.

I understand this impulse. I really do. It can be tough to include somebody that you see as irrational or hard to deal with in your decision making process.

But here’s a news flash. That irrational, difficult, whiny, impossible to understand person who is always complaining? Suck it up, cupcake, and include them in the conversation. They’re paying your salary, and if you ignore them for long enough, they’re likely to stop doing it.

Here are a few ways to make it easier to get feedback from difficult groups of customers.

Keep it one on one

When you’ve got a group of people, all of whom seem hell bent on complaining about how your product is ruining their lives, don’t put them in a room together.

A lot of companies like to establish customer advisory panels or customer forums and the like, where they can get feedback from a lot of people at once. These are fine when the conversation can be kept civil, but they can quickly turn into an angry mob as the group forms a giant echo chamber of hate.

Keeping the conversation one on one allows you to spend more time with each person and understand what’s really upsetting him or her.

When To Get Help With User Research

I don't spend a lot of time on this blog telling you why you should hire me to talk to your customers. In fact, the vast majority of the posts are meant to make it more possible for you to talk to your customers without hiring somebody like me. It's not that I don't like working. It's just that I think that anybody who is responsible for making decisions about products should know how to learn from users on their own. It results in better products for all of us.

Product owners need to be involved in customer research for a lot reasons. Reasons like:

• You're more likely to believe the results if you participated in the research.
• You're more likely to understand the relative importance of customer problems if you observed the problems happening.
• You will come up with more comprehensive solutions to problems when you understand the context in which they're happening.
• It's far too easy to ignore a report written up by a usability consultant, it's incredibly easy to forget to watch the testing videos.
• If you do it yourself, all of the lessons you've learned will stay within the company, long after a consultant has gone on to other projects.

That said, I'm about to tell you why you may need to hire somebody like me. For a little while at least.

When I talk about customer research or customer development or learning from customers, I really mean quite a lot of different techniques. Sure, there are general best practices around talking to customers, tips for improving your research skills, and important things you should avoid, but there are also things like picking the right testing method or tool that you almost certainly have no experience with. You need to know what is the most important thing for you to do right now.

Do you know when it would be helpful to do a card sort? A journal study? A contextual inquiry? Do you know when it's fine to do a remote usability study vs when you should really run one in person? How about when your product will benefit from using an online tool like usertesting.com or fivesecondtest, and when something like that isn't useful? Do you know what sort of testing to do in order to find out why specific metrics are lower than you'd like? Do you know when you should start your visual design and when you need to concentrate on usability? Do you know how many people to talk to in order to answer a specific question? Do you know at what points in the development cycle talking to users is critical and when it's a waste of time? Do you know how to take several hours of free form user conversations and turn it into a small number of features or bug fixes that can be communicated to your engineering team?

If you answered, "of course I know that" to all of those questions, then move along. You almost certainly have no use for somebody like me to come in and help you out. If you answered, "I'm going to learn the answer to all of those questions," then I wish you good luck on your journey of discovery. I'll warn you though. There are more questions just like those.

If, on the other hand, you said, "I don't know the answer to a lot of those questions, but I wish somebody could help me understand the small subset of them that matter to me, as a product owner, so that I could get on with the business of building a great product," then you might want to give me (or somebody like me) a call.

Because it's true that there is a huge amount to know about talking to your users. But it's also true that, at any given stage in your product development, you probably only need to be concerned with only a little bit of it. And, it's also true that figuring out which bit of it you need to know can be really hard to do without help. That's where people like me come in handy. We can help you figure out what to do next, and then we can help you learn how to do what you need to do next.

But be careful. If you're a lean startup, you probably don't want to pay us to actually do what you need to do next. For all the reasons I mentioned above, that's still your job.

Interested in this sort of service? Learn more about Users Know here.

Want to read more posts on how to do this stuff yourself? Follow me on Twitter!

Shut up, and Show Me Something

I admit it. Quite often on this blog I give you long lists of fairly hard things to do. I ask you to change your whole approach to design or product management or customer interviewing or analyzing data. But not today. Today, I share with you one simple thing that is easy to remember and will transform your entire approach to customer research.

Ok, maybe it's not quite that cool, but it really will help you communicate with your customers better. Are you ready? Here it is:

Never have a conversation with a user (or potential user) where you don't show them something.

That seems simple enough, right? But why on earth should you do it, and what could you possibly show them?

Reasons for Communicating with Customers

Let's back up for just a moment. The main reasons that people generally talk with a user are:

• To get information from them - what they like, what they don't like, what's confusing, why they're not buying things, etc.
• To give them information - here are the features of the product, here's how to fix your problem, we swear it's a feature and not a bug, etc.
• To sell them something - whatever it is that sales people do...besides drinking heavily

All of these things are much easier to do when you're looking at visual aids.

Getting Information from Users

Let's perform a thought experience. Without thinking about it, name three things you hate about doing your taxes. Were you able to do it? Of course you were. If you can't think of three things you hate about doing your taxes, either you're not paying attention, or your hiding all of your money in an offshore account in the Caymans. But are they really the three worst things?

Probably not. They're just the three things that you happen to think about when put on the spot. Next tax season, you'll be doing your taxes and think to yourself, "Oh right, THAT thing! I hate that thing! I wish I'd thought of that when I was asked for three things I hate." And you most likely would have thought of it if you'd been going through your tax preparation software when I asked.

Sure, you can just ask users what they like and dislike about your product, but you will get much better information if you're both looking at the product together. Even better, ask them to perform some tasks or just use the product while you watch. This not only jogs the user's memory about all the little annoying things that they're sort of used to, but you can also observe all the things that they don't even notice or are too embarrassed to mention they're having trouble with.

Nobody is Thinking About Your Product

When you're working at a startup it can be all-consuming. You can forget everything else in your life pretty easily when you're neck deep in valuations and minimum viable products and customer acquisition and a million other things that need your attention. You tend think about your product every waking minute.

That's why it can be such a shock to realize that nobody else is thinking about your product. Well, ok, unless you're Apple, but there's clearly some kind of weird mind control thing going on there. In general, when you have a new product, you're incredibly lucky if you're getting more than a few minutes of attention from anybody but your most passionate early adopters.

Why is it important to realize this? It's important, because it has a really big impact on how you design your product and connect with your users.

Make Everything More Discoverable

You know exactly where in the user interface to go to do every task that can be completed with your product (I hope!). Other people, especially new users, don't even know that most of your features exist. This means that it's just as important to design for discoverability as it is to design for usability. But how are they different?

Let's do a quick thought exercise. Imagine somebody hands you a featureless metal box. You might look at it for a minute or two. If it's particularly attractive, you might admire it, but you're probably not going to spend a lot of time with it. Now imagine that the box has $10,000 dollars inside of it. You will probably spend a lot more time figuring out to get it open, yes?

Your product is like that box that is hiding money. If people don't discover very quickly that it provides something valuable to them, they're not going to spend much time figuring out how to use it. You need to help people understand immediately that your product has features they really, really want. That's discoverability.
You also need to make it pretty easy to actual learn how to use those features, once they've decided to dig into the product a bit. That's usability. For bonus points, you can make the whole process interesting and engaging so that people actually enjoy discovering features and using your product. That's fun. 

Key Take Away: Users are not going to spend any time learning to use your product if they don't immediately understand what's in it for them. Make it easy for them to figure out what features exist and why they're useful.

Some Surprising Problems Caused By Using Your Own Product

We've all heard the stories of huge companies that started because the founder wanted to do something specific but didn't have the proper tools. He or she built the proper tools and quickly realized that he or she wasn't the only one who wanted those tools. Paul Graham wrote a very interesting blog post about what he calls organic startup ideas a couple of months back and why building something you need is often the best way to start a company.

There are a lot of benefits to being your own product's first and biggest user:

• You will always have at least one person who likes and uses your product
• Even if nobody buys it, you'll still be happy you built it, since you get to use it
• You always have a user available to consult on which feature to build next
• It should be easy to initially identify a target persona (hint: you may need a mirror)
• You probably have a network of similar people who may also want the product
• You can find a lot of bugs and corner cases by using your product on a regular basis

But there may be a few problems that you weren't expecting.

Understanding the New User Experience

Frankly, there is nobody worse at figuring out how confusing the new user experience can be than an expert. And, if anybody is an expert at the product you've built for yourself, it's YOU.

Have you ever tried to explain something "simple" to somebody and realized that it isn't, in fact, simple at all? It is extremely complicted with lots of steps. The only reason you think it's simple is because you've done it a million times. Unfortunately, it can be very difficult to assess how hard it is to learn something once we know too much about it.

For example, if you already know that a feature exists somewhere in the product, it's much easier to figure out where it is than if you're brand new to the product and have no idea that the feature even exists. It's also easier to understand the logic of how different features work together if you're the one who put them together in the first place. 

The fix: Luckily, there's an easy fix for this. Watch new users with your product. Select people in your target market and just observe their struggles. Use products like usertesting.com to observe the first 15 minutes of their usage. Get in touch with people who have only used your product a few times and ask if you can watch them (not in a creepy way), in order to understand what slightly more experienced people are doing with your product. Whatever you do, when you see somebody making a mistake, don't correct them! It's important to see how people who aren't you are using the product.

Asking for Feedback

Remember all that stuff I recommended you do for new users? You should also be doing it for much more experienced users too. The problem is, you probably won't.

In my experience, people who are big users of their own product are less likely to think that they need to observe customers because they have one right there in the building! It's hard to admit that you don't know everything about a product that you're both building and using extensively.

The issue here is that you're not the only type of user. You may not even be the main type of user. When Twitter was first developed as an internal communications tool, do you think that the creators thought that Ashton Kutcher would one day be using it to tell his fans what he had for lunch? People are out there doing really surprising things with your product. You need to learn what they are, and you can only do that by talking to them and observing them.

The fix: This one's easy. Just make sure to keep getting feedback from other users. Preferably, search out people who are different from you or who are using your product in very different ways.

When You Should Hurt Your Users

I'm going to let you in on a little secret. I don't always do what's best for users. I know, I know. This is a horrible thing for a UX person to admit, but it's true.

This shouldn't come as a surprise to anybody, but it's shocked enough of my clients that it bears stating clearly. As an interaction designer, my job isn't (or at least, it shouldn't be) always to optimize everything exactly for the end user. There are several reasons for this.

The Best Thing For the User May Be Bad for Business

One of my favorite Dilbert cartoons explained that what users really want is "better stuff for free." As much as I dearly love to help the end user, my clients would quickly go broke if I gave users everything they wanted. A big part of the UX job should be balancing what is best for the user with what is best for the business. Sadly, those aren't always the same thing.

What does this mean in reality? It means that every time you see a banner on a free site, some UX designer accepted the necessary evil of showing an advertisement for acai berry in return for offering content for free. It means that sometimes a pricing page will be designed to showcase a more expensive option than the user might otherwise have chosen. It also means we have to work extra hard to identify features that are good for both users and the business (for example, features that make current paying users happy and thereby increase retention), so that we don't always feel like we have to sacrifice either revenue or customer happiness. 

Caution! This can go too far in the other direction. I would argue that a lot of Facebook's recent privacy debacles have been driven by optimizing too much for the business and not enough for the end user experience. The result of this, of course, can be losing end users, which can end up being bad for business in the long run.

Getting It Out There May Be Better Than Getting It Perfect

Nothing is perfect the first time it's released. If it is, you probably spent too much time on it. And sometimes the very best solution for the user is impractical for a lot of other reasons.

What does this mean in reality? Well, it means that if the absolute optimal solution for end users will take 6 years and a team of 20 geniuses to implement, I will try my hardest to find the next best alternative. I'm not saying I'm going to settle for a bad user experience. I'm just saying that there are a lot of different ways of making the user happy, and if it can be done without killing the engineering department, that's a bonus. 

Caution! This does not mean that you get to ship a bunch of crappy features and then never touch them again. If I give you a pass to deliver a suboptimal first version to the user in the interest of time and engineering effort, I expect that it will be iterated upon and improved in the very near future. It may never be perfect, but it had better improve!

When Talking to Customers Isn't Enough

A few weeks ago, I was talking to the head of a small company about next steps. His company had a product with many happy, paying customers, and he wanted to know what to do next to make his current customers happier and attract some new ones.

The company had already made a good start. They had done surveys of current users asking the standard questions like, "How disappointed would you be if you could no longer use this product." They'd even surveyed current users about what new features they would like to see. The problem was, the happy customers couldn't think of anything else they wanted, and while the slightly less happy customers wanted some new features, there was no general consensus on one big, missing piece or fantastic new idea. So, the CEO wanted to know what to do next.

I believe this can be a problem with the "go out and talk to your customers" solution to product development. We can get so focused on talking to customers that we forget that it's not always the best way to figure out what to do next.

Observe Your Customers

Customers lie. They don't always mean to lie, but it often ends up that way. It's especially problematic when you ask people to explain how they currently use your product. Sometimes they forget parts of their process, or they don't even realize that they're doing certain things because it's all become so routine. Also, they tend to explain their process of using your product from start to finish, as if they weren't doing seven other things while trying to use your product. This can give you a totally skewed vision of what people are actually doing with your product.

What's the solution? Go out and watch them. Sit with them in their offices or homes and observe their real behavior. Most importantly, watch what they do immediately before and after they use your product.

I was talking with somebody who used to work for a marketplace that allows people to buy and sell products directly to each other. While observing users, her team noticed that, while people had very little trouble using the marketplace itself, the sellers spent a huge amount of time arranging shipping of the items. In fact, the shipping process took so much time that it limited the number of items they could list. By integrating with a shipping company to help customers print labels and schedule pickups, the company increased the number of items that could be listed which increased revenue.

Why didn't users ask for that? Well, the customers had a particular way of doing things. They thought of the marketplace as a place where they could buy and sell things, not as a product that helped them mail things. They had another solution that helped them mail things, and they didn't know that there was a better way to do that until they were presented with it.

How Many Features Does it Take to Destroy Your Product?

I work with a lot of startup founders who are extremely excited about their products. This is unsurprising, since nobody in their right mind would launch a startup without being totally convinced that their product is going to be awesome. It’s just too much work and risk for something that you aren’t passionate about.

Passion in startups is a great thing. It can also be incredibly dangerous.

The story is the same with pretty much every group of founders I work with: they have hundreds of great feature ideas for their product. Let me be clear: HUNDREDS of great feature ideas. Don’t get me wrong, many of these are genuinely great ideas. Almost every time I hear one, I think, “Wow, that would be really nice to have in this product and would add a lot of value.”

Unfortunately, starting a product with all of those features causes a lot of problems.

More Features = More Time to Build

Imagine building a house that includes a single room vs. one that has twenty rooms on three floors. There’s a lot more work, design, and planning that needs to go into the bigger design to make sure that people don’t get lost and that the house won't collapse.

This may seem obvious, but building one feature takes a lot less time than building ten. And the more time you spend building your initial product, the longer it takes to start getting metrics from actual users. Sure, you can and should start getting feedback from people using mockups and prototypes, but nothing beats actual data from your real product in the hands of users.

More Features = More UX Complexity

Remember that 20 room house? It has a lot more windows, doors, staircases, and electrical outlets than the one room place. It can be hard to find a reasonable place for everything. The more complex your product is, the more challenging it can be to present all of your features to your users in a usable way.

It’s hard to design a product that has a lot of different features. Everything needs to have an obvious, findable place in the product structure. All the features need to fit together in such a way that a user never gets lost or confused. Starting with a couple of key features keeps the overall UX much simpler and vastly reduces both your design time and the chances that your product will look like a giant mess.

More Features = Less Clear Value Proposition

Ever come to a web site or opened a product and thought, “What on earth does it DO?” Generally, the culprit is a confusing jumble of features, menus, buttons, and calls to action that prevent you from understanding the main value proposition of the product. Sure, companies try to combat the problem with wordy feature descriptions, video tutorials, and on-rails first time user experiences, but those often make things worse.

The simple fact is that the fewer features your product has, the easier it is for a new user to immediately understand what your product has to offer them. When all you have is new users, making sure they don’t get confused and leave is a pretty high priority.

Visual Design is Important, But Maybe Not Just Yet

I’ve been thinking a lot lately about visual design. A few weeks ago, at the Startup Lessons Learned conference, somebody asked how important visual design is in user experience, and recently somebody asked how much time I spend on visual design early in the design process. The answers to those two questions are: “incredibly important” and “not much.”

Why Is Visual Design Important in UX?

A lot of people, even a lot of designers, write off visual design as “making something pretty.” Frankly, I’ve been guilty of this myself. Meanwhile, companies like Google and Facebook have made fortunes while keeping their visual design so spare as to be almost non-existent. So you may reasonably wonder, why is visual design important to my startup or new product, and why should I bother spending time and resources on it?
But visual design doesn’t just make things pretty. It’s instrumental in a lot of ways, including:

• Enhancing your information design
• Reinforcing desired user actions
• Setting the tone for your product

Visuals Enhance Information Design

Sure, Facebook has a very simple, blue, gray, and white look. That’s because Facebook is all about delivering an enormous amount of content and information to you quickly. A bright, distracting, or cluttered interface would make it hard to read large numbers of news items and could clash with user-posted pictures.

Same with Google. Google, at least the main search product, is about getting you to type in a search term and then giving you access to everything on the internet that matches it. That’s a lot of information to process. Sites that are about delivering large quantities of information need to keep their visual design simple. But a simple design doesn’t mean no visual design at all.

7 Ways Design Can Speed Up Your Product Development

One thing I hear a lot from startups that don't want to bother with design is, "But design will slow us down!" They're sure that the fastest way to get a good product in front of users is to just jump in and start coding and then iterate on the feature or product once it's been built and released. Guess what? They're wrong.

A good designer who is willing to work with a lean development team can remarkably speed up your progress. Let's look at some different ways that design can help you quickly get your product into the hands of your customers.

I Can Design Faster Than You Can Code

So, you think you're a pretty fast coder, right? I believe you. But, unless the feature is dead simple, you can't build a working version of it faster than I can build a fake version of it. Remember, my prototype doesn't have a backend or a database. It doesn't have privacy controls. It doesn't have to support multiple users. It doesn't even have to work at all. It just has to look like it works.

Getting rid of all those requirements makes my prototype really, really fast to build. That means I can build several versions of it in the time it might take you to get the backend set up. If I make those versions seem like they work, I can get the prototype in front of users, get their feedback on it, and iterate much faster than you can.

That means, by the time you build the real version, it's already been through two or three versions and validated with customers. Many of the major usability problems will already have been found and eliminated. Complicated things will have been simplified. You'll already know how it integrates with the rest of the user interface. If you assume that you'd need to do all of these things anyway, this will save you time.

I Don't Need to Design it All Upfront

Maybe you've only worked with waterfall-style designers in the past, but there are some of us who don't need two months and a full set of specifications to get a product designed. We can get started on the broad outlines just a little before you begin coding and have something ready for you to get started with right out of the gate.

Besides, all of the products I've ever worked on had at least some non-customer facing code that needed to be written, so maybe you could write that part first. If you have some idea what you're coding based on things like customer stories (you did write customer stories, didn't you?), you can get started on anything the user doesn't see, and I can get started on everything the user does see. By the time you're done making sure the feature isn't going to bring down the servers, I can have built and tested a few mockups.

6 Questions To Ask Before Launching a New Feature

There are a lot of things to pay attention to when you’re launching a new feature. This post is going to cover a few of the questions that you really ought to ask before launch, but may not have in the past. Now, to be clear, these are not the obvious questions like:

• Is this addressing a user need or pain point?
• Has this been through and passed some form of QA testing?
• Do I have a marketing plan for letting users know this exists?
• What is the expected ROI for this feature?
• Will this feature scale to a reasonable number of users?
• Is this feature usable, and is the UX reasonably consistent with the rest of my product?

If you have not asked the above questions about the feature you’re about to release, then you should stop reading this blog post and answer them right now.

The questions we’re asking today are more advanced, but your launch will go a lot more smoothly if you ask (and answer!) them ahead of time.

How Am I Going to Get Feedback on this Feature?

It’s not enough to just launch a feature and see if your revenue increases. You need to have some sort of plan for testing to see how it’s affecting key metrics, whether those are revenue, retention, registration, user happiness, or some other number you care about. You need to have a strategy for finding how users feel about the new feature, whether they’re using it, and how it’s affecting your bottom line.

A good answer to this question might include some or all of the following:

• I am initially releasing this feature to a small cohort of random users and A/B testing key metrics of the sample group vs. the control.
• I have embedded a short survey or feedback link into the page and will gather qualitative data from my users that way.
• I have contacted my customer service/community management team and prepped them with information about the new feature and asked them to get back to me if we start to see any significant new support requests.
• I have already released the feature to a small, select group of beta customers or my customer advisory board, and I am meeting with them to discuss their feedback.

Your Users Are Doing Something Surprising

This post is for all of you lucky enough to have a product with real users. Way back before you had users, or even a product, you probably went through a process to figure out what you should build. During that process you may have written user stories and work flows that described, in various levels of detail, how your users would perform each expected task. But you know who didn’t read your user stories? That’s right: your users.

The result? Somewhere out there, a whole lot of your users are doing something totally unexpected with your product.

Ok, maybe it’s obvious that sometimes users do the unexpected. Maybe you expected your SMS-based social network to help people find out where their friends are in real time, but instead celebrities started using it to market directly to their fans. Maybe you created a product designed to help bands promote themselves, but instead ended up with a social network where people hook up with their high school sweethearts. Or, although this seems extremely unlikely, maybe you had an MMO but people just used it to share photos.

Whatever they’re doing, it’s something you didn’t expect, but it’s very important that you learn what it is as soon as possible!

Why Should You Care?

There are three excellent reasons for you to know what your customers are actually doing with your product:

• So you know if you are missing an opportunity to pivot your product or marketing
• So you know if you are missing an important feature
• So you don’t accidentally destroy a commonly used workaround or "unplanned feature"

Pain Driven Design

I have a problem with both User Centered Design and Customer Driven Development. This may come as something of a shock to people who read my blog, since I’m constantly giving advice on better ways to talk to users, analyze data, and improve customer development efforts.

The problem I have with UCD and CDD is not the methods. The problem is that people so often misunderstand them. People hear “user centered” and think, for some insane reason, that we are encouraging them to turn their entire design process over to the user. They hear “listen to your customer” and think that we want them to blindly accept every ridiculous feature request made by somebody with a credit card and an opinion.

Guess what? I rarely speak for entire communities of people, but I think I can safely say that nobody in the user centered design or customer driven development business are asking that of anybody. If they are, they’re idiots and shouldn’t be listened to anyway.

Unfortunately, a lot of us have debunked this myth by explaining that we don’t actually think that customers can predict future behavior (even their own) or that they’re going to have some grand design vision for your product. We just think that customers are great at talking about their problems and pain points, and that those are good things for you and your designers to know when you create a new feature or product.

I’m suggesting that we come up with a new name that will be harder (or at least more fun) to misinterpret: Pain Driven Design.

What Is Pain Driven Design (PDD)?

The PDD methodology requires that, before you start design for a product or feature, you need to figure out what is causing pain for your users and potential users. The desired outcome of PDD is to make that pain go away by some brilliant method of your devising. You then check to make sure you made their pain go away without causing them any more pain.

5 Mistakes People Make Analyzing Qualitative Data

My last blog post was about common mistakes that people make when analyzing quantitative data, such as you might get from multivariate testing or business metrics. Today I’d like to talk about the mistakes people make when analyzing and using qualitative data.

I’m a big proponent of using both qualitative and quantitative data, but I have to admit that qualitative feedback can be a challenge. Unlike a product funnel or a revenue graph, qualitative data can be messy and open ended, which makes it particularly tough to interpret.

For the purposes of this post, qualitative information is generated by the following types of activities:

• Usability tests
• Contextual Inquiries
• Customer interviews
• Open ended survey questions (ie. What do you like most/least about the product?)

Insisting on Too Large a Sample

With almost every new client, somebody questions how many people we need for a usability test “to get significant results.” Now, if you read my last post, you may be surprised to hear me say that you shouldn’t be going for statistical significance here. I prefer to run usability tests and contextual inquiries with around five participants. Of course, I prefer running tests iteratively, but that’s another blog post.

Analyzing the data from a qualitative test or even just reading through essay-type answers in surveys takes a lot longer per customer than running experiments in a funnel or looking at analytics and revenue graphs. You get severely diminishing returns from each extra hour you spend asking people the same questions and listening to their answers.

Here’s an example from a test I ran. The customer wanted to know all the different pain points in their product so that they could make one big sweep toward the end of the development cycle to fix all the problems. Against my better judgment, we spent a full two weeks running sessions, complete with a moderator, observers, a lab, and all the other attendant costs of running a big test. The problem was that we found a major problem in the first session that prevented the vast majority of participants from ever finding an entire section of the interface. Since this problem couldn’t be fixed before moving on to the rest of the sessions, we couldn’t actually test a huge portion of the product and had to come back to it later, anyway.

The Fix: Run small, iterative tests to generate a manageable amount of data. If you’re working on improving a particular part of your product or considering adding a new feature, do a quick batch of interviews with four or five people. Then, immediately address the biggest problems that you find. Once you’re done, run another test to find the problems that were being masked by the larger problems. Keep doing this until your product is perfect (ie. forever). It’s faster, cheaper, and more immediately actionable than giant, statistically significant qualitative tests, and you will eventually find more issues with the same amount of testing time.

It’s also MUCH easier to pick out a few major problems from five hours of testing than it is to find dozens of different problems from dozens of hours of testing. In the end though, you’ll find more problems with the iterative approach.

5 Big Mistakes People Make When Analyzing User Data

I was trying to write a blog post the other day about getting various different types of user feedback, when I realized that something important was missing. It doesn’t do any good for me to go on and on about all the ways you can gather critical data if people don’t know how to analyze that data once you have it.

I would have thought that a lot of this stuff was obvious, but, judging from my experience working with many different companies, it’s not. All of the examples here are real mistakes I’ve seen made by smart, reasonable, employed people. A few identifying characteristics have been changed to protect the innocent, but in general they were product owners, managers, or director level folks.

This post only covers mistakes made in analyzing quantitative data. At some point in the future, I’ll put together a similar list of mistakes people make when analyzing their qualitative data.

For the purposes of this post, the quantitative data to which I’m referring is typically generated by the following types of activities:

• Multivariate or A/B testing
• Site analytics
• Business metrics reports (sales, revenue, registration, etc.)
• Large scale surveys

Statistical Significance

I see this one all the time. It generally involves somebody saying something like, “We tested two different landing pages against each other. Out of six hundred views, one of them had three conversions and one had six. That means the second one is TWICE AS GOOD! We should switch to it immediately!”

Ok, I may be exaggerating a bit on the actual numbers, but too many people I’ve worked with just ignored the statistical significance of their data. They didn’t realize that even very large numbers can be statistically insignificant, depending on the sample size.

The problem here is that statistically insignificant metrics can completely reverse themselves, so it’s important not to make changes based on results until you are reasonably certain that those results are predictable and repeatable.

The Fix: I was going to go into a long description of statistical significance and how to calculate it, but then I realized that, if you don’t know what it is, you shouldn’t be trying to make decisions based on quantitative data. There are online calculators that will help you figure out if any particular test result is statistically significant, but make sure that whoever is looking at your data understands basic statistical concepts before accepting their interpretation of data.

Also, a word of warning: testing several branches of changes can take a LOT larger sample size than a simple A/B test. If you're running an A/B/C/D/E test, make sure you understand the mathematical implications.

Short Term vs. Long Term Effects

Again, this seems so obvious that I feel weird stating it, but I’ve seen people get so excited over short term changes that they totally ignore the effects of their changes in a week or a month or a year. The best, but not only, example of this is when people try to judge the effect of certain types of sales promotions on revenue.

For example, I've often heard something along these lines, “When we ran the 50% off sale, our revenue SKYROCKETED!” Sure it did. What happened to your revenue after the sale ended? My guess is that it plummeted, since people had already stocked up on your product at 50% off.

The Fix: Does this mean you should never run a short term promotion of any sort? Of course not. What it does mean is that, when you are looking at the results of any sort of experiment or change, you should look at how it affects your metrics over time.

An Agile Alternative: Embedded Design

Last week I wrote a blog post about a depressing example of non-agile design. This week, I'd like to show you an alternative method of design, along with some examples drawn from my experiences. This is not, by any means, a new concept, but hopefully I can convince a few people who still think that agile development and interaction design don't go together.

I've worked as a designer and user researcher with agile teams on several different projects. The trick is that, instead of doing all the research and design up front and then throwing the work over the wall to the engineering team, the designer stays embedded with the team throughout the entire development process.

How does an embedded interaction designer help at various stages of the development process?

Strategy and Product Definition

At this stage, your designer (or design and research team, if you’re well-funded) should be out talking to potential users about their problems. A designer at this point in the cycle can help you define who your customer is, learn how potential customers are currently dealing with the problem that your product solves, and come up with a small set of crucial features that will solve those problems better.

Could a great product manager also do some of these tasks? Sure, and they should definitely be involved or even leading the process. But, in my experience, being involved at this stage allows me, as a designer, to really understand the problems I’m going to be solving and the people for whom I’m going to be solving them. In other words, to have a successful design process, I will have to know all these things anyway, and participating in the earliest stage of research is the most efficient way for me to learn them.

Example:  One company with whom I was working wanted to add a new feature that would allow users to play music while interacting with the product. From our discussions with users of the product, we knew that people were already doing this, but in a way that caused them all sorts of issues and made many people unhappy. We interviewed current users about things like:

• How they were currently using music with the product?
• What benefits did they get from using music with the product?
• What major problems were they having with music? 

We also looked at the quantitative data that we had about what sort of music was being played, how much people were likely to pay for it, and  what percentage of people were active music users. The answers to all of these questions helped us come up with a new concept for music that would solve many of the problems and add some fun new benefits

A Depressing Example of Non-Agile Design

I recently had a really depressing experience. I was asked to work on a very cool application that focused on my favorite subject. That’s not the depressing part. That part was awesome.

Despite a lot of misgivings, I agreed to use the waterfall development process and design a lot of stuff up front, because that’s what the client wanted. Don’t get me wrong. They were doing some development work on the back end, but the goal was to figure out the entire application UX and feature set, wireframe it, and get a visual design done before starting work on anything customer-facing.

The application was quite large and had several different areas. I specifically designed it so that many of the features were modular, which meant that the development team could leave out whole chunks of the UI without ruining the application. They could then add more feature modules in as they finished them, which would theoretically allow them to launch as soon as they had just a few modules finished. Even though we were using a waterfall method, I didn’t want to design something that had to be absolutely 100% finished and built before it was usable. Basically, I was giving them the option to be agile in their development process, if they chose to, even if everything was designed up front.

After a few months of design, we had a great product roadmap, a list of features we ideally wanted in version 1.0, a list of features that would be acceptable for 1.0 (still FAR more than a MVP), lots of fully interactive wireframes, a gorgeous visual design (not done by me, of course), and a lot of data ready to go on the back end. We had also burned through our design budget, so I turned everything over to the developers and moved on to other things.

After several months, the very first version of the product shipped. This is the depressing part. It included exactly 0% of the original design. That’s right. Not a single element I’d designed made it into the first version of the product. Unfortunately, because all the design work had been done up front, I had no idea why the company made the decisions they had made. Of course, if I had been involved with the project all the way through, I could have stripped down the design in a way that at least tried to stay true to the original vision of the product. Instead, it came across as something that had never really been designed at all.

Oddly, there’s a part of me that’s happy that they did what they did. After all, they ended up releasing a MUCH smaller version of the product than we had originally envisioned. I’m not sure if it’s a Minimum Viable Product – only time will tell if it’s viable, after all – but it has many fewer features than we originally envisioned.

But, in another way, I’m sad. If I could have convinced them to go this small at the very beginning of the project, I could have designed their Minimum Viable Product in a lot less time, and, I think, with a lot better result than they got doing it on their own. More importantly, we could then have worked together to build out new features based on learning from the MVP, and we probably still would have had some design budget left to eventually implement that gorgeous visual design.

I guess my lesson from all of this is that I need to be stricter with companies who want a giant, fully fleshed-out product all designed for them at the beginning of the process. I need to be clearer that they’d be better off designing something very small and immediately implementable and then working from there. And I need to encourage them to spread the design budget out over a longer period of time so that they are never left without design resources. Either that, or I need to let go of my designs once they’ve been handed over to the client and acknowledge that they get to do (or not do) anything they want with them. I just find that really depressing.

Why Your Customer Feedback is Useless

Here’s the scenario: You have a minimum viable product. You’re talking to your users about it. You’re asking them questions, and they’re answering. But for some reason, it’s just not turning into usable information.

You wonder what’s going on. You imagine that perhapsyour users suck at giving good feedback or else they don’t have anything useful to say. Maybe, you think in a moment of hopeful delusion, your product is so perfect that it can’t be improved by customer feedback.

While these are all possibilities, the reality is that it’s probably not your customers’ fault. So, if you don’t seem to be getting any good data, what IS the problem? Probably one of the following things:

You’re asking the wrong questions

I’ve written before about asking customers the wrong questions, but in summary, customers are very good at giving you certain kinds of information and very bad at other kinds. For example, users are great at telling you about their problems. They can very easily tell you when something isn’t working or interesting or fun to use. What they suck at is telling you how to fix it.

Customers are great at:

• Complaining about problems
• Describing how they currently perform tasks
• Saying whether or not they like a product
• Showing you parts of a product that are particularly confusing
• Comparing one product to another similar product
• Explaining why they chose a particular method of doing something

Customers are bad at:

• Predicting their future behavior
• Predicting what other people will like
• Predicting whether they’ll pay for something
• Coming up with innovative solutions to their own or other people’s problems
• Coming up with brand new ideas for what would make a product more appealing

To take advantages of users’ strengths, have them describe things like, “Tell me about your most recent experience using the product and how that went for you.” Or ask them questions like, “What about [competitor product] do you particularly enjoy? What do you hate?” You can even ask questions like, “Of the following 5 features, which would you prefer?” You’ll want to be a lot more careful about listening to their answers to overly broad questions like, “What brand new feature would you like to see implemented?” or “What would make this product more fun to use?”

Seven More Ways People Suck at Customer Development

I’ve spent many years talking to users about how to improve products. It’s a big part of the job of any UX professional. With the recent emphasis on lean customer development and MVP, talking to customers is no longer limited to a couple of people in an organization. These days, everybody is learning from customers, and that’s a great thing for the usability of products.

Unfortunately, it does cause a few new problems. The main one is that most people who are new at talking to users kind of suck at it.

I’ve written a bit about the mistakes that inexperienced user researchers make when talking to users. However, there are several other serious problems that are far more likely to occur when the people doing the interviewing are deeply invested in the product. If you’re a founder, an engineer, a designer, a product owner, or anybody else with strong, emotional ties to your product, and you’re trying to learn by talking to users, you need to make sure you’re not falling into any of the following traps.

The Big Sale

Often, when you’re sitting down with a potential customer to talk them about your product, your first impulse is to make as good an impression as possible. This is perfectly natural. The problem is, it can really get in the way of getting good information from the potential customer about how to make your product better.

If your goal is to understand what’s not working about your product or what’s preventing people from buying it, stop trying to sell your product to the potential customer. Don’t read off a list of features that are in the product or that will be in the product. Don’t tell the customer how awesome the product is or how good it is at solving all their problems. And, for the love of God, don’t tell them about your Vision for the product.

What should you do? Shut up and let the customer tell you about their problems and how they’d like to solve them. Listen to the customer tell you about his perception of your product and what he sees as wrong with it. Your goal isn’t to convince the customer that your product is great. Your goal should be to let the customer help you make your product great.

How to Make Your Product as Awful as a Corporate Intranet

Recently I was dealing with intranets for a couple of largish companies, and I realized something. In every company where I've worked or contracted, intranets have been a problem. For some reason, the vast majority of intranets turn into huge, unwieldy, out-of-date, unusable link farms with bad search results. I’m told there are exceptions, but I’ve never seen them. Even small companies have big, awful intranets.

I started to figure out why intranets are so bad, and I came up with a list of problems that all of the ones I’ve seen seem to share. That’s when I noticed that these problems are in no way unique to intranets.

Many products I’ve used or worked on have had at least some of these problems.

None of the products I’ve used has ever managed to combine all of these problems into one enormous, unholy mess in quite the same way that a bad intranet can, but that doesn’t mean you shouldn’t give it a try! To help you out, here is a list of horrible things you can do to make your product just as bad as a typical intranet. Good luck!

Never throw anything away

The first key to really ruining your product’s usability is to never throw anything out.

Have a new feature, link, or module? Throw it onto the main screen along with everything else you’ve ever released! If even one person thinks it’s important or useful, you must be sure to support it until the end of time. This will ensure that your interface is so cluttered that nobody will be able to use the 10% of your product that is actually interesting to 90% of your customers.

Don’t put anybody in charge

It is very important to make sure that no single person is responsible for your overall product. If you want a really terrible product, make sure to enforce this total lack of responsibility and ownership.

6 Ways You May Be Failing at Customer Development

Listening to users can be difficult for a small company. Most start ups, especially lean ones, don’t have a dedicated person or team whose only goal is to connect with users, gather feedback, test products, or design new features. Often these roles are being filled by founders, engineers, or product owners. This isn’t necessarily a bad thing. In fact, having all sorts of employees connecting with users can be great.

The biggest problem I’ve noticed in situations like this is that the people who are talking and listening to users aren’t really very good at it.

You see, learning from customers can be hard. Sure, people will tell you it’s as easy as sitting down and watching somebody use your product or asking a few questions– and don’t get me wrong, that alone can be valuable! But actually getting the right information from customers and turning it into a product can take some training and practice.

What You Are Probably Getting Wrong
Let’s take a look at some of the most common and costly mistakes I’ve seen when people are trying to do their own customer development without much experience or guidance.

Bad Interviewing Technique
There are a whole host of problems that people commonly have when they first start moderating studies or interviewing users about products. I covered five of the biggest issues in this post for the Sliced Bread Design blog, but there are dozens of ways to screw up an interview.

But why does interviewing technique matter at all? Because that’s how you’re going to get information from your customers! Good technique makes it a lot more likely that you’ll be able to elicit open, honest, actionable feedback from your users. Bad technique means you may not learn anything useful or, even worse, that you may bias the user enough so that you only hear what you expected to hear.

Needless to say, it you’ve never run a user discussion session before (or even if you have), it can’t hurt to brush up on techniques like not giving a guided tour of the product, letting the user fail, not leading the witness, asking open ended questions, letting the user explore, and shutting the hell up. There is a lot more to being a successful interviewer, but fixing those few common mistakes will make getting good user feedback a whole lot easier.

Not Turning Data into Action
Once upon a time I did some work for a company that boasted of being committed to customer development. They brought people into the office weekly and chatted with them. They solicited customer opinions in surveys and forums. They did everything right! Except, when the time came to make product decisions, those discussions with customers were often conveniently forgotten, and the company just implemented features with very little regard for the data they had so painstakingly collected.