Users Know

Tiny Tests: User Research You Can Do NOW!

2011-12-28T12:41:00.000-08:00

There’s a lot of advice about how to do great user research. I have some pretty strong opinions about it myself.

But, as with exercise, the best kind of research is the kind that you actually DO.

So, in the interests of getting some good feedback from your users right now, I have some suggestions for Tiny Tests. These are types of research that you could do right this second with very little preparation on your part.

What Is a Tiny Test?

Tiny Tests do not take a lot of time. They don’t take a lot of money. All they take is a commitment to learning something from your users today.

Pick a Tiny Test that applies to your product and get out and run one right now. Oh, ok. You can wait until you finish the post.

Unmoderated Tests

Dozens of companies now exist that allow you to run an unmoderated test in a few minutes. I’ve used UserTesting.com many times and gotten some great results really quickly. I’ve also heard good things about Loop11 and several others, so feel free to pick the one that you like best.

What you do is come up with a few tasks that you want to see people perform with your product. When the test is over, you get screen captures of people trying to do those things while they narrate the experience.

Typically, I’ll use remote, unmoderated testing when I want to get some quick insight into whether a new feature is usable and obvious for a brand new user.

For example, if you’ve just added the ability for users to message each other on your site, you can use remote, unmoderated testing to watch people attempt to message somebody. This will help you identify the places where they’re getting lost or confused.

If you’ve done a little recruiting and have a list of users who are willing to participate, you can even ask your own users to be the participants.

And don’t forget, if you don’t have a product, or if you’re looking at other products for inspiration, you can run an unmoderated test on a competitor’s product. This can be a great way to see if a particular implementation of a feature is usable without ever having to write a line of code. It can also be a great way to understand where there might be problems with competing products that you can exploit.

Are you going to get as much in-depth, targeted feedback as you would if you ran a really well designed, in person user test? Probably not. But it’ll take you 10 minutes to set up and 15 minutes to watch each video, so you might actually do this.

Remote Observation

There is something to be said for traveling to visit your users and spending time in their homes or offices. It can be extremely educational. It can also be extremely expensive and time consuming.

Here’s a way to get a lot of value with fewer frequent flyer miles.

Look at the people in your Skype contacts. Find one that doesn’t know much about your product. Ping them. Ask them to do three small tasks on your product while sharing their screen.

Don’t have Skype? Send friends a GoToMeeting or a WebEx link through email.

As with the remote unmoderated testing, this is best for figuring out if something is confusing or hard to do. It’s not very useful for figuring out whether people will like or use new features, because typically the people in your Skype contacts aren’t representative of real users of your product.

The closer the people are to your target market, the better the feedback’s going to be, but almost anybody can tell you if something is hard to use, and that’s information that it would be great if you had right now.

Coffee Shop Guerrilla Testing

Of course, it’s tough to test a mobile app over Skype. You know where it’s easy to test a mobile app? At a coffee shop.

Go outside. Find a Starbucks (other coffee shops are also acceptable if you refuse to go to Starbucks, you insufferable snob). Buy some $5 gift cards. Offer to buy people coffee if they spend 5 minutes looking at your product. Have a few tasks in mind that you want them to perform.

In about an hour, you can watch a dozen people use your app. And if you don’t manage to get any good feedback, at least you can get coffee. But you’ll almost certainly get some good feedback.

This type of feedback is great for telling you if a particular task is hard or confusing. It’s also great for getting first impressions of what an app does or the type of person who might use it.

Five Second Landing Page Testing

Sometimes, all you want to test is a new landing page. What you frequently want to know about a landing page is, “What message is this conveying, and is it conveying it clearly and quickly?” Even the tiniest of tests can seem like overkill for that.

For landing pages, I use UsabilityHub’s Five Second Test. You take a screenshot or mockup of the landing page you want to show. You upload it to the site. You enter a few questions you want people to answer after looking at it.

If the whole setup process takes you more than 5 minutes, you’re doing it wrong, and within a few hours you can have dozens of people look at your landing page and tell you what they think your product does.

This sort of Tiny Test is wonderful for testing several different variations of messages or images that you might put on a landing page. You can get people’s real first impressions of what they think you’re trying to tell them.

CTA Testing

The most important thing to get right on any screen is the Call To Action. After all, you can have the most gorgeously designed images with a wonderfully crafted message, but if people can’t find the damn Buy button, you’re screwed.

But, as with the landing page tests, this is something that takes 5 seconds. Basically, you want to show people a screen and see if they can figure out where they should click. Guerrilla testing works pretty well for this, but even that may be overkill here.

For CTA testing, I often use UsabilityHub’s ClickTest product. Again, you just upload a mock and ask people something like, “Where would you click to purchase the product shown on this page?” or “Where would you go to advance to the next slide?” or whatever CTA you’re testing.

A few hours later, you get a map of where people clicked. If there are clicks all over the place, you’ve got some work to do on your CTA.

The advantage to doing something like this over a/b testing is simply that you can get it set up very quickly with just mockups. You don't have to actually implement anything on your site (or even have a site) in order to test this way. But, if you have enough traffic and a good a/b system already set up, by all means test that way, as well.

What Are You Waiting For?

There you go. Five different options for wildly fast, incredibly cheap feedback on your product. You don’t have to hire a recruiter or write a discussion guide or rent out a usability lab. In a few cases, you don’t even have to interact with a human.

Are they perfect? Do they take the place of more substantial research? Will you be able to get away with avoiding talking to your users forever? No. But they’re easy, and you can do one of them right this second.

So...do one of them right this second!

Like the post? Follow me on Twitter.

STFU About What Women Want

2011-12-11T18:30:00.000-08:00

In a recent post on TechCrunch, Penelope Trunk tells us (again) that most women don’t want to do startups.

First, I’d like to extend that to Asians, African Americans, Gays, and Latinos. Oh, and white men. Most of them don’t want to do startups either, because most people don’t want to do startups for a whole host of reasons.

Penelope tells us that women are different though, because women don’t want to join startups because women want to have babies. As evidence, she points out that most women downshift their careers as soon as they have babies, which of course makes startups impossible.

It’s not that women don’t join startups because of lack of opportunity or sexism or doing what’s expected of them or anything else. Now that we have completed defeated bias, all women can choose to do anything they want, and they are choosing to have babies rather than go to startups. Case closed!

Here’s the problem: Penelope, and other people who say things like this, are making my life a whole lot harder, and I’d like them to knock it the fuck off.

I’m not going to argue that most women don’t want to stay home with their children. Frankly, I don’t care what most women want to do.

I know what I want to do, and what I want to do is to work at startups. I don’t want to have children. I’ve never wanted children. I never will want children, and I certainly wouldn’t want to give up working at startups for them.

So, when a publication like TechCrunch spews some nonsense about what women want, it means that the next time I go into an interview with a male founder (and they are overwhelmingly male for some reason that I’m not going to address here, but that Penelope assures us has nothing to do with bias) who has read that nonsense, he may be thinking, consciously or subconsciously, “she doesn’t really want to work at this startup because she wants to have a baby.”

And frankly, that sucks for me and all the other women like me. Oh, did I mention that there are lots of other women like me? There are.

But let’s just look for a moment at what all of these other women, the ones with babies and without startups, are choosing to do. They are choosing to stay home because of...I don’t know what the current argument is. Hormones? Biology? Bad government policy? Nature?

It couldn’t possibly be bias or lack of opportunity because, of course, some women are choosing to work at startups, so it would be trivially easy for all women to choose to work at startups, right?

Except that my father’s law school class of 1963 had 3 women in it. That’s right. Three. Now, clearly more women could have joined the class. His law school didn’t have a 3 woman quota or anything.

But the women of 1963 chose not to go to law school. And I’m positive that there were all sorts of blowhards opining that women didn’t go to law school because they were too busy having babies, and this was perfectly normal, and we shouldn’t do a damn thing to promote more women going to law school because WON’T SOMEBODY PLEASE THINK OF THE CHILDREN.

After all, we weren’t actively STOPPING women from going to law school (any longer). It was their choice! Except that, as Penelope points out, more than 50% of the law school graduates now are women.

So, far more women are now choosing to be lawyers. You know, despite the fact that they still have babies. What women wanted, with regard to law school attendance, somehow changed between 1963 and today.

Similarly, long before women had the right to vote in the US, many women didn’t actually want the right to vote. Some even felt that women were biologically not capable of voting well. And for years after they had the right to vote, many people still felt that it was the wrong decision.

How many women in the US do you know who don’t care about voting? Fewer than felt that way a hundred years ago, right?

The point is that “what women want” changes over time. What people want changes over time. Because what we want is hugely driven by social norms and massive cultural shifts and all sorts of things that may seem biological at the time but turn out not to be.

In other words, if suddenly there are a ton of women at startups kicking ass and being awesome, it might turn out that more young women want to join startups in the future. And 25 years from now, we’ll all be laughing at the idiots who said things like “women don’t want to vote...er...go to law school...I mean...join startups!”

Penelope, do you vote? Do you know women who went to law school? I do. And I am forever grateful to the women who fought not just for the right to do these things but to make them seem like totally normal things to do.

I salute the women who said, “Hey, wait a minute. Maybe having a vagina doesn’t determine what I have to want from life! Just because a lot of women want something, doesn’t mean that I have to want the exact same thing!”

And I’m still a little annoyed at the women who said, “Oh, women don’t WANT to vote. Voting is for men!” I take that back. I’m a lot annoyed at them. They sucked.

So stop doing it. Stop assuming other women want to make the same choices you do, especially when society has such an enormous and invisible impact on your choices. Stop assuming a young woman just starting her career knows everything about all of the wonderful, exciting career choices she could make.

And mostly, stop making it ok for other people to assume that I want what you want. That’s clearly not true, since what I want most right at this moment is to punch you in the face.

I am a woman. I want to work at a startup. I don’t want to have children. I want to vote. I want to wear stiletto heels and write jQuery, sometimes at the same time. In other words, I am an individual, and I have all sorts of wants that are neither determined nor predicted by my gender.

I am a woman, Penelope, but you don’t have any idea what I want. So, kindly shut the fuck up about it.

Give the Users What They Really Want

2011-12-01T11:47:00.000-08:00

Recently, I’ve been trying to teach startups how to do their own user research. I’ve noticed that I teach a lot of the same things over and over again, since there are a few things about research that seem to be especially difficult for new folks.

One of the most common problems, and possibly the toughest one to overcome, is the tendency to accept solutions from users without understanding the underlying problem. In other words, a user says, “I want x feature,” and instead of learning why they want that feature, new researchers tend to write down, “users want x feature," and then move on.

This is a huge issue with novices performing research, When you do this, you are letting your users design your product for you, and this is bad because, in general, users are terrible at design.

Ooh! An Example!

I participated in some user research for a company with an expensive set of products and services. Users coming to the company’s website were looking for information so they could properly evaluate which set of products and services was right for them. Typically, users ended up buying a custom package of products and services.

One thing we heard from several users was that they really wanted more case studies. Case studies, they said, were extremely helpful.

Now, if you’re conducting user research, and a customer tells you that he wants case studies, this might sound like a great idea.

Unfortunately, the user has just presented you with a solution, not a problem. The reason that this is important is that, based on what the actual underlying problem is, there might be several better solutions available to you.

When we followed up on users’ requests for case studies with the question, “Why do you want to see case studies?” we got a variety of answers. Interestingly, the users asking for case studies were all trying to solve entirely different problems. But were case studies really the best solution for all three problems?

These were the responses along with some analysis.

“I want to know what other companies similar to mine are doing so that I have a good idea of what I should buy.”

The first user’s “problem” was that he didn’t know how to pick the optimal collection of products for his company. This is a choice problem. It’s like when you’re trying to buy a new home theater system, and you have to make a bunch of interrelated decisions about very expensive items that you probably don’t know much about.

While case studies can certainly be helpful in these instances, it’s often more effective to solve choice problems with some sort of recommendation engine or a selection of pre-set packages.

Both of these more quickly help the user understand what the right selection is for him rather than just give him a long explanation of how somebody else found a good solution that might or might not be applicable to the user.

“I want to know what sorts of benefits other companies got from the purchase so I can tell whether it’s worth buying.”

The second user’s “problem” was that he wanted to make sure that he was getting a good value for his money. This is a metrics problem. It’s like when you’re trying to figure out if it’s worth it to buy the more expensive stereo system. You need to understand exactly what you’re getting for your money with each system and then balance the benefits vs the cost.

This problem might have been solved by a price matrix showing exactly what benefits were offered for different products. Alternatively, it would be faster and more effective to display only the pertinent part of the case studies on the product description page - for example, “Customers saw an average of 35% increase in revenue 6 months after installing this product.”

By boiling this down to only the parts of the case study that were actually important to the user, it gives you more flexibility to show this information - statistics, metrics, etc. - in more prominent and pertinent places on the site. This actually increases the impact of these numbers and improves the chance that people will see them.

“I want to see what other sorts of companies you work with so that I can decide whether you have a reputable company.”

The third user’s “problem” was that he hadn’t ever heard of the company selling the products. Since they were expensive products, he wanted the reassurance that companies he had heard of were already clients. This is a social proof problem. It’s like when you’re trying to pick somebody to put a new roof on your house, so you ask your friends for recommendations.

His actual problem could have been solved a lot quicker with a carousel of short client testimonials. Why go to all the trouble of writing up several big case studies when all the user cares about is seeing a Google logo in your client list?

Why This Matters

This shouldn’t come as a surprise to any of you, but users ask for things they’re familiar with, not necessarily what would be best for them. If a user has seen something like case studies before, when he thinks about the value he got from case studies, he’s going to ask for more of the same. He’s not necessarily going to just ask for the part of the case study that was most pertinent to him.

The problem with this is that many people who might also find certain parts of case studies compelling won’t bother to read them because case studies can be quite long or because the user doesn’t think that the particular case study applies to him.

Obviously, this is applicable to a lot more than case studies. For example, I recently saw a very similar situation from buyers and sellers in a social marketplace asking for a “reputation system” when what they really wanted was some sort of reassurance that they wouldn’t get ripped off. I could name a dozen other examples.

The takeaway is that, when somebody asks you for a feature, you need to follow up with questions about why they want the feature, even when you think you already know the answer!

Once you know what their problems really are, you can go about solving them in the most efficient, effective way, rather than the way the user just happened to think of in the study.

Instead of just building what the user asks for, build something that solves the user’s real problem. As an added bonus, you might end up building a smaller, easier feature than the one the user asked for.

The Art of the UX Steal

2011-11-16T16:24:00.000-08:00

I’ve been building interfaces for a very long time, and I can tell you that the number of times I’ve had to solve a completely new and unusual user problem is remarkably small. This isn’t surprising. The vast majority of products we build incorporate a lot of familiar elements.

For example, think about the number of products you use that include one or more of the following: login, purchasing, comments, rating systems, order history, inventory management, or user generated content.

Do you expect that every single login experience gets redesigned completely from scratch in a vacuum? Of course not! It would be annoying if they were, since each new version would almost certainly differ just enough to make things confusing. Having design standards for things like logging in makes a lot of sense for both users and designers.

However, this tendency to fall back on patterns, or just to copy whatever Apple/Amazon/Facebook is doing, can cause some problems, especially for startups. There are a few big reasons why you shouldn’t just adopt another company’s solution without serious consideration.

They May Not Want Exactly What You Want

Companies have hidden agendas. But their agenda is not always your agenda, which means that their optimal design is not your optimal design. And if you think that they’re always optimizing for the best user experience, you’ve lost your damn mind.

Want an example? Ok! Have you ever purchased an item and been opted in to receiving email deals from the company’s partner sites? As a user, who likes that? Who thinks that’s a great user experience? Exactly.

Then why do companies do it? They do it because they have made the business (not UX) decision that they make more money by opting people into partner deals than they lose by slightly annoying their customers. That’s a totally reasonable calculation for them to do.

Now, let’s say your biz dev person comes to you and says he wants to add that feature to your checkout process because he has a partner lined up who is willing to pay for the privilege of getting your users’ email addresses. He says it will be ok to add the feature because other big companies are doing it, so it must make money.

But you have no idea how much money they’re getting for making their UX worse. You have no idea of the number of users they may be losing with this practice. And even if you did know their numbers, you can’t decide whether this feature is the right business decision for you until you know what those numbers are going to be for your product.

In an ideal world we could always just choose whatever made the best possible user experience, but realistically, we make these kinds of business/UX tradeoffs all the time. They’re inevitable. Just make sure that you’re making them based on realistic estimates for your product and not on the theory that it’s right because a bigger company is doing it.

They Don’t Do Exactly What You Do

By my count, Amazon has sold at least one of pretty much everything in the world. Ok, I’m just extrapolating from my purchase habits, but you know what I mean.

Not only do they sell products directly, they also allow other companies and individuals to sell through their marketplace. They also sell a lot of different versions of the same product. This makes their product pages pretty complicated.

Does your product do all of those things? If you work for a startup, I certainly hope not, since many of Amazon’s features were added over the course of more than a decade.

If your product doesn’t have all those features, then you might want to do all sorts of things differently than Amazon does. For example, your product pages could be significantly simpler, right? They could emphasize entirely different things or have clearer Calls to Action or more social proof because they don’t need to account for all of Amazon’s different features.

Whether or not you even have product pages, the point is that no other company is doing exactly what you’re doing (or if they are, you have an entirely different problem), so their optimal UX is, by necessity, going to be different from yours.

They Can Get Away with It

If Dante were writing today, the 9th circle of Hell would have involved trying to sign into multiple Google accounts at once. True story.

A friend of mine decided to make me angry the other day, so he showed me a Google docs screen where the Save button was so cleverly hidden it took him several minutes to locate it. This was on a screen that had maybe four elements, and he’s a very senior software engineer, so this probably wasn’t user error. I find the usability on certain Google products almost sadistically poor.

But I put up with it because Google provides me with incredible value for free that I can’t get anywhere else even by paying for it.

I don’t use things like Google docs for their UX. In fact, I use them in spite of large portions of their UX. And if your UX borrows from Google through some misguided notion that just because Google does it, it must be right, I will quit your product in a freaking heartbeat and bad mouth it to all my friends.

The moral of this story isn’t just “don’t steal UX from Google,” although that’s not bad advice. The moral is that very few companies succeed in spite of their UX, and if you happen to steal UX from them, you’re doing it wrong.

On a side note, you know what had a fabulous UX? The original Google product - the one where there was just a single search box, two buttons, and a hugely successful algorithm for finding great results. Unsurprisingly, that’s the UX that got us all hooked in the first place.

The Right Way to Steal

Now that the horror stories are out of the way, you still shouldn’t be coming up with every design element from scratch.

Not only is it ok to steal a basic login process from another product (although not Google), it’s almost certainly the best possible thing you could do. Having a non-standard way for users to log in to your product is just needlessly confusing.

One product I use on a regular basis used to put their Log In button on the top left of their home page instead of the top right. Just this little change meant that several times I had a hard time remembering how to get into the product, and wasted several seconds searching for the button. I probably wasn’t the only one to complain, since they fixed it relatively quickly.

Logging in isn’t the only thing to standardize. Any time you have a simple activity that users do regularly in lots of other products, you should at least check to see whether there is a standard and consider adopting it.

Of course, you can always choose not to do things the way everybody else is doing them, but you should have a very strong reason for changing things, and you should definitely a/b test your change against the standard design pattern.

Trust But Verify

Most importantly, when you are planning on stealing - or “adopting a standard” as we’re now going to euphemistically call it - it’s still important to test it.

I like to do quick qualitative tests to observe some people actually using the standard. In fact, often I’ll test the standard on competitors’ products before implementing it, rather than implementing it and then finding out that it’s crap. Then, I’ll test again once it’s implemented in my product.

In general, the more companies who are doing things identically the less likely it is to be confusing. But it’s still necessary to make sure that the design works in the context of the rest of your product.

Like the post? Follow me on Twitter!

Idiots, Drama Queens, and Scammers - Improving the Customer Service with UX

2011-11-03T15:03:00.000-07:00

I recently published another article in Smashing Magazine. This one is titled Idiots, Drama Queens, and Scammers - Improving the Customer Experience with UX.

Here's an excerpt:
User experience design isn’t just about building wireframes and Photoshop mock-ups. It extends to areas that you wouldn’t necessarily think are part of the discipline.

For example, your customer service department can have a huge impact on your website’s overall user experience. Similarly, the design of your user experience could have an awfully big effect on your customer service department. Of course, not all of your users will interact with the customer service department, but for those who do, their experience can improve or destroy the customer relationship.

Read more now >

How Metrics Can Make You a Better Designer

2011-09-21T15:00:00.000-07:00

I have another new article in Smashing Magazine's UX section: How Metrics Can Make You a Better Designer.

Here's a little sample:

Metrics can be a touchy subject in design. When I say things like, “Designers should embrace A/B testing” or “Metrics can improve design,” I often hear concerns.

Many designers tell me they feel that metrics displace creativity or create a paint-by-numbers scenario. They don’t want their training and intuition to be overruled by what a chart says a link color should be.

These are valid concerns, if your company thinks it can replace design with metrics. But if you use them correctly, metrics can vastly improve design and make you an even better designer.

Read the rest here >

Need Help with Your Design and Research?

2011-09-15T14:46:00.000-07:00

I used to do a lot of design and research for companies. Don't get me wrong. I still do design and research, but I’ve recently made a pretty significant change.

I no longer do design and research FOR companies. I now do design and research WITH companies.

I promise this isn’t just semantic nonsense. It has a huge impact on my relationship with clients, and I think it has some good lessons for people who choose to work with outside UX help.

Give a Company a Fish

Let’s take a look at the typical experience you have when you hire a contractor or an agency. Typically, you give a lot of input to the contractor about what results you want, and the contractor goes off and produces something that hopefully fits those results.  
With a good contractor, you get a lot of discussion and iteration, but at the end, you get a design or a research report that somebody did for you. And that’s all you get.

If you want to change part of the design after the contractor is gone, you run the risk of making major mistakes, because you are very unlikely to understand all the decisions that were made in creating it. If you have a question about the research or want to do a quick follow up about something you learned, you don’t know how to do that yourself.

This means that the next time you want some research or design done, you need to hire somebody to do it for you again. This is great for the contractor, and it’s not bad for companies with big budgets, but it can be especially hard for startups.

Going Fishing Together

Last year, I decided to try a different model. When I was hired by clients, I came in and worked as part of the team. I was still doing the majority of the design and research, but I came in and worked at the office and tried to be integrated into the teams as much as possible.

That worked better than the old agency style I was used to. I had more contact with the engineers and product owners. We could iterate on the design faster because we were all in the same room. I learned far more about the product and users. Sometimes they learned a little about the design process.

Still, with some clients, I found that I was the only person in the room while doing customer research. I was the only one coming up with questions I wanted answered. I was still having to schedule design reviews rather than having everybody involved in the design process.

The worst part was that I was the only one learning anything about the customers. But they weren't MY customers!

Too often, what this meant was, when a project was over, everything at the company went right back to where it was before.

I started to look at why some projects ended this way, while in others, the companies seemed to incorporate good design and research skills into their own development process.

Teach a Company to Fish

Based on what I learned from the companies who improved, I have a different model now for all of my new clients. I’m helping companies learn to do more design and research on their own.

Instead of running a research study, I help product owners figure out what sort of research they need to do. I then help them plan it, execute it, analyze the data, and create actionable designs. If this were a sports team, I’d be a coach, not a ringer.

Of course, this does mean a lot more work for my clients. They have to figure out what questions they want answered. They have to talk to their customers. They have to do design work. They have to understand the process. It’s really hard, and not everybody wants to learn to do these things.

But the beauty of it is, once they’ve done it a few times, it all gets easier. It becomes part of the company process. More people in the company become interested in conducting research and creating designs.

Of course, eventually, my clients won’t need me any longer. It may not be the best business model, but I think it’s the best thing I can do for my clients.

What This Could Mean for You

This means that I can help you learn how to be better at research and design. For example, I can work with you on things like:

Which type of research is right for you at any given stage of your product development
How to plan that research correctly
How to moderate a user discussion properly
How to analyze your research results
How to create usable personas and write good user stories
How to turn research results into actionable designs
What changes you need to make to your product based on your results
When to use metrics and a/b testing in your design process
What to build now, what to test, and what to iterate on later

If you’re interested in any of those things, you should contact me at laura@usersknow.com. I’m happy to discuss the process in more detail and explain a typical engagement.

Why Your Test Results Don't Add Up and What To Do About It

2011-09-02T12:31:00.000-07:00

Check out my guest blog post for KISSmetrics: Why Website Test Results Don’t Always Add Up & What To Do About It!

Here's a little sample:

If you do enough A/B testing, I promise that you will eventually have some variation of this problem:

You run a test. You see a 10% increase in conversion. You run a different, unrelated test. You see a 20% increase in conversion. You roll both winning branches out to 100% of your customers. You donʼt see a 30% increase in conversion.

Why? In every world Iʼve ever inhabited, 10 plus 20 equals 30, right? Youʼve proven that both changes youʼve made are improvements. Why arenʼt you seeing the expected overall increase in conversions when you roll them both out?

Read the Rest at KISSmetrics.

Breaking the Rules: A UX Case Study

2011-08-18T11:58:00.000-07:00

Recently, I was lucky enough to be featured in Smashing Magazine's brand new UX section! Smashing is already a fabulous resource for web design and coding, and I think it's going to be a great place to learn about user experience.

You should read my first article, Breaking the Rules: A UX Case Study.

Here's a little something to get you started:

I read a lot of design articles about best practices for improving the flow of sign-up forms. Most of these articles offer great advice, such as minimizing the number of steps, asking for as little information up front as possible, and providing clear feedback on the status of the user’s data.

If you’re creating a sign-up form, you could do worse than to follow all of these guidelines. On the other hand, you could do a lot better.

Design guidelines aren’t one size fits all. Sometimes you can improve a process by breaking a few rules. The trick is knowing which rules to break for a particular project.

Read the rest of the article!

Stop Worrying About the Cupholders

2011-08-09T10:55:00.000-07:00

Every startup I’ve ever talked to has too few resources. Programmers, money, marketing...you name it, startups don’t have enough of it.

When you don’t have enough resources, prioritization becomes even more important. You don’t have the luxury to execute every single great idea that you have. You need to pick and choose, and the life of your company depends on choosing wisely.

Why is it that so many startups work so hard on the wrong stuff?

By “the wrong stuff” I mean, of course, stuff that doesn’t move a key metric - projects that don’t convert people into new users or increase revenue or drive retention. And it’s especially problematic for new startups, since they are often missing really important features that would drive all those key metrics.

It’s as if they had a car without any brakes, and they’re worried about building the perfect cupholder.

For some reason, when you’re in the middle of choosing features for your product, it can be really hard to distinguish between brakes and cupholders. How do you do it?

You need to start by asking (and answering) two simple questions:

What problem is this solving?
How important is this problem in relation to the other problems I have to solve?

To accurately answer these questions, it helps to be able to identify some things that frequently get worked on that just don’t have that big of a return. So, what does a cupholder project look like? It often looks like:

Visual Design

Visual design can be incredibly important, but nine times out of ten, it’s a cupholder. Obviously colors, fonts, and layout can affect things like conversion, but it’s typically an optimization of conversion rather than a conversion driver.

For example, the fact that you allow users to buy things on your website at all has a much bigger impact on revenue than the color of the buy button. Maybe that’s an extreme example, but I’ve seen too many companies spending time quibbling over the visual design of incredibly important features, which just ends up delaying the release of these features.

Go ahead. Make your site pretty. Some of that visual improvement may even contribute to key metrics. But every time you put off releasing a feature in order to make sure that you’ve got exactly the right gradient, ask yourself, “Am I redesigning a cupholder here, or am I turbocharging the engine?”

Retention Features

Retention is a super important metric. You should absolutely think about retaining your users - once you have users.

Far too many people start worrying about having great retention features long before they have any users to retain. Having 100% retention is a wonderful thing, but if your acquisition and activation metrics are too low, you could find yourself retaining one really happy user until you go out of business.

Before you spend a lot of time working on rewards for super users, ask yourself if you’re ready for that yet. Remember, great cupholder design can make people who already own the car incredibly happy, but you’ve got to get them to buy it first, and nobody ever bought a junker for the cupholders.

Animations

I am not anti-animation. In fact, sometimes a great animation or other similar detail in a design can make a feature great. Sometimes a well designed animation can reduce confusion and make a feature easy to use.

The problem is, you have to figure out if the animation you’re adding is going to make your feature significantly more usable or just a little cooler.

As a general rule, if you have to choose between usable and cool, choose usable first. I’m not saying you shouldn’t try to make your product cool. You absolutely should. But animations can take a disproportionate amount of time and resources to get right, and unless they’re adding something really significant to your interface, you may be better served leaving them until later.

“But wait,” a legion of designers is screaming, “we shouldn’t have to choose between usable and cool! Apple doesn’t choose between usable and cool! They just release perfect products!”

That’s nice. When you’ve got more money than most first world governments, you’ve got fewer resource constraints than startups typically do. Startups make the usable/cool trade off every day, and I’ve looked at enough metrics to know that a lot of cool but unusable products get used exactly once and then immediately abandoned because they’re too confusing.

Note: this may seem to contradict my point about attracting users first and then worrying about retention, but I’d like to point out that there’s a significant difference between solving long term retention problems and confusing new users so badly that they never come back.

Before you spend a lot of time making your animation work seamlessly in every browser, ask yourself if the return you’re getting is really worth the effort, or if you’re just building an animated cupholder.

Your Feature Here

I can’t name every single different project that might be a cupholder. These are just a couple of examples that I’ve seen repeatedly.

And, frankly, one product’s cupholder might be another product’s transmission. The only thing that matters is how much of an effect your proposed change might have on key metrics.

As a business, you should be solving the problems that have the biggest chance of ensuring your survival. Cupholder projects are distractions that take up too much of your time, and it’s up to you to make sure that every project you commit to is going to give you a decent return.

If you want to identify the cupholders, make sure you’re always asking yourself what problem a feature is solving and how important that problem is compared to all the other problems you could be solving. Cupholders solve the problem of where to put your drink. Brakes solve the problem of how to keep you from smashing into a wall.

Of course, if I got to choose, I’d rather you built me a car that drives itself. Then I can use both hands to hold my drink.

Like the post? Follow me on Twitter!

Hypothesis Generation vs. Validation

2011-08-01T14:51:00.000-07:00

A lot of people ask me what sort of research they should be doing on their products. There are a lot of factors that go into deciding which sort of information you should be getting from users, but it pretty much boils down to a question of “what do you want to learn.”

Today, I’m going to explore one of the many ways you can go about looking at this: Hypothesis Generation vs. Hypothesis Validation. Don’t worry, it’s not as complicated as I’ve made it sound.

What is Hypothesis Generation

In a nutshell, hypothesis generation is what helps you come up with new ideas for what you need to change. Sure, you can do this by sitting around in a room and brainstorming new features, but reaching out and learning from your users is a much faster way of getting the right data.

Imagine you were building a product to help people buy shoes online. Hypothesis generation might include things like:

Talking to people who buy shoes online to explore what their problems are
Talking to people who don’t buy shoes online to understand why
Watching people attempt to buy shoes both online and offline in order to understand what their problems really are rather than what they tell you they are
Watching people use your product to figure out if you’ve done anything particularly confusing that is keeping them from buying shoes from you

As you can see, you can do hypothesis generation at any point in the development of your product. For example, before you have any product at all, you need to do research to learn about your potential users’ habits and problems. Once you have a product, you need to do hypothesis generation to understand how people are using your product and what problems you’ve caused.

To be clear, the research itself does not generate hypotheses. YOU do that. The goal is not to just go out and have people tell you exactly what they want and then build it. The goal is to gain an understanding of your users or your product to help you think up clever ideas for what to build next.

Good hypothesis generation almost always involves qualitative research. At some point, you need to observe people or talk to people in order to understand them better.

However, you can sometimes use data mining or other metrics analyzation to begin to generate a hypothesis. For example, you might look at your registration flow and notice a severe drop off half way through. This might give you a clue that you have some sort of user problem half way through your registration process that you might want to look into with some qualitative research.

What is Hypothesis Validation

Hypothesis validation is different. In this case, you already have an idea of what is wrong, and you have an idea of how you might possibly fix it. You now have to go out and do some research to figure out if your assumptions and decisions were correct.

For our fictional shoe-buying product, hypothesis validation might look something like:

Standard usability testing on a proposed new purchase flow to see if it goes more smoothly than the old one
Showing mockups to people in a particular persona group to see if a proposed new feature appeals to that specific group of people
A/B testing of changes to see if a new feature improves purchase conversion

Hypothesis validation also almost always involves some sort of tangible thing that is getting tested. That thing could be anything from a wireframe to a prototype to an actual feature, but there’s something that you’re testing and getting concrete data about.

You can use both quantitative and qualitative data to validate a hypothesis, but you have to choose carefully to make sure you’re testing the right thing. In fact, sometimes a combination of the two is most effective. I’ve got some information on choosing the right type of test in my post Qual vs. Quant: When to Listen and When to Measure.

Types of Research

Why is this distinction between generation and validation important? Because figuring out whether you’re generating hypotheses or validating them is necessary for deciding which type of research you want to do.

Want to understand why nobody is registering for your site? Generate some hypotheses with observational testing of new users. Want to see if the mockups for your new registration flow are likely to improve matters? Validate your hypothesis with straight usability testing of a prototype.

These aren’t the only factors that go into determining the type of research necessary for your stage of product development, but they’re an important part of deciding how to learn from your users.

Like the post? Follow me on Twitter!

Designers Need to A/B Test Their Designs

2011-05-25T12:24:00.000-07:00

The other day, I posted something I strongly believe on Twitter. A few people disagreed. I’d like to address the arguments, and I’d love to hear feedback and counter-arguments in the comments where you have more than 140 characters to tell me I’m wrong.

My original tweet was, “I don't trust designers who don't want their designs a/b tested. They're not interested in knowing if they were wrong.”

Here are some of the real responses that I got on Twitter with my longer form response.

“There’s a difference between A/B testing (public) and internally deciding. Design is also a matter of taste.”

I agree. There is a big difference between A/B testing in public and internally deciding. That’s why I’m such a huge fan of A/B testing. You can debate this stuff for weeks, and often it’s a huge waste of time.

When you’re debating design internally, what you should be asking is “which of these designs will be better for the business and users.” A/B testing tells you conclusively which side is right. Debate over!

Ok, there’s the small exception of short term vs. long term effects, which is addressed later, but in general, it’s more definitive than the opinion of the people in the room.

With regard to the “matter of taste,” that’s both true and false. Sure, different people like different designs. What you’re saying by refusing to A/B test your designs is that your taste as a designer should always trump that of the majority of your users. As long as you like your design, you don’t care whether users agree with you.

If you want your design aesthetic to override that of your users, you should be an artist. I love art. I even, very occasionally, buy some of it.

But I pay for products all the time, and I tend to buy products that I think are well designed, not necessarily ones where the designer thought they were well designed.

“If Apple had done A/B tests for the iPod in 2001 with a user-replaceable battery, that version would’ve likely won—initially.”

Honestly, it still might win. Is taking your iPod to the Apple store when the battery dies really a feature? No! It’s a design tradeoff. They couldn’t create something with the other design elements they wanted that still had a replaceable battery. That’s fine.  

But all other things about the iPod being totally equal, wouldn’t you buy the one where you could replace the battery yourself? I would. The key there is the phrase “totally equal.”

“Seeing far into the future of technology is not something consumers are particularly great at.”

I feel like the guy who made this argument was confusing A/B testing with bad qualitative testing or just asking users what they would like to see in a product.

This isn’t what A/B testing does. A/B testing measures actual user behavior right now. If I make this change, will they give me more money? It has literally nothing to do with asking users to figure out the future of technology.

“A/B testing has value but shouldn't be litmus test for designer or a design”

Really? What should be the litmus test for a designer or a design if not, “does this change or set of changes actually improve the key metrics of my company”?

In the end, isn’t that the litmus test for everybody in a company? Are you contributing to the profitability of the business in some way?

If you have some better way of figuring out if your design changes are actually improving real metrics, I’d love to hear about it. We can make THAT the litmus test for design.

“Data is valuable but must be interpreted. Doesn't "prove" wrongness or rightness. Designer still has judgment.”

I agree with the first sentence. Data certainly must be interpreted. I even agree that certain design changes may hurt certain metrics, and that can be ok if they’re improving other metrics or are shown to improve things in the long run.

But the only way to know if your overall design is actually making things better for your users is by scientifically testing it against a control.

If your overall design changes aren’t improving key metrics, where’s the judgement there? If you release something that is meant to increase the number of signups and it decreases the number of signups, I think that pretty effectively “proves wrongness.”

The great thing about A/B testing is that you know when this happens.

“Is it the designers fault, surely more appropriate to an IA? After all the IA should dictate the feel/flow.”

First off, I don’t work for companies that are big enough to draw a distinction between the two, but I’m sure there’s enough blame to go around.

Secondly, I think that everybody in an organization has the responsibility to improve key metrics. If you think that your work shouldn’t increase revenue, retention, or other numbers you want higher, why should you be employed?

Design of all kinds is important and can have a huge impact on company profitability. That impact can and should be measured. You don’t get a pass just because you’re not changing flow.

“A/B tests are a snapshot of current variables. They don’t embody nor convey a bigger strategy or long-term vision.”

Also, “That’s only an absolute truth you can rely on if you A/B test for the entire lifespan of the product, which defeats the point.”

These are excellent points, and they are a drawback of A/B testing. It’s sometimes tough to tell what the long term effects of a particular design change are going to be from A/B testing. Also, A/B testing doesn’t easily account for design changes that are a part of a larger design strategy.

In other words, sometimes you’re going to make changes that cause problems with your metrics in the short term, because you strongly believe that it’s going to improve things long term.

However, I believe that you address this by recognizing the potential for problems and designing a better test, not by refusing to A/B test at all.

Just because this particular tool isn’t perfect doesn’t mean we get to fall back on “trust the designers implicitly and never make them check their work.” That doesn’t work out so well sometimes either.

An Argument I Didn’t Hear
 There’s one really good argument that I didn’t get, although some of the above tweets touched on it. Sometimes changes that individually test well don’t test well as a whole.

This is a really serious problem with A/B testing because you can wind up with Frankenstein-style interfaces. Each individual decision wins, but the combination is a giant mess.

Again, you don’t address this by not A/B testing. You address it by designing better tests and making sure that all of your combined decisions are still improving things.

How I Really Feel

Look, if I’m hiring for a company that wants to make money (and most of them do), I want my designers to understand how their changes actually affect my bottom line.

No matter how great a designer thinks his or her design is, if it hurts my revenue and retention or other key metrics, it’s a bad design for my company and my users.

Saying you’re against having your designs A/B tested sounds like you’re saying that you just don’t care whether what you’re changing works for users and the company. As a designer, you’re welcome to do that, but I’m not going to work with you.

Like the post? Follow me on Twitter!

5 Fun Ways to Ruin Your Startup

2011-05-24T14:03:00.000-07:00

So, you’re interested in ruining your startup. At least, that’s what it seems like based on a lot of decisions I see some companies making.

Let’s talk about some of those terrible decisions that really hurt startups.

Hire Big Thinkers

Here’s the thing about Big Thinkers or people who describe themselves as Big Picture People. They don’t execute. At least, they don’t execute in any way that is helpful to a startup.

Sure, there are a few people who can both lead and get their hands dirty with details. If you find one of those people, hire them immediately.

But more often, I see startups stall out because they’ve got somebody making decisions who doesn’t have to actually implement any of those decisions. They’re delegators. And the problem is, at very early stage startups, there just aren’t enough people to delegate TO.

If you’ve got a team of four or five people (or even ten or fifteen), every person should be spending the majority of his or her time actually building, making, designing, writing, testing, selling, or some other verb that isn’t “setting direction” or “planning” or “establishing policies.”

Want a successful startup? Hire Big Doers, not Big Thinkers.

Talk About Awesome Features All The Time

Yes, yes. You have this fantastic idea for the next big pivot that’s going to make you all rich. But you know what? That idea that you had 2 months ago that you still haven’t finished building was also fantastic. So is the one you’ll have 2 months from now. Also the one you’ll have 2 minutes from now.

Startup people are incredibly rich in ideas. Unfortunately, they tend to be broke in every other conceivable resource.

A great way to ruin your startup is to spend all of your time in meetings discussing in detail all the wonderful features you’re going to add in the future. Instead, capture the broad outlines of the idea quickly, put them in your backlog, and, when you’ve actually built something and need to move on to something new, see what ideas you’ve collected that would solve a real customer need. THEN design and build them.

Want a successful startup? Sure, you need to dedicate a little bit of time to thinking about the future, but spend a hell of a lot more time working on the present.

Wait To Ship Until It’s Perfect

It can be tough to release something into the wild before you think it’s perfect. But the thing is, it’s never going to be perfect, and the faster you get it out there, the faster you’re going to start learning which parts are the least perfect.

The longer you put off getting something in front of users, the more money you’re going to spend on something that might very well fail. Wouldn’t it be better to find that out early enough to turn it around and make it awesome?

Want a successful startup? Release small pieces of your product often, and get over worrying that it’s ugly or doesn’t work exactly the way you want it to. You’re just going to end up changing it all anyway.

Work 40 Hours a Week

This one may not be what you expect. It’s not some diatribe about how startup employees need to work 24/7 and not have outside lives and eat all their meals at their desks. If that works for you, great. Personally, I enjoy going outside.

But you do need to acknowledge that work at a startup doesn’t follow a strict 9-5 routine. Sometimes you need to check on things over the weekend or answer customer complaints late at night. Sometimes you need to make a final push to get something out the door quickly. Sometimes decisions need to be made outside of regular business hours, and there isn’t anybody else to make them.

Want a successful startup? You don’t need to live at the office, but you do need to be aware of what’s happening and be able to react when necessary. If you want to turn your phone off at 5pm on Fridays, you might consider working someplace where you’ve got more people to back you up.

Make A Lot of PowerPoint Decks

Sure, investors love them, and you’ve always got to show something to your board, but I’ve seen this get really out of hand. If you’re spending an hour or two a week building slides to share information with five other people, you are wasting everyone’s time.

I get that there’s important information that you need to share with the team, but the problem with PowerPoint is that people start doing things like tweaking display and finding funny pictures to make their points. A whiteboard works just as well for writing a few bullets, and it’ll get you out of meetings faster, not to mention taking far less prep time.  

Want a successful startup? Consider creating a simple dashboard of all the metrics that everybody in the company should be monitoring so that they can see the pertinent information at any time. That way, nobody’s waiting on you to build graphs and paste them into a deck once a week.

Like the post? Follow me on Twitter!

User Research You Should Be Doing (but probably aren't)

2011-04-15T12:09:00.000-07:00

Startups know they should get out of the building and talk to their customers, but sometimes they’re a little too literal about it. There are tons of ways to get great information from your customers. The trick is knowing which technique answers the questions you have right now.

Sure, you’re doing usability tests and trying to have customer development interviews, but here are a few slightly unusual qualitative user research techniques you should be employing but probably aren’t.

Competitor Usability Testing

Have you ever considered running a user test on a competitor’s site?

This one’s fun because it feels a little sneaky. It also gets you a tremendous amount of great information, since chances are somebody is already making mistakes that you don’t have to make.

For example, when one of my clients, Crave, wanted to build a marketplace for buying and selling collectibles, we spent time watching people using other shopping and selling sites. We learned what people loved and hated about the products they were already using, so we could create a product that incorporated only the good bits.

The result was a buying and selling experience that users preferred to several big name shopping sites that will remain nameless.

Bonus tip: There’s always the temptation to borrow ideas from a big competitor with the excuse, “well, so and so is doing it, and they’re successful, so it must be right!” Guess what? Sometimes other companies are successful for a lot of reasons other than that thing you’re stealing from them. Make sure users like that part of a competitor's product before using it in your own.

Super Targeted Usability Testing

Typically, when conducting usability tests, we’ll run several sessions on an entire product with lots of scenarios and tasks. But often that generates a ton of data that you have to wade through and analyze.

Instead, try doing a few sets of three 10-15 minute tests on a very specific feature. That’s a lot of numbers in a row. How about an example?

When we were building Crave, we wanted to test a particular new feature that we thought users would love. When we actually launched it, we were a little concerned that it would be hard to find, so immediately after launch, we ran three quick, unmoderated user tests with one task.

As we suspected, all three users had some trouble finding the feature. We immediately created a contextual help bubble that guided interested users to the feature. Then we ran three more tests. None of the new users had any problems at all.

The entire process took about three hours, and users regularly tell us how much they like that feature.

Bonus Tip: Using unmoderated user testing services like UserTesting.com and TryMyUI.com (and about a dozen others), make testing like this fast and cheap. You can test, build, deploy, and iterate several times in a single day. If you don’t do continuous deployment, you can use them to test high fidelity prototypes rather than your actual product.

Purely Observational Testing

This type of research is the exact opposite of the last one, because you’re not always testing a very specific part of your interface or a brand new feature.

Sometimes you’re trying to generate ideas for what you could do next that would give you the biggest ROI. For example, you might know that there’s a problem somewhere in your metrics, and you’re trying to understand what pain points are causing the drop off.

Whatever the reason, one of the most enlightening things you can do is a purely observational test. This means sitting down, shutting up, and watching people use your software in whatever way they want to do it.

You don’t give them tasks or scenarios. You just schedule them for a time they’d normally be using your product and arrange to observe them, remotely or in person.

Bonus Tip: Make sure to do this with new users, power users, and occasional users, as well as people who fit in all of your various persona groups. This will give you a fabulous overview of what people are really doing with your product.

Micro-Usability Tests

These are quite different, and some don’t fall neatly into the qualitative testing realm, but they can be quite useful.

Navigation Tests
When we were building Crave, we obviously wanted to make sure that things were incredibly easy to purchase, since that’s where we’d make money.

Since we had wireframes and visual mockups of the screens, we simply loaded them into UsabilityHub’s NavFlow and asked users to show us how they’d purchase something.

After a couple of tests, we knew exactly where in the purchase funnel we needed to improve things before ever even had a real funnel!

Landing Page Tests
Another type of Micro-usability test can help you fine tune your messaging.

Ever run one of those landing page tests where you compare two different messages and see which one results in more conversion? Ever wonder why the winner was the winner?

This is one of those questions that’s not very cost effective to answer with standard usability testing, since what you really want is a couple of minutes of testing with a lot of different users rather than an hour with just a few users.

Luckily, you can post a screen on FiveSecondTest with a few simple questions like, “What does this product do?” and “Who is this product for?” and get extremely cheap feedback about people’s first impression of your landing page.

Now you’ll not only know which version won, but you’ll have a better idea of WHY it won. Different tests that we ran at Crave showed that some messages led people to believe that the site was about “buying and selling” while others led people to believe it was about “sharing” or “meeting people.” And, of course, some messages didn’t mean anything to anybody. We didn’t use those.

Bonus Tip: As with everything, I like running smaller versions of these micro-usability tests iteratively. With the FiveSecondTests, I might run each version of the page with 15 people and then update the messaging until I get a landing page where the vast majority of respondents understand exactly what my product is selling.  

Do These Techniques Replace Usability Testing?

Seriously? Is that even a question? Of course not!

You still need to do regular usability testing and conduct standard customer development interviews. You still need to get out and ask your users questions and have them perform predefined tasks and talk about their problems.

But the next time you want a particular question answered, think a little harder about the best way to answer it and the best tools to use.

Like the post? Follow me on Twitter!

Or check out my presentation on DIY User Research for Startups.

Creating a Great Design and Research Culture

2011-04-05T12:34:00.000-07:00

I led a conversation recently at Web 2.0 Expo about creating a great design and research culture at your startup. To be clear, I didn’t offer to run it because I’m an expert, but it’s a topic I’m extremely interested in. I wanted to find out from other people what their problems have been and see if we could help each other solve those problems.

The most interesting thing to me was how similar many of the problems were, which leads me to hypothesize that too many companies are making the same mistakes over and over when trying to integrate design and research into their organizations.

Here are a few of the common complaints I heard and some of the solutions that were proposed.

Keeping Design in a Silo

The most common problem was bad communication between the design team and other teams within the company. One participant said that, in her company, the visual designers were on another floor from the UX designers, and the designs didn’t always translate correctly.

Another participant talked about a company where the engineers, designers, and strategy people were all in different countries. The cultural differences between the different teams led to even more communication problems.

Solution: Our proposed solution to this problem was to blend teams whenever possible. A participant told us that, when they embedded designers with the engineers all sorts of good things happened. Not only did communication improve because they were all sitting together, but they actually became friends, which made them all more willing to listen to different points of view.

Doing Research but Not Acting on Results

Another common complaint was that companies would claim to be “user focused” because they did a lot of user research, but they wouldn’t actually act on the research.

In a couple of cases, participants talked about an overall company culture that was resistant to actually making changes based on the research. The upper management was simply going to build whatever they wanted to build regardless of what the users were experiencing.

In another instance, the problem was that, by hearing the same problems over and over each week, people at the company actually became accustomed to those user problems. The problems seemed less urgent because they’d been hearing about them for so long.

Solution: There were a couple of proposed solutions to this problem. To get the team to truly understand the urgency of the customer problems and to get buy in from upper management, people suggested sharing videos of actual users failing to use the product. Nothing motivates change like seeing somebody struggle with something you think is easy!

Also, to prevent problems from becoming accepted parts of the product, I often suggest running small tests for specific problems with the idea that the team will immediately attempt to fix the problem and then test again to make sure that it was improved. This test->fix->test cycle can then be repeated until the problem goes away and a new one is discovered.

Getting Key Stakeholder Feedback Too Late

One participant shared a common problem that the legal team would come in at the very end of the process and jam a bunch of fine print into the design. In my experience, this can be a problem with any high level, external stake holder who doesn’t come into the process until after the design is almost finished.

Solution: One participant said he’d had good luck hiring a different legal team that was more specialized to his particular product and industry. However, this won’t work as well if it’s the CEO who is diving in at the end of the process and making changes.

The group generally agreed that it was important to get key stakeholders involved much earlier in the process. Janice Fraser, of LUXr, had an excellent suggestion to show stakeholders very early sketches and ask questions like, “What about this looks like it might be scary?”

The Magical Designer

There was one participant with what I hope is an unusual problem. He claimed that his designer thought that he could do no wrong. Despite being at a startup, he would throw fits and hold up production over single pixel changes.

Solution: Look, I know that at least one person is going to tell me that that sort of thing matters deeply and that at Apple things have to be pixel perfect and the designer rules everything and blah blah blah.

Startups can’t work that way. A one pixel difference is not what causes your startup to succeed or fail, so it shouldn’t be what’s determining shipping your product. My recommendation was to fire the designer and hire somebody who understands the concept of ROI.

The Perfect Culture

Ok, will solving all these problems give you an awesome design and research culture at your company. Probably not. But hopefully this can help you avoid some of the most common problems.

Like the post? Follow me on Twitter!

What Makes UX Lean - My Talk from SXSW

2011-03-15T14:31:00.000-07:00

If you couldn’t make it to SXSW this year, there was a fantastic, all day lean startup track with talks from lots of lean startup experts. I was lucky enough to be asked to be on the Lean UX panel, along with the always awesome Janice Fraser, Ian McFarland, and Dan Martell.

I gave a short talk on what makes Lean UX Lean. Since I’m a blogger at heart, I wrote down pretty much everything I was going to say first, which means I can now publish a draft of the talk here! If you didn’t get to hear the panel, or if you did and want a quick refresher, please enjoy!

I’ve been a user experience designer for a lot of years, and I’ve worked with a lot of lean startups, which is part of the reason why I got a call last year from Manuel Rosso, the CEO of Food on the Table.

Now, Food on the Table is a very lean startup here in Austin. Because they’re a lean startup, they measure absolutely everything. And because they measure everything, Manuel knew immediately when the product developed an activation problem.

The whole project has been written up in a post for Eric’s Startup Lessons Learned blog, and I strongly recommend that you go read it if you haven’t already. It has a lot of tips about how to incorporate design into your startup that you’ll hopefully find helpful.

But today, I want to go a little deeper into what made that project a good example of Lean UX. Because, during that project, we did a lot of things that you might do in any sort of a UX project for any sort of a company.

For example, we did qualitative user research to understand why users were having a problem. We made sketches and built interactive prototypes, and we tested and iterated on them.

These are wonderful, helpful things to do, but they’re not unique to Lean User Experience Design. They’re part of User Centered Design, which I’m a huge fan of, and I’ve done all of those things in waterfall projects at giant companies that were anything but lean.

So, what are a few things that made this a lean ux project and not just a regular old redesign?

Integrating Quantitative Research

I think the first hallmark of Lean UX is using quantitative metrics to both drive and validate design changes. What does that mean? Well, it means that the reason we were working on the first time user experience was because a specific metric, activation, wasn’t as high as the team wanted it to be.

Quantitative metrics didn’t tell us exactly why we had a problem - we needed to do our qualitative research to understand that - but it did tell us what our most immediate problem was, which helped us to understand where we should start improving our user experience.

In that way, the metric drove the product decision.

Quantitative metrics also meant that we knew, at the end of the project, we’d be validating our work with an a/b test against the original design. That quantitative validation of design really helps improve the design process over the long run, because we can see what sorts of changes have the biggest positive impact on our end user experience. That lets us improve the ROI on future design projects.

Overlapping Design and Development

Another thing that makes lean UX different is a serious commitment to designing in parallel with engineering. In waterfall design, of course, all the design and research are done up front and then thrown over the wall to engineering.

But Lean UX is different. In Lean UX, design and development are working at the same time. This can be tricky, of course, since a lot of people don’t understand how you can start to build something until you know exactly what you’re building.

Well, here’s how we did it at Food on the Table. Once we had done our very fast, initial user research to understand the reasons for our activation problem, we came up with a lot of different fixes we knew we wanted to make. We then split those fixes into three different types: Fix Now, Redesign, and Iterate later.

The fix now problems were low hanging fruit. Those were usability bugs (or plain old BUG bugs) that could be addressed immediately by the engineering team without waiting for design. That meant that the users (and the company!) started seeing improvements immediately, rather than having to wait for a single, massive rollout of all the changes. Why make small changes wait on big changes?

Also, when it came to the stuff we were going to redesign completely, we didn’t wait for everything to be perfect before sharing the new design with engineering. While we were working on things like button placement, page layout, copy writing, or visual design, they could get started with the major structural changes and backend improvements that would be needed.

By overlapping design and development in this way, the whole project moved much faster.

Planning for Iteration

Perhaps the biggest difference with Lean UX vs User Centered Design, is planning for iteration rather than trying to include all the new features at once.

As I mentioned before, we separated our changes into three buckets: Fix Now, Redesign, and Iterate Later. That last one’s important, since it means that you are getting a good version of your design in front of users quickly in order to learn from it and optimize it rather than trying to come up with a perfect version of the design before building anything. This should sound pretty familiar to all of you.

Here’s an example. Food on the Table lets people choose recipes and add them to a meal plan. We found, during testing, that users enjoyed choosing from a selection of recommended recipes.

But that led to questions. How many recipes would users want recommended? Would they like to see recipes that their friends were recommending? Did they want to see recipes they’d recommended to others? Would they like to see them in a carousel or a list or a coverflow?

Luckily, these were all questions that could optimized later with a/b tests, once we’d implemented the major structural change, which involved letting people select recommended recipes at all.

Why was this important? Well, it meant we didn’t have to guess or debate or spend a lot of time prototyping and testing the exact right number and content of the recipes page before launch. We could concentrate on getting the user flow and the main interactions right and improve the rest later, once we’d validated that our larger design assumptions were correct.

So What Is It?

So, what makes Lean UX lean? Essentially, it’s about more than just applying good design principles to a Lean Startup, although, that is obviously important. It’s also about applying Lean principles to the design process.

Lean UX incorporates quantitative & qualitative research, it overlaps the design and development phases of the project, and it starts small and plans for iteration. And, I think, that all those things together with a great, user centered design process allow Lean UX to deliver huge value to both users and startups very quickly.

Like the post? Follow me on Twitter!

Wish you'd heard the talk? Don't miss the next one! I'll be speaking and running a workshop at Web 2.0 Expo in San Francisco on March 29th.

When Is a Design Done?

2011-03-10T11:53:00.000-08:00

I was talking with a designer about Lean UX. I was explaining that one of the hallmarks of Lean UX is to get a good, but not complete version of a product or feature designed and built and then iterate on it later. She thought this sounded like an interesting approach, but then she asked, “When do you know you’re done?”

Figuring out when you’re “done” is tricky for any design or redesign project, unless you’re a consulting agency, of course, in which case the answer is, “when the client runs out of money.” But I realized that, in Lean UX, figuring out when you’re done is actually incredibly easy.

You’re done when your metrics tell you you’re done.

Let me explain. No product is ever actually “done.” There is always something you could do to improve it. However, projects can certainly be done. The trick is that you have to choose your projects correctly.

What’s the correct way to choose a Lean UX project? Every Lean UX project should be chosen based on a metric.

This may piss off a lot of designers who want to make wonderful, exciting, super cool designs just for the sake of design or user happiness, but when it comes down to it, unless you’re independently wealthy, every design change you make should move a number that is important to your business.

Now, it is a lucky break for those of us who care deeply about our users that improving the overall user experience of the product frequently improves some number that the business people care about. But not every single thing you can do to make a user happy has the same ROI for the business. And not every improvement makes the right people happy at the right time.

That’s why the UX projects you choose should be based on metrics.

Let me give you an example. Whoever it is at your startup who is in charge of running the business should have a pretty good idea of what your various metrics have to be in order for you to all retire and buy yachts. For example, your Activation number may have to be 20% and your Retention may have to be 70%. (Please note, I made these numbers up. Your metrics may vary.)

They pick these numbers because they know that having, for example, a 99% retention rate and a 1% activation rate may lead to retaining 3 incredibly happy users forever, which is suboptimal from a business perspective.

So, if your activation number is at 10%, your business folks may come to you and say, “we need to turn more of our acquired traffic into regular users because we have identified this as the most important problem to solve at this moment.” You respond, “Great! How many more do you need?” They explain that you need to get activation from 10% to 20%.

You will notice that the metrics are not driving your design decisions. Nor are they driving your feature requirements or any other product changes. They are simply telling you what your biggest business problem currently is.

Now, it’s up to you as a designer or product owner to figure out what is keeping the activation number low and then come up with some ideas of how to fix it. You do this with what I like to call “research and design” or alternately “that thing you are paid to do.”

You may have dozens of wonderful ideas for how to fix the problem, and you may love and believe in all of them. You may not, however, actually execute every single one of them.

This is where the Lean part comes in.

Ideally, you will design and execute as many of the fixes as necessary in order to move the number to where you want it to be. Maybe you’re awesome (or awesomely lucky), and you move that activation number on the first try with a very small bug fix.

Does that mean you never get to implement the super sweet, but somewhat complicated, feature that you know will make users incredibly happy and improve activation even more? No! Unfortunately, you may not get to implement it just yet.

You see, once you got your activation number to where it needed to be, it stopped being the most important problem to solve. Now, maybe you need to work on getting retention higher or improving revenue or referral.

On the flip side, maybe you redesign the first time user flow and improve activation, but not by enough. That means you should continue working on it. Figure out why your changes didn’t have as big of an impact as you thought they would, and then try some new things.

You’re not “finished” until your metrics are where you want them to be.

Why is this important? Startups have a ridiculous number of things to do, and they typically have limited resources. It can be incredibly difficult to prioritize when to keep working on a feature or an area of the product, and when to move on.

By setting the goals ahead of time based on metrics that are critical to the business, it becomes much easier to know when you’re “done,” and when you should keep optimizing or redesigning.

Like the post? Follow me on Twitter!

Like Lean UX but hate reading? I'll be on the UX panel at the Lean Startup track at SXSW. You should come see it and then say hi to me afterward.

Qual vs. Quant: When to Listen and When to Measure

2011-02-28T11:31:00.000-08:00

I have written about qualitative vs quantitative research before, but I still get a lot of questions about it. To answer some of those questions, I want to do a bit of a deeper dive here and give a few examples to help startups answer the key question.

To be clear, that key question is “when should I use qualitative research, and when should I use quantitative research for the best results?” Another way of looking at this is, “when should I be listening to users, and when should I just be shipping code and looking at the metrics?”

The real answer is that you should do both constantly, but there are times when one is significantly more helpful than the other.

I will continue to repeat my cardinal rule: Quantitative research tells you WHAT your problem is. Qualitative research tells you WHY you have that problem.

Now, let’s look at what that actually means to you when you’re making product decisions.

A One Variable Change

When you’re trying to decide between qualitative and quantitative testing for any given change or feature, you need to figure out how many variables you’re changing.

Here’s a simple example: You have a product page with a buy button on it. You want to see if the buy button performs better if it’s higher on the page without really changing anything else. Which do you do? Qualitative of quantitative?

That’s right, I said this one was simple. There’s absolutely no reason to qualitatively test this before shipping it. Just get this in front of users and measure their actual rate of clicking on the button.

The fact is, with a change this small, users in a testing session or discussion aren’t going to be able to give you any decent information. Hell, they probably won’t even notice the difference. Qualitative feedback here is not going to be worth the time and money it takes to set up interviews, talk to users, and analyze the data.

More importantly, since you are only changing one variable, if user behavior changes, you already have a really good idea WHY it changed. It changed because the CTA button was in a better place. There’s nothing mysterious going on here.

There’s an exception! In a few cases, you are going to ship a change that seems incredibly simple, and you are going to see an enormous and surprising change in your metrics (either positive or negative). If this happens, it’s worth running some observational tests with something like UserTesting.com where you just watch people using the feature both before and after the change to see if anything weird is happening. For example, you may have introduced a bug, or you may have made it so that the button is no longer visible to certain users.

A Multi-Variable or Flow Change

Another typical design change involves adding an entirely new feature, which may affect many different variables.

Here’s an example: You want to add a feature that allows people to connect with other users of your product. You’ll need to add several new pieces to your interface in order to allow users to do things like find people they know, find other interesting people they don’t know, manage their new connections, and get some value from the connections they’ve made.

Now, you could simply build the feature, ship it, and test to see how it did, much the way you made your single variable change. The problem is that you’ll have no idea WHY it succeeded or failed - especially failed.

Let’s assume that you ship it and find that it hurts retention. You can assume that it was a bad feature choice, but often I find that people don’t use new features not because they hate the concept, but because the features are badly implemented.

The best way to deal with this is to prevent it from happening in the first place. When you’re making large, multi-variable changes or really rearranging a process flow for something that already exists on your site, you’ll want to perform qualitative testing before you ever ship the product.

Specifically, the goal here is to do some standard usability testing with interactive prototypes, so that you can learn which bits are confusing (ps. yes, there are confusing bits, trust me!) and fix them before they ever get in front of users.

Sure, you’ll still do an a/b test once you’ve shipped it, but give that new feature the best possible chance to succeed by first making sure you’re not building something impossible to use.

Deciding What To Build Next

Look, whatever you take from this next part, please do not assume that I’m telling you that you should ask your users exactly what they want and then build that. Nobody thinks that’s the right way to build products, and I’m tired of arguing about it with people who don’t get UCD or Lean UX.

However, you can learn a huge amount from both quantitative and qualitative research when you’re deciding what to build next.

Here’s an example: You have a flourishing social commerce product with lots of users doing lots of things, but you also have 15 million ideas for what you should build next. You need to narrow that down a bit.

The key here is that you want to look at what your users are currently doing with your product and what they aren’t doing with it, and you should do that with both qualitative and quantitative data.

Qualitative Approaches:

Watch users with your product on a regular basis. See where they struggle, where they seem disappointed, or where they complain that they can’t do what they want. Those will all give you ideas for iterating on current features or adding new ones.
Talk to people who have stopped using your product. Find out what they thought they’d be getting when they started using it and why they stopped.
Watch new users with your product and ask them what they expected from the first 15 minutes using the product. If this doesn’t match what your product actually delivers, either fix the product or fix the first time user experience so that you’re fulfilling users’ expectations.

Quantitative Approaches:

Look at the features that are currently getting the most use by the highest value customers. Try to figure out if there’s a pattern there and then test other features that fit that pattern.
Try a “fake” test by adding a button or navigation element that represents the feature you’re thinking of adding, and then measure how many people actually click on it. Instead of implementing an entire system for making friends on your site, just add a button that allows people to Add a Friend, and then let them know that the feature isn’t quite ready yet while you tally up the percentage of people who are pressing the button.

Still Don’t Know Which Approach to Take?

What if your change falls between the cracks here? For example, maybe you’re not making a single variable change, but it’s not a huge change either. Or maybe you’re making a pretty straightforward visual design or messaging change that will touch a lot of places in the product but that doesn’t actually affect the user process too much.

As many rules as we try to make, there will still be judgement calls. The best strategy is to make sure that you’re always keeping track of your metrics and observing people using your product. That way, even if you don’t do exactly the right kind of research at exactly the right time, you’ll be much more likely to catch any problems before they hurt your business.

Like the post? Follow me on Twitter!

What Does Your User Know?

2011-02-24T12:05:00.000-08:00

You and your customer have very different sets of information. This shouldn’t come as a surprise to you, since by now it should be obvious that you are not your user.

Sometimes it can be very hard to distinguish what you know about your product or industry from what your user knows. But it’s important! Making assumptions about your user’s knowledge can lead to products that are impossible for normal people to use.

You Know...

The Details of How Your Product Works

You know all the technical and implementation details of your product. Your user doesn’t even know what those things mean.

One company I worked with had a feature that allowed users to mark off all the items they owned from a list. The engineers knew that, when a user made a selection, that selection was sent to the server in an AJAX request, so the account was kept constantly up to date.

The users didn’t know this. During testing, several users went through, marked off all their selections, and then searched in vain for a Save button. They assumed that they would have to save their work, since they had no idea about the AJAX request that was happening in the background.

The solutions to this were either to allow the users to explicitly save with a button or to give them a tiny amount of feedback while the item was being saved via a very brief wait spinner.

Once we implemented the latter solution, users immediately understood that each click was actually saving the item automatically, and they no longer looked for that Save button.

The take away: Don’t make any assumptions that your users understand what you’re doing for them unless you’re explicitly telling them in some way.

Every Single Feature and Its Purpose

You designed and built every feature in your product, so you know exactly what each of them does and where to find it in your product. Your user knows only what she finds during her time using your product.

One company I worked with had a very useful feature. When I interviewed users about new features they’d like to see, many of them requested the feature, even though it was already in the product! They simply had no idea that it even existed.

The take away: It’s not enough to create fabulous new features. You have to make sure that your features are discoverable by normal users.

Specialized Knowledge About the Product Space

Sometimes we build products to help people do hard things more easily.

Tax preparation software is a fantastic example of this. It is safe to say that most of us who use tax preparation software do not know nearly as much about tax preparation as the people who are building the software. At least, I hope they know more than I do!

The problem arises when we lose touch with exactly which parts of the space the users understand well. We can start to use jargon or terminology that makes perfect sense to us because we hear it all the time. We can design processes that seem completely reasonable if the user already knows the goal.

Unfortunately, this creates complicated, confusing products that assume a much higher level of understanding than the user has.

The take away: When you’re trying to help users accomplish a complicated goal, you need to work even harder to keep the interface simple.

Of course, your customer knows some stuff you don’t know...

How She’s Used to Doing Things

If you’re creating a product that is meant to help a users do something they already do (again, tax preparation or any business software), your goal is to create a generic experience that will satisfy as many people in your core demographic as possible.

But remember, each of those users already does things slightly differently. For example, if you’re helping people who sell things on eBay, you have to understand that each seller already has a process that she follows - from pricing to listing to shipping to dealing with customers.

Asking your users to change too many of their behaviors in order to use your product creates a huge barrier to acceptance.

If you only talk to a few potential customers (or, even worse, none at all), you run the risk of creating something that isn’t broadly usable by all sorts of different users.

The take away: Understanding the variations in user behavior will help you deliver something that is usable by a larger segment of your user base.

Specialized Knowledge About the Product Space

Hang on! Isn’t this something that you have that your user doesn’t have? Actually, yes, but you’re not the only one who knows something special about your product space. Your users may have specialized information about your product that you don’t have.

Imagine you’re creating an application for rating wines (or coffee or chocolate or movies or anything else). You may have very specialized knowledge of how wines are made or how the selling process works or how ratings systems are best implemented.

But your users will likely have very specialized knowledge about various types of wines. They may have quite different opinions about categorization or information display because of this specialized knowledge.

By taking advantage of your users’ knowledge, you can have them help you improve your product without your having to first learn everything there is to know about your product space.

The take away: You don’t need to know everything about your product space, but you should take advantage of the expertise of your users to improve the product and continue learning.

Why Is It Important to Know What Your User Knows?

It’s important to learn what your customer knows so that you’re not creating unnecessary confusion. A product should work with a user’s expectations and understanding rather than demanding the user learn a complicated new way of doing things.

How Do You Learn What Your User Knows?

By listening to them, of course. Having conversations with your users about your product and how they use it will help you to understand exactly what they know and don’t know.

You might even learn something about your own product space, and you’ll definitely learn how to make your product better.

Like the post? There are more like it. You should follow me on Twitter!

Two Stupid Reasons for Complicated Products

2011-01-24T09:00:00.000-08:00

I frequently get asked by startups to simplify products. In general, companies are fantastic at coming up with great feature ideas, but they tend to find it harder to either kill underperforming features or properly integrate new ideas that got tacked on as an experiment or pivot.

Because I get called in when a company already has a product that new users find confusing, I see a lot of the same mistakes repeated. I also hear the same excuses for those mistakes.

Often, when I’m looking at a new product, I’ll find very similar features in different parts of the interface.

For example, one social product had three completely different ways of searching for friends the user might know. Now, I don’t mean that there were different criteria you could use - like email address or interests or user name. That would have been fine.

I mean there were three completely different places the user could go in the product to find three completely different features that were meant to help people search for their friends. There was huge functionality overlap among the three features, but they were all slightly different.

There’s a Reason For That

Of course, I mentioned that it seemed odd and confusing to have three different places to go to do essentially the same thing. And the product owner patiently explained to me that there was a reason for that.

The product owner then launched into a detailed description of how the first one had been built by the team as an experiment. A few months later, since the experiment went well, but was a little slow, the team migrated to a different technology for search, and built a second version of the feature alongside the first one to see if it could be faster.

Since the product owner hadn’t given them any requirements for the new version other than “go faster,” and the new technology made some of the old functionality tricky, the second version didn’t have exactly the same capabilities as the first. So, the team decided it wasn’t an adequate replacement for the old version. They released it anyway, since they’d already built it, and it was faster.

The third version of the feature had been built as part of a larger feature, but the rest of that larger feature had been killed, and only the search part remained. It had some neat new functionality that users liked, but the team felt it didn’t really replace either of the other two versions.

Moreover, the product owner explained to me that different types of users liked the various different versions of friend search, so he was hesitant to kill any of them, because whichever one he killed would upset somebody.

So, he was stuck supporting three different variations of the same feature, and new users were overwhelmed by choice for where they were supposed to go to find their friends.

And here’s the thing: users don’t actually want this sort of choice. This sort of choice is confusing. It makes navigating the product cumbersome, and it’s unpleasantly surprising to constantly find new, slightly different ways to do the same thing.

There’s a Big Difference

This isn’t the only way that the problem manifests. Sometimes when I point out similar features in different places, the product owner reassures me that “there is a big difference” between the features.

This happened with a client when I pointed out that there were two different types of quizzes in one section of the product, but they had different names and placements. I asked why they couldn’t be combined into one section, so that the clutter on the page would be reduced and the user could always know where to go to take quizzes.

He assured me that there was a big difference between the two types of quizzes. When I asked him to elaborate, he went into detail about how the types of quizzes were different on the back end, and how one was often used as a business development tool while the other was user generated.

In other words, both of the “big differences” were things that were only different to the company. They were the sorts of differences that a user would never notice. All a user would notice would be that the quizzes were sometimes in one place and sometimes in another, which would be confusing and frustrating if she was looking for that feature.

How to Avoid This

Take a hard look at your product. Are there any similar features? You’ll be surprised at how often the answer is yes. If there are, ask yourself what your reasons were for building the different variations.

If your reasons for building different versions of the same feature are based entirely on technology or business development (in other words, things that only matter to YOU and not your users), you’ve got a problem.

The most customer friendly way of dealing with it is to try to come up with a superset of the best functionality from all the versions and improve one of the versions until it satisfies as many user stories as possible.

But even if you don’t have the time or resources to consolidate them into one great version of the product, just killing the duplicated features will ultimately simplify your product and make it more useful and understandable for all of your customers.

Like the post? Follow me on twitter!

Lean UX - A Case Study

2011-01-18T08:52:00.000-08:00

For those very, very few (ok, none) of you who read my blog but don't read Eric Ries's blog, Startup Lessons Learned, I have some exciting news for you. But first, why the hell aren't you reading Eric's blog? You really should. It's great.

I've a written a guest post that now appears on the Startup Lessons Learned blog. It's a case study of a UX project I did with the lean startup Food on the Table.

If you're wondering whether design works well with lean startups, I answer that question in the post. Spoiler alert: The answer is 'yes'.

Testing Whether Your Users Will Buy

2011-01-06T11:19:00.000-08:00

As you all know by now, I’m a huge proponent of qualitative user testing. I think it’s wonderful for learning about your users and product.

But it’s not a panacea. The fact is, there are many questions that qualitative testing either doesn’t answer well or for which qualitative testing isn’t the most efficient solution. I cover some of them in my A Faster Horse post.

The trick is knowing which questions you can answer by listening to your users and which questions need a different methodology.

Unfortunately, one of the most important questions people want answered isn’t particularly well suited to qualitative testing.

If I Build It, Will They Buy?

I get asked a lot whether users will buy a product if the team adds a specific feature. Sadly, I always have to answer, “I have no idea.”

The problem is, people are terrible at predicting their future behavior. Imagine if somebody were to ask you if you were going to buy car this year. Now, for some of you, that answer is almost certainly yes, and for others it’s almost certainly no. But for most of us, the answer is, “it depends on the circumstances.”

For some, the addition of a new feature - say, an electric motor - might be the deciding factor, but for many the decision to buy a car depends on a lot of factors, most of which aren’t controlled by the car manufacturer: the economy, whether a current car breaks down, whether we win the lottery or land that job at Goldman Sachs, etc. There are other factors that are under the control of the car company but aren't related to the feature: maybe the new electric car is not the right size or isn't in our price range or isn't our style.

This is true for smaller purchases too. Can you absolutely answer whether or not you will eat a cookie this week? Unless you never eat cookies (I'm told these people exist), it’s probably not something you give a lot of thought to. If somebody were to ask you in a user study, your answer would be no better than a guess and would possibly even be biased by the simple act of having the question asked.

Admit it, a cookie sounds kind of good right now, doesn’t it?

There are other reasons why qualitative testing isn't great at predicting future behavior, but I'm not going to bore you with them. The fact is, it's just not the most efficient or effective method for answering the question, "If I build it, will they come?"

What Questions Can Qualitative Research Answer Well?

Qualitative research is phenomenal for telling you whether your users can do x. It tells you whether the feature makes sense to them and whether they can complete a given task successfully.

To a smaller extent, it can even tell you whether they are likely to enjoy performing the task, and can certainly tell you if they hate it. (Trust me, run a few user tests on a feature they hate. You'll know.)

This obviously has some effect on whether the user will do x, since they’re a lot more likely to do it, if it isn’t annoying or difficult. But it's really better at predicting the negative case (ie. the user most likely won't use this feature as you're currently building it) than the positive one.

Sometimes qualitative research can also give you marginally useful feedback if your users are extremely likely or unlikely to make a purchase. For example, if you were to show them an interactive prototype with the new feature built into it, you might be able to make a decent judgement based on their immediate reactions if all of your participants were exceptionally excited or incredibly negative about a particular feature.

Unfortunately, this, in my experience, is the exception, rather than the rule. It’s rare that a participant in a study sees a new feature and shrieks with delight or recoils in horror. Although, to be fair, I’ve seen both.

What’s the Best Way to Answer This Question?

Luckily, this is a question that can be pretty effectively answered using quantitative data, even before you build a whole new feature. A lot of companies have had quite a bit of success with adding a “fake” feature or doing a landing page test.

For example, one client who wanted to know their expected purchase conversion rate before they did all the work to integrate purchasing methods and accept credit cards simply added a Buy button to each of their product pages. When a customer clicked the button, he was told that the feature was not quite ready, and the click was registered so that the company could tell how many people were showing a willingness to buy.

By measuring the number of people who thought they were making a commitment to purchase, the client was able to estimate more effectively the number of people who would actually purchase if given the option.

The upshot is that the only really effective way to tell if users will do something is to set up a test and watch what they actually do, and that requires a more quantitative testing approach.

Are There Other Questions I Can’t Answer Qualitatively?

Yep. Lots of them. I’ll probably cover them at some point in the future if people are interested. Feel free to ask about other specific questions in the comments, and I’ll try to let you know what sorts of testing methods work best for answering them.

Enjoy the post? Thanks!
How about following me on Twitter?

You Need to Redesign Your Product

2010-12-17T13:18:00.000-08:00

Iterate, iterate, iterate. If there is something that I hear from lean startups more than “pivot” and “fail” it’s that iterating on your product is the way to improve it. And I absolutely believe that iteration is critical to making great features and products.

The problem is, sometimes just improving what you’ve got isn’t enough. Every so often, from a UX perspective anyway, you just need to throw everything out and start from scratch.

I’m not necessarily talking about reskinning the site with a new visual design, although that sometimes has to be done. Sometimes you also need to completely reorganize and refocus everything about your product’s user experience.

Of course, this can be incredibly expensive and time consuming, so it’s not something that you want to do unless it’s really necessary.

Here are a few signs that you may need to do a complete product redesign:

There’s No Room for Your New Feature

A big part of lean startups is coming up with lots of new feature ideas and throwing them in front of users to see what sticks. Another big part is killing the features that don’t make the cut, but that can be hard to do.

There comes a time in the life of every lean startup where they have a new feature that doesn’t quite fit within the navigation and structure of the rest of the product.

Often this new feature is not quite a pivot, but it may be the first step in that direction. Maybe you’re adding a social component to an ecommerce application or you’re adding games or a marketplace to a social site.

Or maybe you’ve just run out of room on your front page, and you simply can’t add another widget.

Whatever the reason, when you have a new feature that you can’t logically fit anywhere into your product, it’s probably time to do an overhaul, or at least a reorganization. It’s probably also a great time to go through and kill some of those underperforming features in order to make room for the new stuff.

You’ve Added a “Miscellaneous” Section to Your Navigation

Ever been tempted to add a section to your product navigation called “Misc.” or “Other” or “Stuff”? Yeah, we’ve all wanted to do it. DON’T.

Having a catchall in your product or site’s global navigation is a really good hint that something is terribly wrong with your information architecture. It may also mean that you have one or more features that don’t fit in well with the rest of your product.

This may mean that it’s time to rethink what your product offers to people or start killing (or spinning off) features that don’t fit. By focusing your offering, you will find that those random catchall categories magically disappear.

Your Users Aren’t Finding the Features You Already Have

You are measuring the adoption and user engagement for all of your current features right? So, you should know if there are features that are underperforming.

The important thing to remember is that sometimes features don’t do well not because they’re bad features but because nobody can find them in the confusing mess that your product has become over time.

You can figure out which it is by talking and listening to current users. If they’re constantly asking for features that you already have or if they get very excited when you describe existing features, you know that it’s time to do a product redesign so that they can find the things that you’ve already built for them.

Your Visual Design is Attracting the Wrong Audience

Sometimes a visual design change can make an enormous difference in attracting the right audience. If your product is meant to appeal to working moms, but in reality appeals to 15 year old gamers, you may just need to do a complete visual redesign.

The best way to test this is to get your product in front of some people in your target demographic who aren’t currently users. Just see how they react to it. Do they recoil in disgust? Do they appear disinterested? Do they say things like, “Oh, that’s not for me”? Then it’s probably time to reskin your product.

New Users Don’t Understand What Your Product Does

Another common problem with unfocused products is that new users just don’t get the value proposition. It may be tempting to try to address this with clever marketing messages or help pages or video tutorials, but this rarely fixes the real problem.

The real problem is frequently that there are simply too many things going on for new users to immediately grasp what the product can do for them. They start to explore, but they quickly become lost in a sea of unrelated features and inconsistent navigation.

By redesigning the UX, you can focus the navigation and features so that a new user can immediately understand what the product does for her.

Your Product Has Become Wildly Inconsistent

When you design and ship each feature separately, both the visual design and interaction can become really inconsistent. Button placements migrate, “Submit” becomes “Go”, and even navigation conventions can change.

Do this enough times, and it can feel like you’re looking at dozens of different products, which is extremely disorienting for your users.

This doesn’t always require a full redesign, but it does require a sweep of your entire product in order to make visual and interaction design details consistent and coherent.

When It’s Not Time

Because complete product overhauls can take a lot of time, it’s not something you want to undertake lightly. It’s not a panacea for a product that’s just not working.

I’ve frequently seen people who were simply out of ideas for engaging their users say that they needed to “redesign the whole thing” out of frustration or lack of vision.

But if your product has grown organically into a big, sprawling, inconsistent mess without a clear purpose or focus, it’s time to bite the bullet and redesign. Your users will thank you.

Like the post? Please, follow me on Twitter!

Want to read more? Check out these related posts:

The Importance of Predictability in Design

2010-11-30T12:01:00.000-08:00

If you watch a few user tests, there’s an excellent chance that, at some point, the user will point at some element of the product and ask, “What does this do?” If I’m moderating the test, it is almost guaranteed that I will respond with “What would you expect it to do?”

I’m not trying to be difficult when I ask that question. I’m trying to learn the user’s expectations, because this is an important thing to understand when evaluating the usability of a product.

In my many years of watching and running usability tests, I’ve noticed a pattern of behavior for users encountering a new screen:

Scan quickly
Click the first call to action that seems remotely relevant to the task at hand
Declare the product confusing if that call to action doesn’t do what they expect
Leave

There are three very simple things that you can do right now to improve this experience in your product.

Calls to Action Should Be Explicit

Users should always know, or at least have a good idea of, what will happen if they click on a call to action. One way to do that is to make the copy or icon associated with the call to action really obvious and descriptive while still keeping it concise enough that people have a chance of reading it.

Recently, a company was testing landing pages. There were two pages, and the ONLY difference between the two was the copy on a button. One said “Login” and the other said “Login with Facebook.” Both buttons popped open the Facebook Connect dialog asking them to enter their Facebook information.

The percentage of people who clicked on the button was not statistically different. However, nearly twice as many people who clicked on the “Login with Facebook” button actually completed the step and ended up logging in.

Why is that? Because they weren’t surprised by the Facebook dialog. The people who simply expected to log in to the site were thrown by suddenly being asked to enter their Facebook credentials. The other group understood what was about to be asked of them, and they continued on.

Ironically, some people who might have been happy to login in with Facebook most likely didn’t want to click the plain Login button at all, since they didn’t want to create a whole new account or weren’t sure they could log in if they didn’t already have an account.

Things That Look the Same Should Act the Same

Have a More Info link in a couple of places in your product? Make sure that every time somebody clicks it, the same thing happens. If it’s an inline popup with some help text in one place, make sure it isn’t a link to a different screen in another place.

One product for which I was conducting user tests had this exact problem in a very bad spot. On most screens, users would click a little help icon next to certain terms and get a nice, short, inline description of the thing they were looking at.

Unfortunately, in the checkout flow, there was a piece of information that the team felt would not fit inline. When users clicked that icon, they were taken away from the checkout flow to a completely different help system. This obviously interrupted the checkout flow, while annoying users who had to find their way back to what they’d been doing. This, unsurprisingly, reduced the number of completed checkouts.

Obviously the other take away here is not to interrupt the checkout flow by sending people elsewhere, but it’s also important to note that, if people had at least expected the behavior, they wouldn’t have had the additional unpleasant surprise of something happening that they didn’t expect. They also might have been less likely to click the icon, if they’d known it was going to pull them away from the expected flow.

Icons Should Be Standard

Do you have a little pencil icon next to some elements? If so, it should always mean “edit.” Sure, you could use some other icon, but why would you? You could also use a pencil to mean something else, but it would confuse the hell out of your users.

Certain icons have simply shown up in enough interfaces that they are as identifiable as actual words. When you change the meaning of those icons or use a different icon instead, you are making your users learn a whole new language. You wouldn’t use “blue” to mean “red,” would you? Treat icons the same way.

Why Is This Important?

Often, people who are building a product think, “well, this might be a little confusing the first time, but people will figure it out.”

There is an important truth to understand about your product: people almost never spend enough time with it to learn all the ways in which you’ve decided to be different or inconsistent.

If they are confused the first time, they will most likely continue to be confused or to make the same mistake over and over until they finally become frustrated and leave with the impression that your product is just too hard to use.

By making your product’s behaviors easy to predict, you are reducing the cognitive load for your users and making it easy to use, not just the first time, but every time.

Enjoy the post? Please, follow me on Twitter!

Or, read some of these related posts:

Can Something Be Too Easy?

2010-11-08T09:39:00.000-08:00

I spend a huge amount of my time trying to make things easier to use. Most of the time, the problem with products isn’t the idea, it’s that the people who are meant to use them just get horribly confused.

If you’d asked me last year, I would have said there’s no way to make something too easy.

However, I’ve recently seen a couple of examples that reminded me that there are some surprising dangers to simplification.

It Can Piss Off Certain Users

I’ve worked with companies who have a small but passionate group of power users who are deeply involved with the product and the community. The products that gather this particular sort of following tend to have a lot of user generated content.

Now, when you first start out, your tools for users may be…well, a polite word would be “rough.” A better way of putting it would probably be “pieces of crap.” And yet, if your offering is compelling enough, you will end up with a group of people who care deeply enough to put up with all of your bugs and failings and complicated interfaces, and they will create something awesome in spite of all that.

Of course, they will bitch and whine about how hard they have it, but when you actually take steps to make your product too easy to use – for example, if noobs can all of a sudden use templates to create something that used to take power users hours or days – the early adopters who put all that time into your product will scream. By making it possible for anybody to recreate their hard work, you’ve devalued their contribution.

This is especially true in companies where users can make money or gain points from user generated content. You see, your early adopters want all those hours they put into learning the tools to be worth something. They still want to be the highest ranked on the leaderboard and to make the most cash from selling their products.

What do you do about it? Honestly? This is a great problem to have. It means that you’ve taken something that’s been proven to be fun to do and made it accessible to a whole lot more people. The most important thing is to try to get your community onboard with the changes before they happen so that you reduce the outcry.

You also want to make sure that the old guard still feels valued and useful. For example, enlist their help in creating templates. Get their feedback on adding power user features to the new tools. Give them credit for creating content without using the tools.

Sure, you’ll lose some early adopters, but that was going to happen when you went mainstream, anyway. Your true power users may even learn the new tools and push the boundaries to create something even more awesome with them.

It Can Defy Expectations

Recently, I was working with a team developing a social marketplace where users can buy and sell products. One of our goals was to make the selling process as simple as possible so that anybody could sell anything.

Boy, did we succeed.

In fact, we were so successful that some users in testing simply didn’t know where to start. They all expected that they would have to find some place in the interface to start selling, set up an elaborate account with lots of information, and then put in a lot of effort describing the product they wanted to sell.

What they actually had to do was connect with Facebook, find the thing they wanted to sell in the comprehensive catalog, and make a few simple selections from drop down menus. When we showed them where to get started, they were absolutely floored by how simple it was.

Not only was our method easier than anything they’d ever used, it was also far easier than they expected. This was, of course, the crux of the problem. They expected selling to be hard, so they looked for a hard way to do it, totally overlooking the giant Sell This button on virtually every page.

What to do about it? One option is to stick to user expectations, even when users expect things to be harder than necessary. I’m not really a fan of this option though, since it hurts everybody at the expense of making things obvious to only very new users. In the case of my client, we went with very lightweight, simple user training.

We provided people an easy to locate place to get started selling and some simple contextual help that solved the problem. The key was, we didn’t make the process any harder; we just made the starting point more obvious to people so that they didn’t start looking for some more complicated solution.

It Can Hide Key Features

I know! I’m always harping on you to reduce the number of features you have in your product. And, nine times out of ten, it’s a much bigger problem to have too many features than too few. But there’s sometimes this weird compulsion to hide features from new users in order to make the first time user experience as simple as possible.

I can’t tell you how many times I’ve had clients ask if we should drastically alter the first time user experience so that many power user features are hidden.

Now, I’m all for streamlining the onboarding process or providing all sorts of help for new users, but don’t do it by hiding stuff. You’re just training your user to do things wrong or telling them that certain features don't actually exist in your product. And, of course, hiding things the first time around just means that your returning users are coming back to an entirely unfamiliar product, so they have to go through a first time user experience all over again!

What to do about it? I’m a big fan of in-product contextual help along with optional tutorials, if your product is really complicated enough to need that much help. This allows the user to jump right into the product and start poking around, which makes some user extremely happy, while providing just enough guidance for other users who want more help.

And honestly, sometimes the problem isn't too many features. It's that the features you have are all jumbled together in a confusing way. You can often simplify the design without simplifying the actual product.

How Easy Is Too Easy?

Unfortunately, this is going to vary from product to product. But the best way to find out if you’ve made something too easy is the same way that you find out if you’ve made things too hard. You listen to your users.

Now, get out there and simplify things. Just not too much.

Like the post? Follow me on Twitter, please!

The Dangers of Metrics (Only) Driven Product Development

2010-10-26T16:09:00.000-07:00

When I first started designing, it was a lot harder to know what I got right. Sure, we ran usability tests, and we looked generally at things like page counts and revenue before and after big redesigns, but it was still tough to know exactly what design changes were making the biggest difference. Everything changed once I started working with companies that made small, iterative design changes and a/b tested the results against specific metrics.

To be clear, not all the designers I know like working in this manner. After all, it's no fun being told that your big change was a failure because it didn't result in a statistically significant increase in revenue or retention. In fact, if you're a designer or a product owner and are required to improve certain metrics, it can sometimes be tempting to cheat a little.

This leads to a problem that I don't think we talk about enough: Metrics (Only) Driven Product Development.

What Is Metrics (Only) Driven Product Development?

Imagine that you work at a store, and your manager has noticed that when the store is busy, the store makes more money. The manager then tells you that your job is to make the store busier - that's your metric that you need to improve.

You have several options for improving your metric. You could:

Improve the quality of the shopping experience so that people who are already in the store want to stay longer
Offer more merchandise so that people find more things they want to buy
Advertise widely to try to attract more people into the store
Sell everything at half off
Remove several cash registers in order to make checking out take longer, which should increase the number of people in the store at a time, since it will take people longer to get out
Hire people to come hang out in the store

As you can see, all of the above would very likely improve the metric you were supposed to improve. They would all ensure that, for awhile at least, the store was quite busy. However, some are significantly better for the overall health of the store than others.

The same thing happens all the time when designing products. If your assigned goal is to increase the number of active users, there are lots of different design changes you could make, but not all of them will be equally effective for improving the actual goal, which is probably increasing the number of people who use the product and generate revenue.

How Does This Happen?

I think the biggest reason that this happens is that people fixate on metrics without understanding the reason behind the numbers. Designers and product owners are then pressured to move a number that represents a particular metric rather than focusing on improving the product.

One company I talked with had this problem with acquisition. The person who was responsible for acquiring new customers was simply given a budget and told to get as many users as possible for that amount of money. Unfortunately, the users that were cheapest to acquire were the least likely to spend money on the site. If, instead of trying to maximize the number of users, he had concentrated on maximizing the number of users who were likely to spend money, he would have acquired fewer people and missed his metric, but he would have increased revenue.

Another company had a design problem. They wanted to redesign their Invite a Friend feature to encourage people to invite more friends. Unfortunately, the "most effective" method of getting people to invite friends was to forcibly spam users' Facebook feeds and make it easy for users for accidentally invite everybody in their address books. While this resulted in more invitations sent, it also vastly increased the number of unhappy customers and decreased the percentage of invitations that were accepted. It also caused the company to be banned from Facebook and put on the spam list of several ISPs. It sure improved that invitation metric, though.

How Should You Avoid It?

There are three ways to avoid this problem, and you should use all of them.

Make sure that you're measuring the right metric.

If you care about revenue (and you should), measure revenue. If you care about retention, measure retention. If you care about page views, you're probably doing something wrong.

Unfortunately, it can be difficult to immediately see the impact of a particular design change on things like revenue and retention, which makes it tempting to use substitutes for the important number.

If you are using a substitute - for example, if you're using something like "customers returning once" as a shorthand for "becoming an active customer" make sure that the link is actually causal. In other words, if you can increase the number of people who come back once, make sure that that really does lead to an increase in people who become an active customer.

Make sure that you're not gaming the metrics.

Paying people to come back to your site may result in more returning customers, but it doesn't necessarily result in more customers paying you. If you cut your prices in half, you may end up selling twice as many items, but you're not making any more money. Make sure that, if you're moving a metric, you understand the second (and third and nth) order effects of whatever change improved the metrics.

Make your customer experience better.

This may seem obvious, but pissing off your customers is a terrible long term strategy, even if it briefly moves a metric. On the other hand, improving your customers' overall experience leads to happy, contented customers who stick around and continue to pay for your service. Over time, that's going to improve all of your metrics.

And always remember, metrics are just shorthand for real customer behaviors that are important to your business. They are a tool to help you understand your product, not a goal to be met at any cost.

Like the post? Follow me on Twitter.

A Review of UserTesting.com

2010-10-21T12:27:00.000-07:00

A lot of people recently have asked me about the new crop of user testing tools available on the internet. One specific tool that comes up a lot is usertesting.com, and I’d like to talk a little bit about my experience with it.

Frankly, when I first saw the site, my initial reaction was, “Well, that’s not going to be very useful.” I’ve heard a similar gut reaction from several user researchers. Once I’d given it a try, my reaction changed to, “Oh, shit. This could seriously cut into my income.”

Having used it several times now, I can happily say that neither of these reactions was correct.

The Cons:

Let’s start with the things that I originally noted about usertesting.com that made me think it wouldn’t be very useful.

There is no moderator.

Not having a moderator means that there isn’t a human being running the test who can ask follow up questions and delve deeply into issues that come up naturally during a session. Good moderators don’t just follow a script; they ask the right questions to really understand why users are doing what they’re doing.

It’s less useful for testing incomplete interactive prototypes.

I have not yet found a good way to test prototypes with usertesting.com. When I design, I create very sketchy, incomplete mockups of products. Typically, these only run in one browser, they have no visual design, and large parts of them won’t really work or will use fake data.

I can deal with all of these issues in a one-on-one session by explaining that the prototypes are not real, giving the participant some help in areas where the prototype isn’t perfect, or gently reminding them not to fixate on the visual design. This isn’t really possible without a human moderator.

Iteration between sessions can be challenging.

Often, when I’m testing, I’ll schedule a break of a few hours or overnight between the first few participants and the last few. This gives me time to make fast changes to the prototype. Let’s say that the first couple of sessions show a really big problem. By quickly fixing that problem, I can get users past the problem and find the next biggest problem. Other times, I’ll actually change the discussion guide on the fly because something interesting comes up, and I want to learn more about it.

If you want to iterate between sessions with usertesting.com, you have to request a few users, wait for them to finish, watch the videos, and then decide whether to get more users or make a change. It’s not impossible, but it’s somehow doesn’t flow as naturally as it does during a standard series of sessions.

The Pros:

There is no moderator.

Hey, wait a minute! That was a con, too! Yep, it was.You see, great moderators are fantastically helpful for user testing. Unfortunately, most untrained people are terrible moderators. They lead, they explain, they won’t shut up…basically they do everything you can imagine to ensure that you’ll have a biased test.

If you don’t have somebody who is great at moderating user sessions, it’s very possible that you’ll be better off without a moderator at all. At least nobody will be actively screwing up your sessions and ruining your results.

It is incredibly fast and cheap.

User testing can be expensive and time consuming. Recruiting alone can cost hundreds of dollars. Often, recruiting firms want at least a week to line up participants, and you always have to schedule extra people to make up for no-shows and unexpectedly crazy people. (Note: My rule of thumb is that out of about every 8 people, you’ll get two no-shows and one crazy person. Your mileage may vary.)

This is simply not true with usertesting.com. Set up your test, pay a ridiculously small amount of money per participant, go out for a longish lunch, and come back to completed videos of people using your product. There’s no such thing as a no-show, and if one participant is crazy, you can give them a bad rating and have another one in a matter of hours.

Seriously. This is the feature that totally sold it for me. It reduced my turn around time on many tests from two weeks to overnight.

It is very easy to share the videos.

You know who should be watching user tests? EVERYBODY IN YOUR ORGANIZATION. You know who typically watches user tests? The moderator. Sometimes you get a product manager or an engineer in one or two sessions, if you’re lucky or you offer bribes.

But sending out a link to online videos of people using the product vastly increases the number of people who will see at least one user test. And yes, you could video tape or screen capture in-person tests, convert then to video, post them on a server and send the information to people, but this is just much easier.

When I’d Use It

So, what’s the upshot? Should you use usertesting.com or not? I would say sure. Just use it for very specific purposes.

For example, it’s fantastic when you want to just get an overall “state of the product” idea of how people are using specific parts of your product.

Another great use of it is for spinning up a new employee or making sure that current employees understand how people are using new features. Sitting down and watching three or four videos pretty quickly gets employees to feel their users’ pain.

When I Wouldn't Use It

There are times I wouldn’t use it, too. For example, I wouldn’t use it to try to get feedback on in-progress wireframes. It’s just too hard to get people to understand the limitations of an incomplete prototype without a moderator.

I also wouldn’t use it in any sort of test where I wanted to do a really deep dive into a particular experience or get specific sorts of feedback from a user. Sometimes, especially in early research, I’m not entirely sure what I want to learn, and being able to sit and explore with a participant for an hour or two is tremendously valuable. Usertesting.com doesn’t give you that, and that’s ok. That’s what researchers are for.

I’m not really sure how I’d use it to do a comparison of various different versions of features either. If I wanted to show someone three different possible implementations of a feature to see which they preferred, I can’t think of a way to do that with usertesting.com. I’d be thrilled if somebody had a clever way to do it though, so feel free to share it with me in the comments.

The Final Word

So, is it going to replace great user researchers? Absolutely not. Is it going to be a great tool that researchers can use in specific circumstances to make certain types of testing faster and cheaper? Absolutely!

Perhaps the best argument for using usertesting.com is that you can introduce it into companies that aren't currently doing any user research at all. The next time somebody in your organization says they don't have the time or budget for user testing, laugh maniacally and prove them wrong.

Like the post? Follow me on Twitter!

Pie-Jacking and Other Tips for Making Engineers Tolerate UX

2010-10-11T11:13:00.000-07:00

Obviously, I find UX to be incredibly important. And these days, I’m finding more and more people who agree with me. Unfortunately, in many organizations, there are people who still feel that user research and interaction design slow things down or aren’t really necessary.

Sometimes, those people are engineers. Not always, of course. I’ve worked with lots of engineers who are very excited about the idea of good design and getting user feedback. But occasionally you run into groups of engineers who have yet to be convinced.

In the interest of achieving harmony between the Engineering and UX departments at your company, here are a few tips for convincing people (especially engineers) of the value of user research and design.

To be clear, I am in no way suggesting that you trick engineers or lie to them or manipulate them in any way. I’m working on the assumption that engineers are frequently extremely logical people who just need evidence that things are useful before they buy in.

Involve Them in the User Research

The number one way to get anybody excited about user research and UX is to get them involved with it. Too frequently, the only thing about user research that engineers get to see is a thirty page paper detailing all of the problems with their product. This is boring, painful, and easily ignored.

In my experience working with dozens of companies, the single most powerful tool for getting engineers in touch with users was having them sit in on a few usability sessions. The sessions didn’t have to be formal. I’d just put the engineers in chairs in the back of a small conference room and have a participant use the product in front of them.

Without fail, the engineers started to understand the pain that their users were feeling. And, since engineers are not monsters, they wanted to help those people. They would fix bugs they observed during the sessions. They would ask for advice on how to improve screens that they had previously thought were just fine.

Most importantly, the engineers would suddenly understand how different they were from their users! Suddenly, it became much harder for the engineers to believe that they had all the answers to their customers’ problems, since they had seen that they were really quite different.
Work in Parallel
One of the valid problems that engineers often have with design is that they feel that it keeps them from getting to work and compresses their schedules. After all, if product management has decreed that the product must be out by Christmas, every week spent on design is a week that the engineers are not writing code.

Now, I’ve written before about the ways in which design can actually speed up the development process, but it’s true that inserting a long design process between product definition and engineering can extend a project.

So, don’t do that. Start the research and design process while product management is still determining the features. In fact, good UX practices and customer research make feature definition much faster and more accurate. Then make sure that you’re working with the engineering team so that they can start building parts of the design before it’s done.

Look, I know that you want everything to be pixel perfect before you show your design to another living soul. Get over that. You need to get engineering involved with your design process for so many reasons, not the least of which is that they can actually get started building it, if they have some idea what’s coming.

Give Them Useful Deliverables

Ever written one of those three hundred page design specs? The one that you have to update every time anything changes? The one that has highlighting in seventeen different colors meant to indicate which design changes are new and which were made for version 4.3.2.4.3.53 of the document? STOP. FUCKING. WRITING. THOSE.

I’m sorry to channel Dave McClure there, but seriously, those sorts of things make me homicidal, and I don’t blame engineers for hating designers who create them.

What is a useful deliverable? Interactive prototypes are great, since they show the engineers exactly how things should work. Once you’ve got an interactive wireframe, you can make a few notes at the bottom of each screen explaining anything that might be confusing or any edge cases. Index cards with specific design tasks are also super useful in an agile environment.

Another thing that’s useful as a deliverable is your time. In other words, don’t dump a spec on engineering and then run off to another project. Make sure that you’re around to answer questions and troubleshoot problems that come up. Being available to answer questions for engineering makes their development process go much faster and prevents simmering resentment and anger when they can’t figure out why you designed some stupid animation that’s going to take them three weeks to debug on IE6.

Understand Their Criticism and Respect Their Time

Hey, you know that really complicated design you created with eighty seven different edge cases, each with its own gradient and scalloped corners? Yeah, somebody has to implement that, and it isn’t you. When engineers push back on your design work, it’s not always because they’re tasteless heathens. Sometimes it’s because what you’ve designed will cause them no end of pain and suffering for relatively little benefit to the user.

Listen to their arguments and be ready to compromise. I’m not saying you should accept crappy, stripped down versions of your designs. I’m saying work together to come up with good, usable alternatives that will deliver value to the user without causing anybody in the engineering department to have a nervous breakdown.

Show Them How Good Design Prevents Rework

Remember those interactive prototypes I talked about? Those are great for demonstrating how iterating in the design phase keeps engineers from having to constantly change what they’ve already built.

Here’s how it works. First, you build some interactive wireframes of what you would normally have engineering build. Then, you run a usability test and catch any problems with the design. Maybe you’re such a fabulous designer that you never catch any little issues in your usability testing, but the rest of us mortals often find stuff that can be improved and fine tuned.

Then, you fix the wireframes and iterate as many times as you need to until you have something that people like. Let the engineers watch the testing process. When the engineers get the wireframes, they’ll see all the changes you made in the design phase that they won’t have to make in code after shipping.

Speaking from experience, try to refrain from smugly saying, “You’re welcome,” at this point.

As a Last Resort…Pie-Jacking

I am not proud of this last method of motivating engineers, but it’s probably the one that has worked the best for me. At IMVU, I will admit that I frequently resorted to bribing engineers with snacks.

This not only made it so that the engineers were happy when I walked into their area (as long as I had a chocolate chip cookie in my hand), it also lured them to presentations on UX, usability testing sessions, and other design related activities. “Let’s go to Laura’s brown bag,” I imagine them saying. “What’s she talking about?” “Who cares? She brought cookies!” I would then talk about UX for as long as it took them to eat the cookies. Note: bring a lot of cookies.

It got quite blatant. Engineers had a lot of autonomy about what they built and released at IMVU. A well-timed pie ensured that my features made it to the top of the list. This happened so frequently that they started referring to it as pie-jacking the development process, which is considerably less dirty than it sounds.

I guess what I’m saying is that building bridges between Engineering and UX is important. If all else fails, try bribery.

Like the post? Want more like it? Follow me on Twitter!

6 Incredibly Important Lean UX Lessons

2010-10-05T17:21:00.001-07:00

A project I’ve been working on recently should really be featured in all books on agile and lean design. I’ve found that many projects and clients have similar issues with their design processes, but it’s rare that a single project clearly demonstrates so many incredibly important tenets of lean UX.

So far, over the course of about a month, this one has driven home all of the following lessons:

You shouldn’t be a slave to your metrics, especially when they’re telling you something surprising
Qualitative testing is necessary for understanding why your metrics are what they are
Iteration in the design process is critical
Very small changes can make a huge difference
You need to understand the actual experience your user is having with your product
Don’t give up on your vision too early!

So, what happened?

I’m working with a very cool startup that is absolutely devoted to metrics. They A/B test and measure everything. When they first brought me on, the team went over some history of the product.

They originally had a particular flow for the product, but when they did qualitative testing, it became clear to them that users expected things to behave differently. The team designed a new version of the product that they felt addressed the issues that testers were having. They then built and released the new version.

It bombed. Well, perhaps that’s a bit strong, but it performed significantly worse than the original design in an A/B test.

They were surprised, but the data had spoken. The team reasonably assumed that their original reading of the qualitative testing results had been flawed, and that users really didn’t want this new flow. Since this had gone so badly, they then decided to hire a designer to help them come up with a better version.

That’s where I came in. The problem was, when they showed me the two different versions, I quite honestly could not understand why the new version had not crushed the old version in the test. It was obviously better!

I decided I needed to do some user testing to understand why their users would prefer what I felt was obviously the worse option. Was I misunderstanding the audience? Was the test flawed? Were they insane?

Over a couple of days, we observed quite a few test participants going through the two different new user flows. Because the company has users all over the country, we did a lot of remote testing and made use of usertesting.com so that we could observe people in various locations using their own computers.

The results were astonishing. It turned out that there was a strange bug that happened every time a session participant tried to use the new flow. There was one action that the user had to perform a few times, and every time they did it, there was a lag of a few seconds during which the user got no feedback. This led to multiple button clicks which led to a confusing experience when the clicks finally registered.

This problem hadn’t shown up in testing onsite, since the company's office had very powerful machines and a lot of bandwidth, so the team never saw the lag internally.

Instead of redesigning their entire site, I encouraged the company to fix this small bug to see if they could get a fair test and determine whether the new flow really was a better direction for the product than the old flow. Once we’d determined which flow was a better direction, we could make our design changes based on a better understanding of what their users really wanted.

The team fixed the bug and released the new experiment. While the final results still aren’t in, the new design is now trending higher than the old design, rather than the other way around. More importantly, it’s clear that fixing that one small bug had a significant impact on the performance of the new design.

So, let’s look at what we learned:

Don’t Be a Slave to Your Metrics

In this case, the A/B test was quite clearly showing the old version as the winner. But that simply didn’t tell the whole story. If the team had simply believed their metrics rather than investigating a surprising outcome, they would have abandoned a very promising direction for their product.

Even worse, they might have assumed that the new version failed because users didn’t like the new flow, which might have led them to make other similar decisions based on a flawed assumption.

Qualitative Testing is Necessary for Understanding Metrics

As I said in my Web 2.0 Expo talk, quantitative metrics only tell you what your problem is. To understand why you’re experiencing that problem, you need to observe users with your product in person.

Quantitative metrics tell you WHAT. Qualitative research tells you WHY.

Iteration in the Design Process is Critical

When you’re testing an entirely new version of something against an older version that may have already been optimized and tested, it’s often necessary to go through a few versions of the new design even to get it to parity with the old version.

Sometimes a new design wins the test right out of the gate, but not always. Spending a little bit of time understanding its performance and iterating on the design can deliver something much better in the long run.

Small Changes Make a Huge Difference

A very small bug nearly sunk a whole new design approach. The fix was to provide immediate feedback about a button press with a small “loading” animation. It wasn’t quite a one–line fix, but it was awfully close, and it made the difference between failure and success.

You Need to Understand Your User’s Experience

I’ve said it before. Just as you are not your user, your machine is not your user’s machine, and your bandwidth is not your user’s bandwidth, etc. This small problem would never have been found without direct observation of users in their natural habitats. Although, to be clear, we did it all remotely so we didn’t have to spend the money to fly all over the country.

Don’t Give Up On Your Vision Too Early!

Sometimes you’re wrong. It happens. You can think that something is going to improve your user’s experience and just be off-base. But don’t give up immediately. You had this vision for a reason, and you spent time designing and building something that you believed would solve a problem.

You owe it to yourself to dig a little bit deeper and understand why your product is failing, if only to help you learn what you did wrong and help you avoid the same mistake in the future.

User Research Tips

2010-09-30T14:31:00.000-07:00

After my talk at Web 2.0 Expo on combining qualitative research, quantitative metrics, and design vision for better products, there were some questions from the audience. Interestingly, the large majority of questions were about the qualitative research part of the talk.

And that makes sense. Qualitative research can be tough to incorporate into your development process. Until fairly recently, it's been a big, expensive, time-consuming endeavor. Often it required having outside consultants come in to run tests in a rented lab behind a one way mirror. Additionally, a lot of product folks assumed that it would slow down the development process, since it would often add a step between design and engineering.

Now, if you read my blog or listen to me speak, you know that I advocate quick and cheap testing over large, formal studies, and I like taking advantage of tools that let me run remote usability studies. I also feel that testing and research speeds up your development process, since it tends to catch problems early, when they're easier to fix.

That said, user research is easy to get wrong. It takes some practice to be good at things like moderating sessions and analyzing data. For those of you who are interested in learning more about these things, I've compiled a list of resources to get you started.

My Blog Posts

These older posts should help you fix some of the common problems people have with user research:

Books

There are a million books about user research. These are two very good ones. Let me know in the comments if you've read any other particularly helpful ones.

Online Tools

These tools do NOT eliminate the need to actually interact with your users in person, but they can be extremely valuable additions to your user research process.

Skype, GoToMeeting, WebEx, etc. - Allow you to screen share so that you can observe how your users are interacting with your product.
usertesting.com - Very fast & cheap way to test your new user experience.
NavFlow - Lets you test your site navigation using mockups, which allows you to get feedback before you build the product.
Five Second Test - Great for testing things like landing pages or whether your calls to action are obvious enough.
Ethnio - Helps recruit session participants who are currently using your product.
Revelation - Helps you run longer term studies with current users.

Please Stop Annoying Your Users

2010-09-24T14:25:00.000-07:00

Once upon a time, I worked with a company that was addicted to interstitials. Interstitials, for those of you who don’t know the term, are web pages or advertisements that show up before an expected content page. For example, the user clicks a link or button and expects to be taken to a news article or to take some action, and instead she is shown a web page selling her something.

Like many damaging addictions, this one started out innocently enough. You see, the company had a freemium product, so they were constantly looking for ways to share the benefits of upgrading to the premium version in a way that flowed naturally within the product.

They had good luck with one interstitial that informed users of a useful new feature that required the user to upgrade. They had more good luck with another that asked the user to consider inviting some friends before continuing on with the product.

Then things got ugly.

Customers could no longer use the product for more than a few minutes without getting asked for money or to invite a friend or to view a video to earn points. Brand new users who didn’t even understand the value proposition of the free version were getting hassled to sign up for a monthly subscription.

Every time I tried to explain that this was driving users away, management explained, “But people buy things from these interstitials! They make us money! Besides, if people don’t want to see them, they can dismiss them.”

How This Affects Metrics

Of course, you know how this goes. Just looking at the metrics from each individual interstitial, it was pretty clear that people did buy things or invite friends or watch videos. Each interstitial did, in fact, make us some money. The problem was that overall the interstitials lost us customers and potential customers by driving away people who became annoyed.

The fact that the users could simply skip the interstitials didn’t seem to matter much. Sure people could click the cleverly hidden “skip” button – provided they could find it – but they had already been annoyed. Maybe just a little. Maybe only momentarily. But it was there. The product had annoyed them, and now they had a slightly more negative view of the company.

Here’s the important thing that the company had to learn: a mildly annoyed user does not necessarily leave immediately. She doesn’t typically call customer service to complain. She doesn’t write a nasty email. She just gets a little bit unhappy with the service. And the next time you do something to annoy her, she gets a little more unhappy with the service. And if you annoy her enough, THEN she leaves.

The real problem is that this problem is often tricky to identify with metrics. It’s a combination of a lot of little things, not one big thing, that makes the user move on, so it doesn’t show up as a giant drop off in a particular place. It’s just a slow, gradual attrition of formerly happy customers as they get more and more pissed off and decide to go elsewhere.

If you fix each annoyance and A/B test it individually, you might not see a very impressive lift, because, of course, you still have dozens of other things that are annoying the user. But over time, when you’ve identified and fixed most of the annoyances, what you will see is higher retention and better word of mouth as your product stops vaguely irritating your users.

Some Key Offenders

I can’t tell you exactly what you’re doing that is slightly annoying your customers, but here are a few things that I’ve seen irritate people pretty consistently over the years:

Slowness
Too many interstitials
Not remembering information - for example, not maintaining items in a shopping cart or deleting the information that a user typed into a form if there is an error
Confusing or constantly changing navigation
Inconsistent look and feel, which can make it harder for users to quickly identify similar items on different screens
Hard to find or inappropriately placed call to action buttons
Bad or unresponsive customer service

It’s frankly not easy to fix all of these things, and it can be a leap of faith for companies who want every single change to show a measurable improvement in key metrics. But by making your product less annoying overall, you will end up with happier customers who stick around.

Like the post? Follow me on Twitter!

Also, come hear me speak on Wednesday, Sept. 29^th, at Web 2.0 Expo New York. I’ll be talking about how to effectively combine qualitative research, quantitative analytics, and design vision in order to improve your products.

Everything In Its Place

2010-09-16T14:11:00.000-07:00

I talk to a lot of designers. We’re all different kinds of designers: visual, interaction, user experience, information, blah blah blah, but many of us take the same things for granted. Because of this, designers will probably be bored to tears by this post, while non-designers may learn something that can make it much easier to build products that people can use.

But first, a story. A designer friend of mine had a baby. She asked her husband to put up a note asking people to please not ring the doorbell, since the baby was sleeping. Later, after somebody rang the doorbell and the baby woke up and she was contemplating divorce, she wondered why her husband hadn’t put up the damn note like she had asked.

The thing is, he HAD put up the note. He had put it right on the door at eye level so anybody could see it. What he hadn’t done was associated the call to action with the actual action.

What the hell does that mean?

A big part of any user experience design is figuring out where to put stuff. This may sound obvious, but it’s best to put stuff where people are most likely to use it. That means associating calls to action with the thing that is being acted upon.

Here’s an example you may have considered. Where do you put a buy button on a page? Well, when a user is trying to decide whether or not to buy something, which pieces of information is the user most likely to need? He definitely needs to know how much he’s paying for the item. He might need pictures of the item. He almost certainly needs to know the name of the item and perhaps a short description.

Considering those needs, the Buy button should probably go near those things on the page. It should even go in a defined visual area with just those things. Here’s the hard part: it needs to go near those things EVEN IF IT LOOKS BETTER SOMEPLACE ELSE.

What's with all the screaming?

I’m all for having a nice visual design. I believe that a page should be balanced and pretty and have a reasonable amount of white space and all that. But if one element of your gorgeous visual design has separated your Buy button from the information that your user needs in order to decide to buy, then your gorgeous visual design is costing you more money than you think.

This isn’t just true for Buy buttons; it’s true any time the user has to make a decision. The call to action to make the decision must be visually associated with any information that the user needs to make that decision. Additionally, any information that is NOT related to the decision should be visually separate.

This also applies to things that aren't calls to action, of course. Related information should all be grouped together while unrelated information should be somewhere else. It's just that simple. Oh, and bonus points if you keep all similar items in the same place on every screen of your product so people always know where to look.

So, where should my friend’s husband have put the note? He should have put it within inches of the doorbell itself. Why? Because the decision the user was making was whether or not to ring the doorbell. The husband needed to put the information about the sleeping baby right at the point where the user was making the decision, not in a completely different part of the interface (the door) where the user might or might not even notice it.

The next time you're deciding where something goes, remember, this strategy is not only important for creating a usable product, it just might save your marriage!

Like the post? I'm a veritable font of useful information, I swear. Follow me on Twitter!

Your User's Computer Sucks

2010-09-09T10:55:00.000-07:00

I often tell people that “you are not your user.” I’m an interaction designer. It’s part of my job description. What I should probably also be reminding people is that your computer is not your user’s computer. Nor is your internet connection, monitor, environment, or a lot of other things.

Why is this important? These things mean that, aside from making your product usable in a lab setting or in your office, it’s also got to work well in the standard environment of your user.

So, ok, if you build tools for lean startups, you can probably ignore most of this post. Your users all have dual core machines (probably more than one), fiber internet connections, 24 inch flat screen monitors, and a cubicle or office in which to use those things. They probably also have a smart phone that never leaves their hand and a comparable set up at home so that they are never more than a few inches from enough computer power to get them burned as witches in some parts of the world.

But the rest of you probably build products for teens or busy families or doctors or construction workers or business travelers or, you know, the vast majority of humanity that doesn’t use a computer for a living. And you need to not only understand your users; you need to understand your user’s computing environment.

I do a lot of in-home/office studies as well as remote usability testing. This means that, not only do I get to see real users with my products, I get to see them use them on their computers, and I’ve seen this over and over.

Here are a few examples of how your user’s environment really matters:

Your Computer is Faster than Your User’s

Interactions that take a split second on my machine sitting right next to the server may take two or three seconds on an old computer half way around the world. This doesn’t seem like much, but it has a pretty big impact.

When an interaction takes a split second, I don’t need to plan for any intermediate feedback (a spinner, a progress bar, a disabled button, etc.). When an interaction takes three seconds, I really, really do need one of those things, or else I’m going to get repeated button presses and confused users who don’t know whether anything is happening. Of course, interactions that are annoyingly slow on my computer are going to be completely intolerable to a lot of my users.

You Have More Computers than Your User Does

I have a laptop and a smartphone at home that are all my own, and I use them constantly. The other day, I asked a user why she chose to use a product late at night. She explained that she had school aged kids who needed the computer in the afternoons, and during the day she was typically out of the house. Her only computer time was a few minutes after the family was all in bed.

Many products are used by multiple people during the day on the same computer, sometimes at the same time. Having limited time on a shared computer creates all sorts of design implications for things like privacy and the need to be able to interrupt and resume tasks.

Your Monitor is Bigger than Your User’s Monitor

Of course, then there’s monitor size. Many people have declared the death of “the fold” because people don’t mind scrolling a bit for interesting information, but I still see products with really important calls to action that don’t show up on screens smaller than a bay window.

Guess what? I’ll scroll to read the second half of a blog post, but I’m not going on a damn treasure hunt for your Buy button. If I don’t find it quickly, there are a whole lot of other sites that will sell me what I want. If your target audience accesses your site on laptops or smartphones or Etch A Sketches, figure it out and design accordingly.

Look, “getting out of the building” isn’t just another way of saying “chatting with customers.” It means understanding customers and how they use your product. A big part of how people use your product is dictated by the environment in which they use it. So make sure that you don’t only understand who is using your product and how they are using it. Learn WHERE they’re using it and on what sort of equipment. It can make a huge difference.

Like the post? Follow me on Twitter!

The Best Visual Design in the World

2010-09-02T15:06:00.000-07:00

As I’ve mentioned before on this blog, I don’t do visual design. It’s not that I don’t think it’s important. I’m just not very good at it.

Even though I can’t produce gorgeous visual designs, like every other person on the planet, I know what sorts of visual design I prefer. I don’t have one particular style that I’m in love with, but I have pretty strong reactions, both positive and negative, to different “looks.”

Recently, I worked with a company that had a visual design I didn’t like. Now, since I’m not a visual designer, I’m not going to speculate on whether it was badly designed or just not to my taste, but I will tell you that when I showed it to many people in Silicon Valley, they didn’t like it either.

In fact, enough people reacted negatively that I stopped showing it to people in the Valley. I even found myself apologizing for it, despite the fact that I didn’t design it, and I don’t love it.

And then I did some user testing on the site. And do you know what? The users love it. They LOVE it. It is absolutely fantastic for this particular demographic, which, by the way, has nothing to do with the Silicon Valley CEOs and designers who were horrified by it.

I was showing some wireframes, with the usual disclaimers of “this isn’t how it will look; these are just black and white mockups of the final site; we’re not losing the other color scheme; blah blah blah.” Despite repeated statements to this effect, users would periodically interrupt the test to volunteer how much they love the visual design of the current site and how they really don’t want it to change.

Why is this important? It’s a great example of the fact that your visual design should reflect the aesthetic of your target market and not necessarily you. Say it with me, “You are not your user.”

Designing a beautiful, elegant, slick site that will appeal to designers, Silicon Valley executives, and Apple users is fantastic…if you’re selling to people like designers, Silicon Valley executives, or Apple users. That’s not the market for this company, so they’re smart not to build a product that appeals aesthetically to that market.

Is there such a thing as bad visual design? Sure. I’ve certainly seen visual designs that interfered with usability. Buttons can be too small; calls to action can be de-emphasized; screens can be too cluttered; navigation can be hard to find. But just because something isn’t visually appealing to you, doesn’t make it a bad visual design. The only people who have to like it are your users.

In your next design meeting, remember this: the best visual design in the world is the one your users love.

Love Thy User: 5 Tips for Dealing with Tough Customers

2010-08-17T15:02:00.000-07:00

Sometimes we build products for ourselves, but most of the time our target market is somebody completely different. It can cause all sorts of problems when we’re asking questions and observing people completely different from ourselves. Sometimes, and this can be tough to admit, we don’t really like our users very much.

Maybe you’re not like this. Maybe you’ve never had a difficult set of users who constantly yell and scream about their needs and how they’re not being met regardless of what you do for them. Maybe you’ve never spent time building a brand new feature designed to make your users happy only to have them shrug and say, “oh, that’s not what we wanted at all.” Maybe you’ve never had a passionate community of early adopters all grumpy because their favorite suggestions aren’t being followed to the letter. But trust me, the rest of us have.

The problem is, because most of our users are so different from us, their behavior can be extremely hard to understand or predict. On many occasions, this has led people to ignore their customers or neglect to include them in the development process.

I understand this impulse. I really do. It can be tough to include somebody that you see as irrational or hard to deal with in your decision making process.

But here’s a news flash. That irrational, difficult, whiny, impossible to understand person who is always complaining? Suck it up, cupcake, and include them in the conversation. They’re paying your salary, and if you ignore them for long enough, they’re likely to stop doing it.

Here are a few ways to make it easier to get feedback from difficult groups of customers.

Keep it one on one

When you’ve got a group of people, all of whom seem hell bent on complaining about how your product is ruining their lives, don’t put them in a room together.

A lot of companies like to establish customer advisory panels or customer forums and the like, where they can get feedback from a lot of people at once. These are fine when the conversation can be kept civil, but they can quickly turn into an angry mob as the group forms a giant echo chamber of hate.

Keeping the conversation one on one allows you to spend more time with each person and understand what’s really upsetting him or her.

Keep the conversation in person (or on the phone)

TALK to your customers. Sure, forums, suggestion boxes, and other written forms of communication are useful tools, but when you’ve got angry or unhappy customers, written communication is just too easy to get wrong. Many people who are more than happy to dash off an angry screed in a forum will calm down immediately once they get a real human being to explain their problems to.

It also makes it quick to ask follow up questions and really get to the root of the matter without inadvertently making the customer angrier.

Identify the root cause

When people are angry, they’re not always great about expressing what is upsetting them. What sounds like, “You are ruining my life,” might actually mean, “There is a very specific bug that you’ve been ignoring that is affecting the way I run my business.”

Remember, there’s almost certainly a nugget of truth at the center of the craziest assertion. The best way to find out that truth is to get to the root cause of the problem as non-confrontationally as possible.

Try asking the users to walk you through exactly what is causing their problems. Make sure to ask when and how the problems first started. If they give you a laundry list of complaints, try to get them to prioritize. Often, there is one core issue that is causing the anger, and dozens of other minor complaints that just add fuel to the fire.

Also, once you know the real problem, don’t assume you understand the solution. Remember, you caused the problem in the first place. Ask users to help you come up with solutions. I’m not saying you should just do whatever they ask, but make sure you check possible fixes with them before implementation to make sure that you’re actually fixing the root cause of the problem.

Maintain trust

One important lesson you must learn is that it’s much easier to maintain trust than to reestablish it after it’s been lost.

One company with which I worked had a passionate group of power users who all seemed to be conspiracy theorists. Whenever something went wrong or a bug brought down the site, they would immediately begin concocting wild stories about nefarious future plans that the company had to institute various Orwellian policies.

The company could never understand why the users didn’t believe it when the company explained that there were no nefarious plans in the works. Many at the company wrote the users off as crazy.

The problem was that the company had, in the past, issued blanket statements denying policy changes and then gone ahead with those exact policy changes. After that had happened once, there was absolutely no reason for the users to believe anything the company had to say, even when the company was telling the truth.

Once lost, trust was almost impossible to regain, and many of those passionate power users moved on to other products.

Actually fix their damn problems

Ok, understanding that your users aren’t just raving lunatics is a great first step. Connecting with them and identifying their problems is even better. Now you actually have to FIX THINGS. The best communication in the world isn’t going to help you if you continue to ignore the things that your customers are complaining about.

I know, I know. You’re working on the next release or a hot new feature or an integration with a partner or any one of a number of other things that your customer may or may not care about one bit. But now that you you’ve asked them, you know a lot about what your customer DOES care about, and you should work on those things too. Right now.

In conclusion

Look, I’m not saying that the customer is always right. Sometimes people really are crazy, or maybe they’re just not right for your particular product. That’s fine. You’re going to lose those customers regardless of what you do.

But sometimes the fact that you don’t understand your customers isn’t their fault. It’s yours. Learn to listen to your customers rather than writing them off as angry cranks. Within a very short while, you’ll find that they don’t sound so crazy, after all.

Like the post? Follow me on Twitter!

When To Get Help With User Research

2010-07-22T13:30:00.000-07:00

I don't spend a lot of time on this blog telling you why you should hire me to talk to your customers. In fact, the vast majority of the posts are meant to make it more possible for you to talk to your customers without hiring somebody like me. It's not that I don't like working. It's just that I think that anybody who is responsible for making decisions about products should know how to learn from users on their own. It results in better products for all of us.

Product owners need to be involved in customer research for a lot reasons. Reasons like:

You're more likely to believe the results if you participated in the research.
You're more likely to understand the relative importance of customer problems if you observed the problems happening.
You will come up with more comprehensive solutions to problems when you understand the context in which they're happening.
It's far too easy to ignore a report written up by a usability consultant, it's incredibly easy to forget to watch the testing videos.
If you do it yourself, all of the lessons you've learned will stay within the company, long after a consultant has gone on to other projects.

That said, I'm about to tell you why you may need to hire somebody like me. For a little while at least.

When I talk about customer research or customer development or learning from customers, I really mean quite a lot of different techniques. Sure, there are general best practices around talking to customers, tips for improving your research skills, and important things you should avoid, but there are also things like picking the right testing method or tool that you almost certainly have no experience with. You need to know what is the most important thing for you to do right now.

Do you know when it would be helpful to do a card sort? A journal study? A contextual inquiry? Do you know when it's fine to do a remote usability study vs when you should really run one in person? How about when your product will benefit from using an online tool like usertesting.com or fivesecondtest, and when something like that isn't useful? Do you know what sort of testing to do in order to find out why specific metrics are lower than you'd like? Do you know when you should start your visual design and when you need to concentrate on usability? Do you know how many people to talk to in order to answer a specific question? Do you know at what points in the development cycle talking to users is critical and when it's a waste of time? Do you know how to take several hours of free form user conversations and turn it into a small number of features or bug fixes that can be communicated to your engineering team?

If you answered, "of course I know that" to all of those questions, then move along. You almost certainly have no use for somebody like me to come in and help you out. If you answered, "I'm going to learn the answer to all of those questions," then I wish you good luck on your journey of discovery. I'll warn you though. There are more questions just like those.

If, on the other hand, you said, "I don't know the answer to a lot of those questions, but I wish somebody could help me understand the small subset of them that matter to me, as a product owner, so that I could get on with the business of building a great product," then you might want to give me (or somebody like me) a call.

Because it's true that there is a huge amount to know about talking to your users. But it's also true that, at any given stage in your product development, you probably only need to be concerned with only a little bit of it. And, it's also true that figuring out which bit of it you need to know can be really hard to do without help. That's where people like me come in handy. We can help you figure out what to do next, and then we can help you learn how to do what you need to do next.

But be careful. If you're a lean startup, you probably don't want to pay us to actually do what you need to do next. For all the reasons I mentioned above, that's still your job.

Interested in this sort of service? Learn more about Users Know here.

Want to read more posts on how to do this stuff yourself? Follow me on Twitter!

Shut up, and Show Me Something

2010-07-07T11:45:00.000-07:00

I admit it. Quite often on this blog I give you long lists of fairly hard things to do. I ask you to change your whole approach to design or product management or customer interviewing or analyzing data. But not today. Today, I share with you one simple thing that is easy to remember and will transform your entire approach to customer research.

Ok, maybe it's not quite that cool, but it really will help you communicate with your customers better. Are you ready? Here it is:

Never have a conversation with a user (or potential user) where you don't show them something.

That seems simple enough, right? But why on earth should you do it, and what could you possibly show them?

Reasons for Communicating with Customers

Let's back up for just a moment. The main reasons that people generally talk with a user are:

To get information from them - what they like, what they don't like, what's confusing, why they're not buying things, etc.
To give them information - here are the features of the product, here's how to fix your problem, we swear it's a feature and not a bug, etc.
To sell them something - whatever it is that sales people do...besides drinking heavily

All of these things are much easier to do when you're looking at visual aids.

Getting Information from Users

Let's perform a thought experience. Without thinking about it, name three things you hate about doing your taxes. Were you able to do it? Of course you were. If you can't think of three things you hate about doing your taxes, either you're not paying attention, or your hiding all of your money in an offshore account in the Caymans. But are they really the three worst things?

Probably not. They're just the three things that you happen to think about when put on the spot. Next tax season, you'll be doing your taxes and think to yourself, "Oh right, THAT thing! I hate that thing! I wish I'd thought of that when I was asked for three things I hate." And you most likely would have thought of it if you'd been going through your tax preparation software when I asked.

Sure, you can just ask users what they like and dislike about your product, but you will get much better information if you're both looking at the product together. Even better, ask them to perform some tasks or just use the product while you watch. This not only jogs the user's memory about all the little annoying things that they're sort of used to, but you can also observe all the things that they don't even notice or are too embarrassed to mention they're having trouble with.

Giving Information to Users

Think about getting slightly complicated directions from somebody verbally. Now imagine that they're showing you the route on a map while they talk to you. Which do you remember best? What about if you have a gps screen showing you turn by turn directions in the moment? Even better? You bet.

Looking at things together helps your customer understand the information you're delivering much better than just talking.

Selling Users Something

I'm not a sales person, so I'm not going to spend any time on this one, but I'm a hell of a lot more likely to buy something that I have actually seen in action (and liked) than something that has been extensively described to me. If anybody has any examples, pro or con, of how showing people a product impacts sales, share them in the comments.

But What Should I Show Them?

Good question. You should show them whatever it is you want to get feedback on, give information about, or sell. The last two are pretty self-explanatory, but the first one can be tricky. Let's look at it more closely.
If you have a product and want to know what people think about it and how they're using it, look at the product with the user. Easy enough.

If you have an idea for a product, and you want to get feedback on it, show sketches or interactive wireframes. The closer to a real product it looks, the better the feedback will be, but there are probably times when you just want to show a bunch of quick sketches or a few different visual designs in order to get quick impressions.

But what if all you have is a vague idea of a problem you want to solve? You're not even at the sketch or wireframe stage yet, but you know that you want to solve some problems for a particular group of people. What do you show them then? Easy, does the user have some other method or product that they're using to solve the problem currently? Watch them use that! Are they not currently solving the problem at all? Have them show you how far along they got in solving the problem before they eventually gave up.

Here's an example. A client of mine once wanted to create a solution for people who sell things online. Now, there are two potential markets for this:

People who are currently selling things online through other services
People who want to sell things online but don't, for some reason

For the first market, we needed to watch people selling things online the way they currently do, in order to figure out what their problems and pain points were. For the second market, we needed to watch the way people sold things offline and then watch them try to use competitors to understand what was keeping them offline.

Could we have just asked people those questions? Sure, but we wouldn't have gotten information that was nearly as detailed or complete. In every observation, there were times when the user would get to some point in the process and say something like, "Oh, right. That thing there. That drives me nuts." There was also a point where we noticed that the customer was jumping through crazy hoops to complete a task, but they were so used to it working badly that they didn't even imagine that it could possibly work better. Later, when we showed them our possible solutions to problems they didn't even know existed, they were thrilled.

But Laura, Why Don't You Ever Include Visual Aids In Your Posts?

I was hoping you wouldn't notice that, frankly.

Know of a case when you wouldn't want to show somebody something? Argue with me about it in the comments!

Like the post? Follow me on Twitter!

Nobody is Thinking About Your Product

2010-06-25T11:56:00.000-07:00

When you're working at a startup it can be all-consuming. You can forget everything else in your life pretty easily when you're neck deep in valuations and minimum viable products and customer acquisition and a million other things that need your attention. You tend think about your product every waking minute.

That's why it can be such a shock to realize that nobody else is thinking about your product. Well, ok, unless you're Apple, but there's clearly some kind of weird mind control thing going on there. In general, when you have a new product, you're incredibly lucky if you're getting more than a few minutes of attention from anybody but your most passionate early adopters.

Why is it important to realize this? It's important, because it has a really big impact on how you design your product and connect with your users.

Make Everything More Discoverable

You know exactly where in the user interface to go to do every task that can be completed with your product (I hope!). Other people, especially new users, don't even know that most of your features exist. This means that it's just as important to design for discoverability as it is to design for usability. But how are they different?

Let's do a quick thought exercise. Imagine somebody hands you a featureless metal box. You might look at it for a minute or two. If it's particularly attractive, you might admire it, but you're probably not going to spend a lot of time with it. Now imagine that the box has $10,000 dollars inside of it. You will probably spend a lot more time figuring out to get it open, yes?

Your product is like that box that is hiding money. If people don't discover very quickly that it provides something valuable to them, they're not going to spend much time figuring out how to use it. You need to help people understand immediately that your product has features they really, really want. That's discoverability.
You also need to make it pretty easy to actual learn how to use those features, once they've decided to dig into the product a bit. That's usability. For bonus points, you can make the whole process interesting and engaging so that people actually enjoy discovering features and using your product. That's fun.

Key Take Away: Users are not going to spend any time learning to use your product if they don't immediately understand what's in it for them. Make it easy for them to figure out what features exist and why they're useful.

Remind People You Exist

This is going to sound obvious, but people have lots of things to do every day. What with their own jobs and families and checking Facebook and standing in line at the Apple store for whatever is coming out next, their schedules are pretty full. If you're going to fit into that schedule, you can't just sit around and wait for them to come back to you.

Remember, they're not thinking about you. Ever. That's why you have to contact them regularly via email or Facebook or Twitter or post cards or sky writing or whatever you think will get the attention of the people you're targeting. Think that you can put off writing your welcome emails or your mass notification system? The longer you put it off, the more early users you're going to lose because they didn't think to come back to your product the next day.

Key Take Away: Assume every single one of your users forgot about the existence of your product five seconds after closing it. If you want them to come back, you need to remind them.

Design for Distraction

You know what you never see in a traditional user test? The seventeen million other things that your users are doing while using your product.

See, even when people are thinking about your product, they're not thinking about only your product. They're thinking about the phone that's ringing and when they have to pick up the kids from soccer and what they're going to make for dinner and the fact that their boss wants a TPS report finished before 5.

Things like shopping carts that time out or registration forms that need to be filled out from scratch if you make a small error or login redirects that don't send you back where you wanted to go are all poisonous to the distracted user. They dramatically increase the number of people who are going to simply give up on your product halfway through.

That's why it's important to make sure that you're actually watching people use your product in their natural environments whenever possible so that you can understand the kinds of interruptions that you need to plan for. It's also critical to make sure that, if somebody were to get called away from your product in the middle of a task (which they will), that they can easily come back and finish that task without having to start over from scratch.

Key Take Away: People have other things going on in their lives, so make sure that your product allows for interruption, inattention, and distraction.

Make It Addictive

In the way that heroin addicts always think about heroin and WoW addicts always think about Wow and Apple fans always think about new Apple products, if you can design something addictive people will think about your product more. The problem is, making things addictive is harder than just throwing in a few simple game mechanics or copying whatever WoW or Apple does.

There are a lot of good blog posts on how to make your product stickier, but some of the common themes include:

creating social bonds with other users (Joe wants to be your friend!)
having time sensitive tasks that require users to return at certain intervals (Log in in the next 15 minutes to get this great deal!)
providing incentives and achievements for regular use (You unlocked the Foozle Badge!)
providing competition with other users (You are now the mayor of the Scranton, PA Taco Bell!)
creating a sense of disaster if they don't return (Your fake crops will die!)
offering quality content that the user can't get anywhere else and that updates daily (Learn about the five things in your pantry that could KILL YOU!)

The important thing to do here is to pick styles of addiction that fit with your product. Your tax preparation software probably doesn't need a leaderboard, but a good, weekly blog on ways to save tax-free money for retirement might be a draw.

Key Take Away: Provide reasons for your users to want to come back daily by using things like game mechanics, social pressure, or new content, but make sure that the features fit comfortably with your product.

Cultivate the People Who DO Think about Your Product

If you're lucky and you work really hard to get people's attention, everything I've said isn't going to apply to a tiny group of early adopters. These are the people who will think about your product all the time and want to be heavily involved in its growth and improvement. Use those people! Talk to them. Learn from them. Get them to evangelize your product to other people just like them. Give them jobs, or let them monitor your forums and answer questions for other users. They will love that they're contributing to something they care about, and your product will improve as a result.

Also, if you ignore them, they'll most likely stop thinking about your product, and go think about somebody else's product.

Key Take Away: Understanding the few people who are deeply attached to your product can help you understand and improve the features that may make it appealing to other, less dedicated users.

But Most Importantly...

You need to internalize the fact that, even once they've visited your site or downloaded your app or become a registered user of your product, the vast majority of people simply aren't thinking about you or your product. At all. This means that a big part of your job is not so much about countering loathing or dislike as it is about countering total indifference.

So, get out there and make them remember you exist.

Like the post? There are more like it. Follow me on Twitter!

Some Surprising Problems Caused By Using Your Own Product

2010-06-07T20:36:00.000-07:00

We've all heard the stories of huge companies that started because the founder wanted to do something specific but didn't have the proper tools. He or she built the proper tools and quickly realized that he or she wasn't the only one who wanted those tools. Paul Graham wrote a very interesting blog post about what he calls organic startup ideas a couple of months back and why building something you need is often the best way to start a company.

There are a lot of benefits to being your own product's first and biggest user:

You will always have at least one person who likes and uses your product
Even if nobody buys it, you'll still be happy you built it, since you get to use it
You always have a user available to consult on which feature to build next
It should be easy to initially identify a target persona (hint: you may need a mirror)
You probably have a network of similar people who may also want the product
You can find a lot of bugs and corner cases by using your product on a regular basis

But there may be a few problems that you weren't expecting.

Understanding the New User Experience

Frankly, there is nobody worse at figuring out how confusing the new user experience can be than an expert. And, if anybody is an expert at the product you've built for yourself, it's YOU.

Have you ever tried to explain something "simple" to somebody and realized that it isn't, in fact, simple at all? It is extremely complicted with lots of steps. The only reason you think it's simple is because you've done it a million times. Unfortunately, it can be very difficult to assess how hard it is to learn something once we know too much about it.

For example, if you already know that a feature exists somewhere in the product, it's much easier to figure out where it is than if you're brand new to the product and have no idea that the feature even exists. It's also easier to understand the logic of how different features work together if you're the one who put them together in the first place.

The fix: Luckily, there's an easy fix for this. Watch new users with your product. Select people in your target market and just observe their struggles. Use products like usertesting.com to observe the first 15 minutes of their usage. Get in touch with people who have only used your product a few times and ask if you can watch them (not in a creepy way), in order to understand what slightly more experienced people are doing with your product. Whatever you do, when you see somebody making a mistake, don't correct them! It's important to see how people who aren't you are using the product.

Asking for Feedback

Remember all that stuff I recommended you do for new users? You should also be doing it for much more experienced users too. The problem is, you probably won't.

In my experience, people who are big users of their own product are less likely to think that they need to observe customers because they have one right there in the building! It's hard to admit that you don't know everything about a product that you're both building and using extensively.

The issue here is that you're not the only type of user. You may not even be the main type of user. When Twitter was first developed as an internal communications tool, do you think that the creators thought that Ashton Kutcher would one day be using it to tell his fans what he had for lunch? People are out there doing really surprising things with your product. You need to learn what they are, and you can only do that by talking to them and observing them.

The fix: This one's easy. Just make sure to keep getting feedback from other users. Preferably, search out people who are different from you or who are using your product in very different ways.

Accepting Feedback

Ok, so you're talking to users who aren't like you. You have to make sure you're actually listening to them.
When you're a regular user of a product, it can be tough to be completely neutral about feedback.

Confirmation bias means that you're more likely to listen to the feedback that agrees with your experience and to discount the feedback that doesn't.

The fix: You may need to bring in a neutral third party to help you moderate customer discussions and make sure that you're not letting yourself be biased. But just knowing that confirmation bias exists should help you to combat it.

Killing Your Favorites is Tough

The time comes in every product when you have to kill a feature. Sometimes you release something and only 5% of your users adopt it, which makes it just not worth supporting. Or maybe it's a fine idea, but it doesn't fit with the rest of the product offering and just clutters up the interface. Maybe it's just too expensive to maintain. Whatever the reason, it has to go.

But what if you are one of that 5% of users? That makes it harder to kill the feature doesn't it? It's tricky to respond completely rationally to metrics if it results in losing access to something that you like. And pivoting away from your big, pet idea? That can be nearly impossible, even when nobody's buying your product.

The fix: You thought I was going to say, "kill it anyway," but I'm not. If you feel that the feature is truly useful to you (or to any small percentage of your user base), why not do some investigation into why the adoption rate is low? Maybe people don't know it exists, or it's too hard to use. Those are both fixable problems. If the problem is that the feature doesn't really fit in with your product offering or is cluttering up the interface, consider spinning it off as its own product or moving it into a less obvious section of the UI meant for power users. If it's expensive to maintain, maybe you should try offering it as a premium option to offset the costs. If none of these work though, you may have to kill it anyway.

So, Should You Not Build a Product You Want?

Of course you should build products you want. Waiting around for somebody else to build them is such a waste of time.

Besides, building something for yourself can give you a huge jump on customer development at the very beginning stages of your product development. The problem comes later, when you want your product to be used, enjoyed, and (hopefully) paid for by other people. That's when you have to step back and say, "This product isn't just for me any longer. I need to start treating it like something that is used by other people who also have opinions and preferences, many of which will be just as important as mine." If you don't, you may end up as your product's only user.

Like the post? Follow me on Twitter!

When You Should Hurt Your Users

2010-05-26T15:39:00.000-07:00

I'm going to let you in on a little secret. I don't always do what's best for users. I know, I know. This is a horrible thing for a UX person to admit, but it's true.

This shouldn't come as a surprise to anybody, but it's shocked enough of my clients that it bears stating clearly. As an interaction designer, my job isn't (or at least, it shouldn't be) always to optimize everything exactly for the end user. There are several reasons for this.

The Best Thing For the User May Be Bad for Business

One of my favorite Dilbert cartoons explained that what users really want is "better stuff for free." As much as I dearly love to help the end user, my clients would quickly go broke if I gave users everything they wanted. A big part of the UX job should be balancing what is best for the user with what is best for the business. Sadly, those aren't always the same thing.

What does this mean in reality? It means that every time you see a banner on a free site, some UX designer accepted the necessary evil of showing an advertisement for acai berry in return for offering content for free. It means that sometimes a pricing page will be designed to showcase a more expensive option than the user might otherwise have chosen. It also means we have to work extra hard to identify features that are good for both users and the business (for example, features that make current paying users happy and thereby increase retention), so that we don't always feel like we have to sacrifice either revenue or customer happiness.

Caution! This can go too far in the other direction. I would argue that a lot of Facebook's recent privacy debacles have been driven by optimizing too much for the business and not enough for the end user experience. The result of this, of course, can be losing end users, which can end up being bad for business in the long run.

Getting It Out There May Be Better Than Getting It Perfect

Nothing is perfect the first time it's released. If it is, you probably spent too much time on it. And sometimes the very best solution for the user is impractical for a lot of other reasons.

What does this mean in reality? Well, it means that if the absolute optimal solution for end users will take 6 years and a team of 20 geniuses to implement, I will try my hardest to find the next best alternative. I'm not saying I'm going to settle for a bad user experience. I'm just saying that there are a lot of different ways of making the user happy, and if it can be done without killing the engineering department, that's a bonus.

Caution! This does not mean that you get to ship a bunch of crappy features and then never touch them again. If I give you a pass to deliver a suboptimal first version to the user in the interest of time and engineering effort, I expect that it will be iterated upon and improved in the very near future. It may never be perfect, but it had better improve!

I Have More Than One End User

Some users are more useful than others. There I said it. As much as we'd love to believe that a change will make everybody happy, we probably have lots of different types of users who all want something slightly different. In the same way that optimizing for the end user can sometimes be bad for engineering or revenue streams, optimizing for one type of end user can be bad for other end users.

What does this mean in reality? Consider freemium products. Freemium products have several distinct sets of users: those who pay and those who don't; new users and returning users; power users and casual users, etc. Sometimes, a feature that is optimized for a brand new, non-paying user can annoy or bore a retained, paying, power user. Since annoying people who pay me is not something I'm generally in favor of, when I'm adding a new user feature, I need to make sure that I'm considering the effect I'll have on everyone else. I also need to make sure that power user features are introduced correctly to (or hidden from) new users so that nobody gets confused or frightened away. These tradeoffs may make some things worse for some types of users.

Caution! Try not to always screw over one type of user, unless you're absolutely sure that that type of user will never convert into a more valuable type of user. If you've got a freemium model, most of your paying customers were probably once non-paying customers. And, of course, all of your power users were once noobs. You need to do at least enough for them to get them stick around and decide to pay you.

I Don't Always Know Exactly What's Best

Once, I had an engineer come up to me out of the blue and ask exactly what color and size would be optimal for a button. He was more than a little shocked when my answer was, "I have no idea." As much as I'd like to pretend that I have a mind meld with all of my users, I just don't. Sure, I tend to have good instincts based on years of designing and testing products, but the best way to find out what users really like is still to run a test of some sort.

What does this mean in reality? It means that even if you have the best designer in the world working for you, you should still be following up with users in some way to make sure that everything is on track. Whether it's usability testing, contextual inquiry, collecting metrics, or (preferably) all of the above, you need to be constantly making sure that your designer is making the right decisions. Because sometimes we're not.

Caution! Relying solely on testing for design decisions can have drawbacks. People misinterpret both qualitative and quantitative data all the time, and metrics can't replace vision. Still, combining qualitative and quantitative data with design is an outstanding way to improve your product.

What's the Right Balance?

The ideal solution is always to try to find those magical features and projects that benefit as many different people as possible. When you've run out of those, you're going to have to start making some tough decisions.

My approach is to try to find the optimal user solution and work backwards from there, if it's absolutely necessary. If the optimal user solution is really bad for some other part of the business or user base, I try to find close alternatives that deliver the same benefits without most of the drawbacks. Every time I change something crucial, I reevaluate to make sure that it's still delivering value to the original user for whom it was intended.

It's probably more art than science, but starting from the best possible user solution and subtracting always seems to give me a better balance than starting with something inherently bad for users and trying to make it nicer.

Do you have other examples of when it's ok (or necessary) to do what's not best for your users? Share them in the comments, please.

Like the post? Please use the widget below to share it with your networks! Also, follow me on Twitter.

When Talking to Customers Isn't Enough

2010-05-24T15:52:00.000-07:00

A few weeks ago, I was talking to the head of a small company about next steps. His company had a product with many happy, paying customers, and he wanted to know what to do next to make his current customers happier and attract some new ones.

The company had already made a good start. They had done surveys of current users asking the standard questions like, "How disappointed would you be if you could no longer use this product." They'd even surveyed current users about what new features they would like to see. The problem was, the happy customers couldn't think of anything else they wanted, and while the slightly less happy customers wanted some new features, there was no general consensus on one big, missing piece or fantastic new idea. So, the CEO wanted to know what to do next.

I believe this can be a problem with the "go out and talk to your customers" solution to product development. We can get so focused on talking to customers that we forget that it's not always the best way to figure out what to do next.

Observe Your Customers

Customers lie. They don't always mean to lie, but it often ends up that way. It's especially problematic when you ask people to explain how they currently use your product. Sometimes they forget parts of their process, or they don't even realize that they're doing certain things because it's all become so routine. Also, they tend to explain their process of using your product from start to finish, as if they weren't doing seven other things while trying to use your product. This can give you a totally skewed vision of what people are actually doing with your product.

What's the solution? Go out and watch them. Sit with them in their offices or homes and observe their real behavior. Most importantly, watch what they do immediately before and after they use your product.

I was talking with somebody who used to work for a marketplace that allows people to buy and sell products directly to each other. While observing users, her team noticed that, while people had very little trouble using the marketplace itself, the sellers spent a huge amount of time arranging shipping of the items. In fact, the shipping process took so much time that it limited the number of items they could list. By integrating with a shipping company to help customers print labels and schedule pickups, the company increased the number of items that could be listed which increased revenue.

Why didn't users ask for that? Well, the customers had a particular way of doing things. They thought of the marketplace as a place where they could buy and sell things, not as a product that helped them mail things. They had another solution that helped them mail things, and they didn't know that there was a better way to do that until they were presented with it.

Another critical thing you can learn by watching people is the environment in which your product is being used. In one study I conducted, I was watching people process payroll. When asked how they processed payroll, customers could easily explain all the steps they went through. However, when I sat down beside them and watched, I realized that it wasn't nearly that simple. Not a single person got through the process uninterrupted. Phones rang. Coworkers stopped by to ask questions. Information was missing and had to be hunted down. Bosses needed tasks performed immediately. What they had described as a quick, linear process actually happened in fits and starts and could take place over a day or two.

Were they lying when they described their experiences? Not intentionally. They weren't asked to describe everything that could possibly happen while processing payroll, and they probably couldn't have answered that question if I'd asked it, since the interruptions varied wildly depending on the day and the office. In the end, the observations helped make the product more tolerant of this working style by allowing people to save state, skip areas where they didn't have the right information, and easily track what had already been done and what was still pending.

Connect With People Who Were Almost Your Customers

Don't forget, there's another really important group of people out there: people who were almost your customers. For every one person who signs up for your service or converts to a paying customer, there are lots of people who took a look or maybe used a free trial and then decided not to pull the trigger. A great way to build your customer base is to figure out why they made that decision.

The company I mentioned at the beginning of the post had a perfect audience for this. They offered a one month free trial, which meant that they had information about people who used the product, saw exactly what they had to offer, and then chose not to become customers. Maybe they didn't convert because the product was lacking a key feature. Maybe they didn't understand how to use it. Maybe it didn't do what they expected it to from reading the description on the website. These are all totally fixable problems, but you can't fix them until you know what they are.

Let me just head off the inevitable criticism of this approach right now. I am not advocating that you listen to every single thing that your almost-customers ask for and start building those features immediately. That would be insane. What I am suggesting is that you listen to the reasons that they give for not using your product and then analyze the data for patterns. Are lots of people saying that the product didn't do what they expected? Maybe the problem is that your marketing materials are deceptive. Are they complaining that it didn't do what they wanted? Find out what they wanted to do and what they're currently using to do it. How you incorporate their feedback is up to you, but you can't respond to feedback unless you're asking for it.

Of course, non-customers can be a little harder to connect with than customers, and they do tend to be less likely than customers to allow you to come hang around their offices all day and watch them work. Starting with a survey or an email asking to interview them on the phone can get you lots of good information about what is keeping them from becoming customers. Once you've built a rapport, some of them might even let you come watch them use the product.

Take Another Look at Customer Data

Not all companies have the ability to collect extensive customer data, but if you do, you may not be taking full advantage of it. For example, companies often fail to figure out the difference between the sorts of people who do become customers and those who don't.

Is your product only being purchased by companies with fewer than five employees? If so, that may be a signal to increase your marketing efforts to small companies while decreasing your spend on advertising to larger ones. Are your customers disproportionately mothers or college students or left handed circus performers? If so, start connecting with people who fit that profile to see what they think of your product and whether it solves a particular need for them that you might not have known anything about.

Or, the difference could be based on behavior rather than demographics. For example, if you have a freemium model or a free trial period, you should be looking at the behavior leading up to a user converting to paid or abandoning the product.

One client I worked with created a huge model showing all the different behaviors of users to try to understand which behaviors were most likely to lead to higher revenue and retention. Once we knew that users who explored a particular part of the product in the first five minutes were most likely to pay us, we could start experimenting with what would happen if we emphasized that part of the product early. Once we found that people who went down a different path abandoned the product, we could study that particular flow and find out if we were unintentionally confusing people or driving them away.

So Should I Still Talk To Customers?

Of course you should! Staying in constant contact with customers is vital to understanding your market and keeping people happy. It's just not always enough. If you feel like talking to customers has left you at a dead end or you want to get a perspective from somebody who isn't already a customer, give some of these alternate methods a try. You might be surprised at what you learn.

Like the post? Follow me on Twitter, please.

You should also check out some of my other posts on user research and customer development:

How Many Features Does it Take to Destroy Your Product?

2010-05-17T16:43:00.000-07:00

I work with a lot of startup founders who are extremely excited about their products. This is unsurprising, since nobody in their right mind would launch a startup without being totally convinced that their product is going to be awesome. It’s just too much work and risk for something that you aren’t passionate about.

Passion in startups is a great thing. It can also be incredibly dangerous.

The story is the same with pretty much every group of founders I work with: they have hundreds of great feature ideas for their product. Let me be clear: HUNDREDS of great feature ideas. Don’t get me wrong, many of these are genuinely great ideas. Almost every time I hear one, I think, “Wow, that would be really nice to have in this product and would add a lot of value.”

Unfortunately, starting a product with all of those features causes a lot of problems.

More Features = More Time to Build

Imagine building a house that includes a single room vs. one that has twenty rooms on three floors. There’s a lot more work, design, and planning that needs to go into the bigger design to make sure that people don’t get lost and that the house won't collapse.

This may seem obvious, but building one feature takes a lot less time than building ten. And the more time you spend building your initial product, the longer it takes to start getting metrics from actual users. Sure, you can and should start getting feedback from people using mockups and prototypes, but nothing beats actual data from your real product in the hands of users.

More Features = More UX Complexity

Remember that 20 room house? It has a lot more windows, doors, staircases, and electrical outlets than the one room place. It can be hard to find a reasonable place for everything. The more complex your product is, the more challenging it can be to present all of your features to your users in a usable way.

It’s hard to design a product that has a lot of different features. Everything needs to have an obvious, findable place in the product structure. All the features need to fit together in such a way that a user never gets lost or confused. Starting with a couple of key features keeps the overall UX much simpler and vastly reduces both your design time and the chances that your product will look like a giant mess.

More Features = Less Clear Value Proposition

Ever come to a web site or opened a product and thought, “What on earth does it DO?” Generally, the culprit is a confusing jumble of features, menus, buttons, and calls to action that prevent you from understanding the main value proposition of the product. Sure, companies try to combat the problem with wordy feature descriptions, video tutorials, and on-rails first time user experiences, but those often make things worse.

The simple fact is that the fewer features your product has, the easier it is for a new user to immediately understand what your product has to offer them. When all you have is new users, making sure they don’t get confused and leave is a pretty high priority.

How Do I Know How Many Features I Need?

I wish there were a magic number I could give you. Since there isn’t, I’m going to give you one way of approaching the problem that's worked for me in the past.

Define the Product

The first thing you need to do is stop thinking of your product as a collection of awesome features and start answering the question, “What does it DO?” Answer it as simply as possible, preferably in one sentence. Like this:

“It lets people buy any book in the world.”
“It helps people make new friends.”
“It helps small businesses with fewer than 5 employees find and purchase group healthcare.”
“It connects travelers with people who want to rent out a room.”

This is harder than it sounds. You’re probably used to dealing with products or web sites that have been around for awhile and started to add other features. For example, you could define Facebook in any of the following ways plus dozens of others:

“It helps people get in touch with old friends.”
“It allows people to play casual games with friends online.”
“It lets people post pictures to share with friends and family.”
“It lets brands market directly to consumers.”
“It helps people organize parties and events with their friends and contacts.”

But remember, it didn't start out that way! Any one of those definitions would be perfectly reasonable as a single product, and each one would have its own minimum set of necessary features.

Define the Minimum Set of Features Necessary

Now that you’ve got your very simple product definition, you need to strip it down the very minimum set of features necessary to let people do this thing. BE RUTHLESS. You want to let people buy any book in the world? Let them find and purchase a book. You don’t need to give them recommendations or friends or wishlists yet. You don’t need to let them buy a blender at the same time. You need to give them the best damn book buying experience they can imagine, and when they first come to your site, they had better immediately understand that they can buy books.

This may be the hardest thing that you will ever have to do with your product. It takes a tremendous amount of willpower not to add all those awesome features that you know people are going to love. Just keep telling yourself, you can always add them later, once you’ve nailed the core experience of your product. Remember, if users don’t love the core experience of your product, all the extra features in the world aren’t going to save it. In fact, they'll probably make it worse.

Like the post? You should follow me on Twitter!

Visual Design is Important, But Maybe Not Just Yet

2010-05-12T13:12:00.000-07:00

I’ve been thinking a lot lately about visual design. A few weeks ago, at the Startup Lessons Learned conference, somebody asked how important visual design is in user experience, and recently somebody asked how much time I spend on visual design early in the design process. The answers to those two questions are: “incredibly important” and “not much.”

Why Is Visual Design Important in UX?

A lot of people, even a lot of designers, write off visual design as “making something pretty.” Frankly, I’ve been guilty of this myself. Meanwhile, companies like Google and Facebook have made fortunes while keeping their visual design so spare as to be almost non-existent. So you may reasonably wonder, why is visual design important to my startup or new product, and why should I bother spending time and resources on it?
But visual design doesn’t just make things pretty. It’s instrumental in a lot of ways, including:

Enhancing your information design
Reinforcing desired user actions
Setting the tone for your product

Visuals Enhance Information Design

Sure, Facebook has a very simple, blue, gray, and white look. That’s because Facebook is all about delivering an enormous amount of content and information to you quickly. A bright, distracting, or cluttered interface would make it hard to read large numbers of news items and could clash with user-posted pictures.

Same with Google. Google, at least the main search product, is about getting you to type in a search term and then giving you access to everything on the internet that matches it. That’s a lot of information to process. Sites that are about delivering large quantities of information need to keep their visual design simple. But a simple design doesn’t mean no visual design at all.

Reinforcing Desired User Actions

There is something that you want your users to do with your product. Maybe you want them to buy books or make new friends or search for information. Whatever it is, good visual design can reinforce that behavior by drawing attention to the appropriate parts of the screens.

Consider call to action buttons. Something as simple as making them look clickable and noticeable can have a huge impact on whether users actually follow through on the desired action. This is one of those gray areas where visual design and interaction design converge. A bad visual design can render a perfectly decent interaction much harder to notice and perform by hiding important screen elements. Conversely, a good visual design can improve the interaction by making important things seem more important.

Visuals Set the Tone

One company I worked with had a very distinctive visual design. It featured cartoony characters with giant heads and oversized anime eyes. The colors were bright and cheerful. When we ran usability tests with new people, their first response to the question, “Who is this site for?” was “pre-teens.” Many responded, “Oh, my 12 year old cousin would love this.” It was very clear that these users did not think that the product was for them.

Unfortunately, the product WAS for them. A decent percentage of the users and revenue came from adults, but too many people in that age range felt that the product was for children based on their first impressions of the site, and would leave immediately rather than diving in and starting to use it. Luckily, the solution to this was simple. A sophisticated makeover of the visual design, especially concentrating on the landing pages, registration, and first few minutes of use, dramatically improved the response of adult users and increased the percentage who actually started using the product.

Depending on your industry and market, your company may be trying to convey trustworthiness, edginess, playfulness, warmth, or hundreds of other emotions to your users. Would you put your money at a bank that looked like a surf shop or a fast food restaurant? You probably wouldn’t even go inside. When a user first comes to your site or opens your product, they’re making an almost instantaneous decision about whether this product is what they’re looking for. Visual design plays a huge role in that decision making process.

Why I Put Off Visual Design

Just because it’s incredibly important, doesn’t mean that I spend huge amounts of time on visual design early in the process. In fact, I tend to nail down the vast majority of the interaction design before moving on to applying a good visual design. I do this for several reasons:

It is much faster for me to iterate on something if I don't have to worry about making it pixel perfect. When I get feedback from a user test or a beta customer, simple, grayscale wireframes can be changed quickly and without too much regard for things like margins and font sizes, while fully designed visual mockups can take much longer to fix.
Remember how I said that visual design sets the tone almost instantly for a user? Because of this, visual design can screw up interaction testing. If your tester has an immediate positive or negative reaction to the visuals, you're going to get different information than you would if they could effectively ignore the visuals. Grayscale wireframes or Balsamiq-style sketches make it much easier to ignore the look and concentrate on the interactions.
I am going to want to make several versions of visual design to test independently. That's easier for a visual designer to do if they have a reasonably nailed down version of the wireframes to work from. The last thing you want is to try to test various different versions of visual design along with different versions of interaction design, since it makes it much tougher to know which element is affecting the test results.
Once some of the basic interactions are scoped, engineering can get to work on the product and then apply a visual design later. This means they’re not waiting around for me to get both the interactions and the visual design done upfront. Engineering, interaction design, and visual design can all be happening in parallel (provided you have at least three different people to work on the different areas).
And, of course, if it’s a brand new product, your visual design is actually going to be significantly less important to highly motivated early adopters using a beta product than it will be once you’re trying to attract a larger, more mainstream market. Early adopters forgive a lot, but they’re more likely to forgive ugly than hard to use, so concentrate on making it easy and useful first, and make it pretty later.

7 Ways Design Can Speed Up Your Product Development

2010-04-27T13:23:00.000-07:00

One thing I hear a lot from startups that don't want to bother with design is, "But design will slow us down!" They're sure that the fastest way to get a good product in front of users is to just jump in and start coding and then iterate on the feature or product once it's been built and released. Guess what? They're wrong.

A good designer who is willing to work with a lean development team can remarkably speed up your progress. Let's look at some different ways that design can help you quickly get your product into the hands of your customers.

I Can Design Faster Than You Can Code

So, you think you're a pretty fast coder, right? I believe you. But, unless the feature is dead simple, you can't build a working version of it faster than I can build a fake version of it. Remember, my prototype doesn't have a backend or a database. It doesn't have privacy controls. It doesn't have to support multiple users. It doesn't even have to work at all. It just has to look like it works.

Getting rid of all those requirements makes my prototype really, really fast to build. That means I can build several versions of it in the time it might take you to get the backend set up. If I make those versions seem like they work, I can get the prototype in front of users, get their feedback on it, and iterate much faster than you can.

That means, by the time you build the real version, it's already been through two or three versions and validated with customers. Many of the major usability problems will already have been found and eliminated. Complicated things will have been simplified. You'll already know how it integrates with the rest of the user interface. If you assume that you'd need to do all of these things anyway, this will save you time.

I Don't Need to Design it All Upfront

Maybe you've only worked with waterfall-style designers in the past, but there are some of us who don't need two months and a full set of specifications to get a product designed. We can get started on the broad outlines just a little before you begin coding and have something ready for you to get started with right out of the gate.

Besides, all of the products I've ever worked on had at least some non-customer facing code that needed to be written, so maybe you could write that part first. If you have some idea what you're coding based on things like customer stories (you did write customer stories, didn't you?), you can get started on anything the user doesn't see, and I can get started on everything the user does see. By the time you're done making sure the feature isn't going to bring down the servers, I can have built and tested a few mockups.

It Takes Very Little Time to Not Build A Feature

Imagine that the product owner has a fabulous idea for a feature that will really wow a particular set of current users. Now imagine that you build that feature and get it in front of those users and they just shrug, totally unimpressed. I'll bet that's happened to some of you.

Now, imagine that, instead of taking the time to build the feature - even a minimum viable version of the feature – I was able to determine that the users wouldn't be the least bit interested based on interactive mockups. Granted, this doesn't work in every case, but it can sure save a lot of time when it does. It might even save enough time to let you build something that we figured out users DO want based on our interviews using the mockups.

I've Probably Designed One of Those Before

I don't know how to design a multi-threaded event system that can scale to millions of users. That's ok, because it's not my job to design those. My job is to know how many steps I should break a particular registration flow into and to have some idea of how that will affect user behavior. My job is to know when and how to use progressive disclosure of information to keep a screen from becoming too cluttered and distracting. My job is to know how to subtly encourage a user to follow a call to action.

Now, maybe you know how to do all of those things too, but most engineers I've worked with don't. I know how to do all of those things because I've spent a lot of time designing things, reading about designing things, talking about designing things, and watching people use things that I or other people have designed.

Just because you can put a button on a screen doesn't mean you know how to get people to click on it. This means that you may need a lot more iterations to get to the right solution than I do. This means that things tend to go faster if I'm the one deciding design things and you're the one deciding engineering things. Don't worry. I won't tell you how to write your multi-threaded event system.

I Know How to Change Things Without Hurting the UX

There are a lot of ways of doing most things in an interface. Some of them take a lot longer to code than others. Now, maybe you're one of those few engineers who love to do things the hard way, but most of the ones I've worked with have deadlines and want to get things done as fast as possible. This may mean not wanting to do the things that take a long time.

When you're deciding how to build things, I can help you understand how your decisions are going to affect the user experience and suggest other ways to get the same effect, some of which may be faster to build.

Tossing a Prototype Has No Side Effects

Let's say we do let you just start coding without testing any mockups. Let's say that a whole lot of what you built wasn't right. That means you may have to throw out large chunks of your code.

Now let's say that we test my mockups and that I was the one who wasn't right. That means that I have to throw out large chunks of my prototype. But you know what? I was going to throw away the entire prototype anyway, so it's not that big of a deal. I mean, I don't love being wrong, but that's part of the process. Sometimes we don't get it right the first time.

Throwing away a stand-alone prototype is a lot less hassle than ripping code out of production. It never generates any surprising bugs in other parts of the code. It doesn't leave any bad data in the database. It has absolutely no impact at all, because it's just a prototype and was always meant to be thrown away. That's not always true of production code, especially when you're dealing with adding a feature to an existing product.

I Can Keep You from Throwing Away Something Almost Great

You spent all that time building something and now users hate it, so you have to throw it away, right? Maybe not. Even the best feature can be so confusing or poorly designed that users don't know how desperately they want it.

Having a decent, validated design up front means that you're less likely to throw out a feature that has potential. At least if users hate the feature, you'll know that they actually hate the idea and not just that they didn't understand how to use it or what it was supposed to do.

How Do You Get Started?

Now, to be clear, not all designers are going to improve your velocity. You need to integrate the designer into your development team and make sure that he or she is comfortable working at these sorts of speeds and in a startup environment. You can read my post on Embedded Design for more tips on how that works or send me email at laura at usersknow dot com with specific questions that I will try to answer.

Make sure you get somebody who can quickly make interactive prototypes and who has enough of a background in user centered design to get feedback on those designs. Also, get your designer involved very, very early. As soon as your product owner is writing user stories, the designer can start thinking and sketching, so the design can get a bit of a head start on development.

If you do it right, your end product will not only be significantly better, but it will also take less time to develop. And yes, at times it may seem like it will postpone coding for a brief period at the beginning of the development cycle, but it will actually speed up learning, which is something I think we can all agree is a good thing.

Like the post? You should follow me on Twitter!

6 Questions To Ask Before Launching a New Feature

2010-04-19T12:03:00.000-07:00

There are a lot of things to pay attention to when you’re launching a new feature. This post is going to cover a few of the questions that you really ought to ask before launch, but may not have in the past. Now, to be clear, these are not the obvious questions like:

Is this addressing a user need or pain point?
Has this been through and passed some form of QA testing?
Do I have a marketing plan for letting users know this exists?
What is the expected ROI for this feature?
Will this feature scale to a reasonable number of users?
Is this feature usable, and is the UX reasonably consistent with the rest of my product?

If you have not asked the above questions about the feature you’re about to release, then you should stop reading this blog post and answer them right now.

The questions we’re asking today are more advanced, but your launch will go a lot more smoothly if you ask (and answer!) them ahead of time.

How Am I Going to Get Feedback on this Feature?

It’s not enough to just launch a feature and see if your revenue increases. You need to have some sort of plan for testing to see how it’s affecting key metrics, whether those are revenue, retention, registration, user happiness, or some other number you care about. You need to have a strategy for finding how users feel about the new feature, whether they’re using it, and how it’s affecting your bottom line.

A good answer to this question might include some or all of the following:

I am initially releasing this feature to a small cohort of random users and A/B testing key metrics of the sample group vs. the control.
I have embedded a short survey or feedback link into the page and will gather qualitative data from my users that way.
I have contacted my customer service/community management team and prepped them with information about the new feature and asked them to get back to me if we start to see any significant new support requests.
I have already released the feature to a small, select group of beta customers or my customer advisory board, and I am meeting with them to discuss their feedback.

Can I Change this Feature Based on the Feedback I Get?

If your feature is absolutely perfect the first time you release it, either you’re incredibly lucky, or you didn’t release it early enough. Let's just say, that's not usually the case, so just because your feature is released doesn’t mean you don’t need to put it on the schedule for the next sprint! Please plan for iteration.

A good answer to this question should really include all of the following:

I have both development and design time scheduled for quick follow up work on the feature over the next few sprints.
My code base can be changed or refactored to support major changes to the feature.

Do I Have the Time and Willingness to Support and Improve this Feature Going Forward?

Your dedication to this new feature needs to extend past immediate iterations. Remember, if significant numbers of users adopt this feature, you could end up supporting it for a very long time. The more features your overall product has, the harder it can be to support all of them well and the more complicated your product can get, so this is not a question that should be treated lightly.

The answer to this question must be:

I have enough design, engineering, QA, and customer support resources to dedicate to supporting and improving this feature without causing me to neglect any of the other important features that I’m already supporting. This is not just true for the upcoming sprint, but for the entire future of this feature.

What’s My Plan for Killing the Feature if Things Go Wrong?

There are a lot of reasons to kill a feature. Your users hate it or don’t adopt it at a high enough rate to justify supporting it. It causes massive scalability or performance problems. It doesn’t fit in with the rest of your offering. The ROI isn’t good enough. Your UX has become too cluttered, and you need to pare down your feature list.

You must be prepared to kill features, and it’s always easiest to kill a feature in the earliest stages before it develops a small but vocal group of devoted users. It’s also easier to kill a feature when you have a plan for doing it.

The answer to this question should include several of the following:

I am closely monitoring the feature to let me know if it needs to be killed for performance or scalability reasons.
I have an automatic rollback system in place that will shut down the rollout of this feature if there are obvious and immediate performance problems.
I have a kill switch on the entire feature that will allow me to shut it down manually or limit the damage if performance or scalability problems threaten the system later.
I have clearly labeled the feature as “beta” or “experimental” to let users know that it is being evaluated, and I have provided a clear warning that it is not necessarily permanent.
I am rolling the feature out to a small group of users first, so that, if I have to kill it, it will not affect my entire user base.
I am not charging customers for it yet, since I know that it is much harder to pull a feature that customers have paid for than one that is provided for free. Or, if I am charging, I have a plan in place for how to compensate users when it is killed.
I have alerted my customer service or community management staff so that they can prepare a consistent message to my users if the feature must be killed.

Is This Feature Likely to Annoy Some of My Users?

Obviously, it would be lovely if all of your products features thrilled all of your users. Still, sometimes you know that a certain group of people are likely to be antagonized by a particular change, and you decide to make it anyway. That’s a completely valid business decision, but you need to be prepared for the reaction and ready to contain the damage. As a note, this applies equally to killing features, so feel free to ask yourself this question before you pull the plug.

Appropriate answers to this question may include:

It may anger an acceptable percentage of my users, but my customer support and community management teams are fully prepared with a good message and possibly some sort of compensation for unhappy users.
It is possible that it will anger enough of my users that I will have to implement my well prepared plan to kill the feature.
It is extremely unlikely to annoy anybody, but I have still briefed my customer contact points that a new feature is being released so that they can immediately inform me in the event of unexpected problems.
I have already reached out to the group of users that this feature is most likely to antagonize and included them in the design process, so they feel more invested in the feature and are less likely to complain.

Do I Have a Plan to Monetize it?

I toyed with putting this one into the list of really basic questions that nobody would need reminding of, but then I realized that I was being hopelessly optimistic. Look, I don’t actually think that every single feature needs to make direct revenue. I’m pretty sure that just making your customers happier leads to more revenue if you have any sort of monetization model in place. That, unfortunately, is a big “if,” so make sure that you’re at least asking the question about whether you should be monetizing a feature before you ship it.

Changing your monetization strategy after the fact can be quite difficult. Suddenly and without warning charging for something you’ve been giving away for free can anger your customers. Deciding to give away something you’ve been charging for can make customers who paid for it feel ripped off. And, once you’ve made something available to the general public, it can be impossible to then make it exclusive. So, at least have a plan for what you’re going to do with this thing in terms of making money off of it, ok?

Appropriate answers to this question could be along the lines of:

While this feature will not earn direct revenue, it is expected to increase registration/retention/happiness and will cause users to sign up for the service or remain subscribers/paying customers.
This feature will be part of a premium package that will only be available to subscribers/VIP members/paying customers.
This feature will be an add-on that users will have to pay to access.
This feature will be free while it’s in beta, but there is an end date at which we will begin charging, and users are made aware of this fact before they start using it.
It will be very clearly stated that this feature will be available to everybody for a trial period, but they will have to pay to continue using it.
The basic version of this feature will be available to everybody, but an upgraded version will cost money.

My God, How Long Will This Take?

I know it seems like a lot to do, but really, a lot of these questions can be handled once for your whole product. For example, putting in a system to kill or roll back dangerous features can be done upfront and used for all product improvements.

Besides, each individual question shouldn’t take that long to answer. And trust me, however long they do take, answering these questions now will save you a huge amount of time in the future.

Like this post? Do some of the following:

Tweet about it!
Follow me on Twitter.
Come hear me speak at the Startup Lessons Learned conference in San Francisco on April 23rd. Use the code “LAURAK” for 20% off your ticket price!
Hire me to help you learn how to get better information from your customers.

Your Users Are Doing Something Surprising

2010-04-14T14:10:00.000-07:00

This post is for all of you lucky enough to have a product with real users. Way back before you had users, or even a product, you probably went through a process to figure out what you should build. During that process you may have written user stories and work flows that described, in various levels of detail, how your users would perform each expected task. But you know who didn’t read your user stories? That’s right: your users.

The result? Somewhere out there, a whole lot of your users are doing something totally unexpected with your product.

Ok, maybe it’s obvious that sometimes users do the unexpected. Maybe you expected your SMS-based social network to help people find out where their friends are in real time, but instead celebrities started using it to market directly to their fans. Maybe you created a product designed to help bands promote themselves, but instead ended up with a social network where people hook up with their high school sweethearts. Or, although this seems extremely unlikely, maybe you had an MMO but people just used it to share photos.

Whatever they’re doing, it’s something you didn’t expect, but it’s very important that you learn what it is as soon as possible!

Why Should You Care?

There are three excellent reasons for you to know what your customers are actually doing with your product:

So you know if you are missing an opportunity to pivot your product or marketing
So you know if you are missing an important feature
So you don’t accidentally destroy a commonly used workaround or "unplanned feature"

Missing a Huge Opportunity

The first reason is pretty well demonstrated by the examples I’ve already given. Flickr could have gone on being an MMO that its customers also used for sharing photos, but the company really took off once they jettisoned the majority of their product and started concentrating on the part that customers found most valuable. Figuring out the surprising thing your customers are doing with your product allows you to decide if you would be better off eliminating 90% of your product and concentrating on the 10% people actually want.

Missing a huge business opportunity doesn’t only apply to pivoting your product. You could also be missing a huge marketing opportunity. One of the benefits to understanding more about what your users are doing with your product is that you can start to segment them better. For example, maybe a small group of your users are using your product in a very particular way. Now, maybe those users are all alike in some important way. If this is true, maybe you should be marketing this new use of your product to other people like them or emphasizing this use in your marketing materials.

Missing a Feature

Sometimes understanding what your users are trying to do with your product can tell you a lot about what you should be helping them do with your product. We all know that users are terrible at predicting their future behavior, but this isn’t their future behavior we’re talking about. It’s their current behavior. Your customers want to do something with your product so badly that they’re going out of their way to come up with clever ways to do it on their own.

Again, to use Twitter as an example, think of retweets and hashtags. Neither of those was built into the original product, but users saw missing functionality and provided it.

Destroying a Workaround or Unplanned Feature

The third reason is a little less obvious, but it’s an outstanding way to make yourself loathed by some of your most passionate users. Sometimes even a small change can completely destroy a workaround that people were using to get your product to behave the way they want it to.

Some form of this happens virtually every time a new browser update gets released. Each time a new version of IE or Firefox comes out, thousands of web developers tear their hair out trying to figure out why the website that looked so lovely in the previous version starts looking like it was designed by Salvador Dali. The reason is often because some hack that they used to achieve a particular effect is no longer supported in the upgraded version.

Now, you can argue with me all you want about how web developers shouldn’t be using browser hacks to get their CSS right or that this only happens to bad developers, but let me remind you that this is just an example. And, in this example, the web developers represent your paying customers. If you ruin the way that they’re used to doing something, they become a lot more likely to go find somebody else who will let them do things the way they want.

How Can You Find Out What They’re Doing?

This is one of those cases where there is absolutely nothing more beneficial than watching your current users interacting with your product in their own environment. You will never get this information in usability tests with people who have not used your product, and you will not get nearly as much good information by simply interviewing current customers without observation.

Let me be perfectly clear: the best way to get this information is to spend time observing your users in their natural habitat, whether that’s their home, office, car, or wherever they use your product. You should schedule appointments with them at times when they would naturally be using your product anyway, and you should ask them to think out loud as you sit quietly in a corner and take notes. After they’ve spent a good amount of time with your product, you can interview them about their experiences and follow up on any workarounds or odd behaviors that you saw. Note: Be respectful, please! Don’t make your customer feel like a freak for doing something you didn’t expect.

If you absolutely can’t be in the same room as your user, you can also get good information by observing remotely using screen sharing and webcams, but in person is really ideal. One of the big reasons that simply interviewing them about their habits won't work as well is that they don't realize they're doing anything unexpected, so they can't tell you where their behavior differs from what you expected.

You should, of course, also be recording as many facts as possible about usage stats so that you have a statistical overview of which parts of your product are being used most, but this is one of those cases where you can learn things you never expected just by watching customers in person. It's often tough to spot these things in data. In fact, you may not even think to gather data on this particular behavior, since you didn't expect it in the first place!

What Should You Do About It?

Obviously, this doesn’t mean that you can never change anything that a customer is using as a workaround unless you’re totally pivoting your whole product. That would be insane. What it does mean is that you should be aware of what your actual users are doing so that you can improve the experience for them, or at least not make it worse by removing a beloved workaround or unplanned feature.

Remember, if your customers want so very badly to use your product for a particular purpose, even if it’s not the purpose you originally intended, you might want to make it easier for them to do it.

Like this post? Do some of the following:

Tweet about it!
Follow me on Twitter.
Check out my related post: 6 Reasons Users Hate Your New Feature.
Come hear me speak at the Startup Lessons Learned conference in San Francisco on April 23rd. Use the code “LAURAK” for 20% off your ticket price!
Hire me to help your company get to know its users.

Pain Driven Design

2010-04-08T12:45:00.000-07:00

I have a problem with both User Centered Design and Customer Driven Development. This may come as something of a shock to people who read my blog, since I’m constantly giving advice on better ways to talk to users, analyze data, and improve customer development efforts.

The problem I have with UCD and CDD is not the methods. The problem is that people so often misunderstand them. People hear “user centered” and think, for some insane reason, that we are encouraging them to turn their entire design process over to the user. They hear “listen to your customer” and think that we want them to blindly accept every ridiculous feature request made by somebody with a credit card and an opinion.

Guess what? I rarely speak for entire communities of people, but I think I can safely say that nobody in the user centered design or customer driven development business are asking that of anybody. If they are, they’re idiots and shouldn’t be listened to anyway.

Unfortunately, a lot of us have debunked this myth by explaining that we don’t actually think that customers can predict future behavior (even their own) or that they’re going to have some grand design vision for your product. We just think that customers are great at talking about their problems and pain points, and that those are good things for you and your designers to know when you create a new feature or product.

I’m suggesting that we come up with a new name that will be harder (or at least more fun) to misinterpret: Pain Driven Design.

What Is Pain Driven Design (PDD)?

The PDD methodology requires that, before you start design for a product or feature, you need to figure out what is causing pain for your users and potential users. The desired outcome of PDD is to make that pain go away by some brilliant method of your devising. You then check to make sure you made their pain go away without causing them any more pain.

Is There a Clever Analogy?

There is! Imagine you’re a doctor. You want to help people feel better. The first thing you need to do is to talk to patients who come to see you. Now, of course, you’re not going to ask them what disease they have and how they think you should treat it. You’re going to ask them, “Where does it hurt?” You’re probably going to also ask them a lot of other questions about how they normally feel, what their medical history is, what their family is like, and other things like that. You’ll probably also, if you’re a good doctor, examine them, double check their reported symptoms, and check for symptoms they may not even know they have.

Then, when you have figured out what is causing them pain, you will determine a good course of treatment that is likely to work based on your knowledge of various diseases, your extensive medical training, other work in the field, and how the patient reacts to treatments. You will then closely monitor their progress and adjust your approach as necessary.

Pain Driven Design is a lot like that. You will talk to your users and potential users and find out what causes them pain when they are trying to solve a problem. You will interview them about their habits, likes, and dislikes. You will observe them using the product or competitors' products and look for commonly appearing symptoms. You will then decide how to go about curing their pain. And, of course, you will closely monitor all of your users to see how they respond to your treatment. Since you have a large number of users, and there aren’t any pesky rules against human experimentation in this context, you will run tests to see which treatment performs best.

Does It Work Before I Have a Product?

Sure it does! Presumably, your eventual product will solve somebody’s problem, yes? Maybe their problem is that it is too hard to find a great restaurant while traveling or that they are sort of bored while waiting for the train. OK, those don’t seem like big problems, but they are problems nonetheless and should have solutions.

Since you don’t have a product yet, you need to figure out how people are currently solving this problem. Are they using a similar product? A completely different one? Are they simply suffering in silence without even knowing that their lives would be infinitely better if this problem would go away?

You can find these things out by asking people about how they’re dealing with their problems and what the pain points are with their current solutions (or non-solutions). You can learn more about their pain by watching them suffer through it. Don’t worry, it’s not so bad to watch them suffer, because you know your product will help them!

What If I Already Have a Product?

It still works! You see, no matter how much you love your product, unless it’s perfect, it’s causing pain to somebody. I’m sure it’s not on purpose. You’re not a monster. But something about your product is confusing or hard to use, and it’s driving at least one of your customers nuts.

Again, examine them. Find out when they feel the most pain while using your product. Watch brand new people use your product to see how much pain it takes to learn. Watch old customers use your product to figure out what weird workarounds they've created to avoid the pain that you’re causing them.

Find all the pain points. Then come up with devastatingly clever ways to fix them.

What If My Product Is Disruptive?

It's still solving a problem, right? Even if it's solving that problem in a completely novel way or solving it for a new group of users, presumably if people are going to adopt the product, the product will be solving a particular problem for them.

That’s great . Even if people have never seen anything like your product, you can get a huge amount of information by talking to users about how they currently solve their problems as well as their general environment for problem solving. And once your disruptive product has been launched, chances are it’s causing some people some pain, so you should observe people interacting with it to learn where the pain points are.

What If My Customers Try to Tell Me How to Fix Their Problems?

Well, I suppose you could plug your ears and scream loudly so that you won’t be in danger of hearing them talk about their solutions. Or you could listen to their solutions and then politely follow up to make sure you understand the underlying pain that they’re trying to cure. Frankly, I prefer the latter approach, but it’s up to you.

One interesting thing that I’ve found in my many, many years of listening to customers is that sometimes the customers are right about the solution. I know, crazy! I mean, we've been assured by hundreds of people that listening to customers' solutions is completely useless and that they're always wrong! Guess what? They're not. This doesn’t mean you should take their word as gospel, but I can't imagine that people within your company have a patent on coming up with the right solutions to customer problems. Just because an idea for a solution comes from a user doesn’t automatically make it useless or wrong, even if the anti-UCD crowd seems to think so.

How Will Pain Driven Development Be Misinterpreted?

Who can say? I sort of hope somebody reads only the title of this post and writes a scathing retort about how I’m a horrible person for advocating that designers be subjected to pain until they come up with better designs (note: they shouldn’t, except in certain very specific cases, and they know who they are). Or maybe they’ll dissect my doctor/patient analogy and point out all the ways in which it’s flawed (note: there are 17 ways. See if you can spot them all!) and thereby conclude that, since my analogy isn’t perfect, my methodology must also be crap.

But I hope a few people will say, “Hey, that Pain Driven Design methodology is just a catchy name for understanding our customers’ problems so that we can come up with better solutions!” And more importantly, I hope a lot of people will say, “You know, that Pain Driven Design sounds like a great idea, and we should try it in our organization!”

Like the post? You could leave me a message in the comments, tweet about this post, read some of my other posts, tweet about them, or hire me to help you learn how to find your customers' pain points! Those are all valid options.

You should also follow me on Twitter.

5 Mistakes People Make Analyzing Qualitative Data

2010-04-02T12:43:00.000-07:00

My last blog post was about common mistakes that people make when analyzing quantitative data, such as you might get from multivariate testing or business metrics. Today I’d like to talk about the mistakes people make when analyzing and using qualitative data.

I’m a big proponent of using both qualitative and quantitative data, but I have to admit that qualitative feedback can be a challenge. Unlike a product funnel or a revenue graph, qualitative data can be messy and open ended, which makes it particularly tough to interpret.

For the purposes of this post, qualitative information is generated by the following types of activities:

Usability tests
Contextual Inquiries
Customer interviews
Open ended survey questions (ie. What do you like most/least about the product?)

Insisting on Too Large a Sample

With almost every new client, somebody questions how many people we need for a usability test “to get significant results.” Now, if you read my last post, you may be surprised to hear me say that you shouldn’t be going for statistical significance here. I prefer to run usability tests and contextual inquiries with around five participants. Of course, I prefer running tests iteratively, but that’s another blog post.

Analyzing the data from a qualitative test or even just reading through essay-type answers in surveys takes a lot longer per customer than running experiments in a funnel or looking at analytics and revenue graphs. You get severely diminishing returns from each extra hour you spend asking people the same questions and listening to their answers.

Here’s an example from a test I ran. The customer wanted to know all the different pain points in their product so that they could make one big sweep toward the end of the development cycle to fix all the problems. Against my better judgment, we spent a full two weeks running sessions, complete with a moderator, observers, a lab, and all the other attendant costs of running a big test. The problem was that we found a major problem in the first session that prevented the vast majority of participants from ever finding an entire section of the interface. Since this problem couldn’t be fixed before moving on to the rest of the sessions, we couldn’t actually test a huge portion of the product and had to come back to it later, anyway.

The Fix: Run small, iterative tests to generate a manageable amount of data. If you’re working on improving a particular part of your product or considering adding a new feature, do a quick batch of interviews with four or five people. Then, immediately address the biggest problems that you find. Once you’re done, run another test to find the problems that were being masked by the larger problems. Keep doing this until your product is perfect (ie. forever). It’s faster, cheaper, and more immediately actionable than giant, statistically significant qualitative tests, and you will eventually find more issues with the same amount of testing time.

It’s also MUCH easier to pick out a few major problems from five hours of testing than it is to find dozens of different problems from dozens of hours of testing. In the end though, you’ll find more problems with the iterative approach.

Extrapolating From Too Small a Sample

I always do this don’t I? Say one thing, and then immediately warn you not to go too far in the opposite direction. The thing is, I get really tired of running five person tests and having a product owner only show up for one session and then go off and address whatever problems s/he saw during that one hour. One or two participants aren’t enough to really get a sense of the pattern of problems in your product.

Besides, I have this little rule of thumb I’ve developed for studies. No matter how great your screener or recruiter, on average for every 10 participants you schedule, one will be a no-show, one will be some sort of statistical outlier (intelligence, computer savvy…something), and one will be completely insane. If the product owner happens to show up only for one of the last two types, their perception of the product’s problems will be totally skewed.

I had one product where we interviewed ten people over the course of two tests. Nine of the ten people were wildly confused by the product, but one, who I swear was a ringer, nailed all the tasks in record time. Guess which session the product manager showed up for? Yeah.

The Fix: As the person making the decisions about what changes you should make in your product, you should be attending all or at least most of your user interview sessions, even if you’re not running them yourself. You should also be looking directly at all of your survey data, not just skimming it or reading a high level report. Honestly, if you’re the one making decisions about product direction, then you are the one who most benefits from listening to your users. If you’re not paying attention to the results, then the testing is really just a waste of time.

Look at all your data before drawing conclusions. I mean it.

Trying to Answer Specific Questions

Qualitative data is very bad at answering specific questions like “Which landing page will people like better?” or “How much will people pay for this?” What it’s great for is generating hypotheses that can then be tested with quantitative means.

In more than one test, I’ve had clients ask me to test various different images to use on landing pages to see which one was most appealing. I always explain that they’re better off just doing a split test to see which one does best, but sometimes they insist. Unfortunately, these sorts of preference differences are often very subtle. Since people are not making the decisions consciously, it's very hard for them to explain why they prefer one thing over another. We always end up getting a lot of people trying to rationalize why they something, and I rarely trust the results.

The Fix: Use qualitative data to generate hypotheses that you then test quantitatively OR to find major problems in your interface. Don’t try to use qualitative data to get a definitive answer to questions about expected user preferences.

Ignoring Inconvenient Results

Because qualitative testing doesn’t generate hard numbers, it’s easy to let confirmation bias sneak into the analysis. While it might be tough to argue with “this registration flow generated 12% more paying customers than the other one,” it’s pretty easy to discount problems observed in user sessions.

I dealt with a particularly resistant product owner who had an excuse for every single participant’s struggles with the product. One was unusually stupid. Another just didn’t understand the task. Another actually understood it but was, for some reason, actively screwing with us. This went on and on while every single participant had the same problems over and over. Also, the discussion guide, which the product owner and everyone on the team had originally thought was perfectly fair, suddenly became wildly biased and the tasks were judged to be impossible. The problem couldn’t possibly have been with the product!

The Fix: If you are finding fault with all of the participants or the moderator or the questions or the survey, it’s time to get somebody neutral into the room to help determine what is biasing the results. Hint: it’s almost certainly you.

Remember, your customers, moderator, and test participants don’t have a stake in making your product seem worse than it is. You, however, may have an emotional stake in making it better than it actually is. Make sure you’re not ignoring results just because they’re not what you want to hear.

Not Acting on the Data

Why would you even bother to run test if you’re not going to pay attention to the results? I mean, tests aren’t free. Even running surveys has an associated cost, since you’re interrupting what your user is doing to ask them to help you out. And yet, so many clients do exactly this.

One client I worked with wanted to set up a system where they ran tests every week. They wanted to have a constant stream of users and potential users coming in all the time so that they could stay in contact with their users. I thought this was a fantastic idea, and so I started bringing people in for them. Unfortunately, after a few months, people began to complain that they were hearing the same problems over and over again.

I explained that they were going to continue to hear the same problems over and over again until they fixed the problems. I gave them a list of the major issues that their current and new users were facing. Every once in awhile, if I complained loudly enough, they would fix one of the easier problems, and unsurprisingly these changes always positively affected their metrics. And yet, it was always a struggle to get the results from the tests incorporated into the product. I eventually stopped running tests and told them that I would be happy to come back and start again as soon as they had addressed some of the major problems.

The Fix: This one should be simple. If you’re going to spend all that time and money generating data, you should act on the results.

I want your feedback!

Have you had problems interpreting or using your qualitative data, or do you have stories about people in your company who have? Please, share them in the comments section!

Want more? Follow me on Twitter!

Also, if your company is currently working on getting feedback from users, I’d love to hear more about what you are doing and what you’d like to be doing better. Please take this short survey!

5 Big Mistakes People Make When Analyzing User Data

2010-03-30T13:13:00.000-07:00

I was trying to write a blog post the other day about getting various different types of user feedback, when I realized that something important was missing. It doesn’t do any good for me to go on and on about all the ways you can gather critical data if people don’t know how to analyze that data once you have it.

I would have thought that a lot of this stuff was obvious, but, judging from my experience working with many different companies, it’s not. All of the examples here are real mistakes I’ve seen made by smart, reasonable, employed people. A few identifying characteristics have been changed to protect the innocent, but in general they were product owners, managers, or director level folks.

This post only covers mistakes made in analyzing quantitative data. At some point in the future, I’ll put together a similar list of mistakes people make when analyzing their qualitative data.

For the purposes of this post, the quantitative data to which I’m referring is typically generated by the following types of activities:

Multivariate or A/B testing
Site analytics
Business metrics reports (sales, revenue, registration, etc.)
Large scale surveys

Statistical Significance

I see this one all the time. It generally involves somebody saying something like, “We tested two different landing pages against each other. Out of six hundred views, one of them had three conversions and one had six. That means the second one is TWICE AS GOOD! We should switch to it immediately!”

Ok, I may be exaggerating a bit on the actual numbers, but too many people I’ve worked with just ignored the statistical significance of their data. They didn’t realize that even very large numbers can be statistically insignificant, depending on the sample size.

The problem here is that statistically insignificant metrics can completely reverse themselves, so it’s important not to make changes based on results until you are reasonably certain that those results are predictable and repeatable.

The Fix: I was going to go into a long description of statistical significance and how to calculate it, but then I realized that, if you don’t know what it is, you shouldn’t be trying to make decisions based on quantitative data. There are online calculators that will help you figure out if any particular test result is statistically significant, but make sure that whoever is looking at your data understands basic statistical concepts before accepting their interpretation of data.

Also, a word of warning: testing several branches of changes can take a LOT larger sample size than a simple A/B test. If you're running an A/B/C/D/E test, make sure you understand the mathematical implications.

Short Term vs. Long Term Effects

Again, this seems so obvious that I feel weird stating it, but I’ve seen people get so excited over short term changes that they totally ignore the effects of their changes in a week or a month or a year. The best, but not only, example of this is when people try to judge the effect of certain types of sales promotions on revenue.

For example, I've often heard something along these lines, “When we ran the 50% off sale, our revenue SKYROCKETED!” Sure it did. What happened to your revenue after the sale ended? My guess is that it plummeted, since people had already stocked up on your product at 50% off.

The Fix: Does this mean you should never run a short term promotion of any sort? Of course not. What it does mean is that, when you are looking at the results of any sort of experiment or change, you should look at how it affects your metrics over time.

Forgetting the Goal of the Metrics

Sometimes people get so focused on the metrics that they forget the metrics are just shorthand for real world business goals. They can end up trying so hard to move a particular metric that they sacrifice the actual goal.

Here’s another real life example: Once client decided that, since revenue was directly tied to people returning to their site after an initial visit, they were going to “encourage” people to come back for a second look. This was fine as far as it went, but after various tests they found that the most successful way to get people to return was to give them a gift every time they did.

The unsurprising result was that the people who just came back for the gift didn’t end up actually converting to paying customers. The company moved the “returning” metric without actually affecting the “revenue” metric, which had been the real goal in the first place. Additionally, they now had the added cost of supporting more non-paying users on the site, so it ended up costing them money.

The Fix: Don’t forget the actual business goals behind your metrics, and don’t get stuck on what Eric Ries calls Vanity Metrics. Remember to consider the secondary effects of your metrics. Increasing your traffic comes with certain costs, so make sure that you are getting something other than more traffic out of your traffic increase!

Combining Data from Multiple Tests

Sometimes you want to test different changes independently of one another, and that's often a good thing, since it can help you determine which change actually had an effect on a particular metric. However this can be dangerous if used stupidly.

Consider this somewhat ridiculous thought experiment. Imagine you have a landing page that is gray with a light gray call to action button. Let's say you run two separate experiments. In one, you change the background color of the page to red so that you have a light gray button on a red background. In another test, you change the call to action to red so that you have a red button on a gray background. Let's say that both of these convert better than the original page. Since you've tested both of your elements separately, and they're both better, you decide to implement both changes, leaving you with...a red call to action button on a red page. This will almost certainly not go well.

The Fix: Make sure that, when you're combining the results from multiple tests that you still go back and test the final outcome against some control. In many cases, the whole is not the sum of its parts, and you can end up with an unholy mess if you don't use some common sense in interpreting data from various tests.

Understanding the Significance of Changes

This one just makes me sad. I’ve been in lots of meetings with product owners who described changes in the data for which they were responsible. Notice I said “described” and not “explained.” Product owners would tell me, “revenue increased” or “retention went from 2 months to 1.5 months” or something along those lines. Obviously, my response was, “That’s interesting. Why did it happen?”

You’d be shocked at how many product owners not only didn’t know why their data was changing, but they didn’t have a plan for figuring it out. The problem is, they were generating tons of charts showing increases and decreases, but they never really understood why the changes were happening, so they couldn’t extrapolate from the experience to affect their metrics in a predictable way.

Even worse, sometimes they would make up hypotheses about why the metrics changed but not actually test them. For example, one product owner did a “Spend more than $10 and get a free gift” promo over a weekend. The weekend’s sales were slightly higher than the previous weekend’s sales, so she attributed that increase to the promotion. Unfortunately, a cursory look at the data showed that the percentage of people spending over $10 was no larger than it had been in previous weeks.

On the other hand, there had been far more people on the site than in previous weeks due to seasonality and an unrelated increase in traffic. Based on the numbers, it was extremely unlikely that it was the promotion that increased revenue, but she didn’t understand how to measure whether her changes actually made any difference.

The Fix: Say it with me, "Correlation does not equal causation!" Whenever possible test changes against a control so that you can accurately judge what effect they’re having on specific metrics. If that’s not possible, make sure that you understand ahead of time which changes you are LIKELY to see from a particular change and then judge whether that happened. For example, a successful “spend more than $10 promo” should most likely increase the percentage of orders over $10.

Also, be aware of other changes within the company so that you can determine whether it was YOUR change that affected your metrics. Anything from a school holiday to an increased ad spend might affect your numbers, so you need to know what to expect.

I want your feedback!

Have you had problems interpreting your quantitative data, or do you have stories about people in your company who have? Please, share them in the comments section!

Also, if your company is currently working on getting feedback from users, I’d love to hear more about what you are doing and what you’d like to be doing better. Please take this short survey!

An Agile Alternative: Embedded Design

2010-03-23T12:26:00.000-07:00

Last week I wrote a blog post about a depressing example of non-agile design. This week, I'd like to show you an alternative method of design, along with some examples drawn from my experiences. This is not, by any means, a new concept, but hopefully I can convince a few people who still think that agile development and interaction design don't go together.

I've worked as a designer and user researcher with agile teams on several different projects. The trick is that, instead of doing all the research and design up front and then throwing the work over the wall to the engineering team, the designer stays embedded with the team throughout the entire development process.

How does an embedded interaction designer help at various stages of the development process?

Strategy and Product Definition

At this stage, your designer (or design and research team, if you’re well-funded) should be out talking to potential users about their problems. A designer at this point in the cycle can help you define who your customer is, learn how potential customers are currently dealing with the problem that your product solves, and come up with a small set of crucial features that will solve those problems better.

Could a great product manager also do some of these tasks? Sure, and they should definitely be involved or even leading the process. But, in my experience, being involved at this stage allows me, as a designer, to really understand the problems I’m going to be solving and the people for whom I’m going to be solving them. In other words, to have a successful design process, I will have to know all these things anyway, and participating in the earliest stage of research is the most efficient way for me to learn them.

Example: One company with whom I was working wanted to add a new feature that would allow users to play music while interacting with the product. From our discussions with users of the product, we knew that people were already doing this, but in a way that caused them all sorts of issues and made many people unhappy. We interviewed current users about things like:

How they were currently using music with the product?
What benefits did they get from using music with the product?
What major problems were they having with music?

We also looked at the quantitative data that we had about what sort of music was being played, how much people were likely to pay for it, and what percentage of people were active music users. The answers to all of these questions helped us come up with a new concept for music that would solve many of the problems and add some fun new benefits

Design

I don’t have to go over this one, right? You need a designer at the design phase. You could always let your engineers design all the screens and the interactions, of course, but that strategy often goes very, very badly. (not always, of course, but I’ve seen more disasters in this area than I care to remember…)

Besides, if you’re building in a Minimum Viable Product in an agile environment, the design phase itself can be quite short, since the designers tend to stay only a couple steps ahead of the engineers.This means that there is often considerable overlap between the design phase and the development phase, rather than a clear delineation between the two.

Example: When working with a couple of different agile teams practicing scrum, we would start our design process one or two sprints ahead of the development team. When the engineering started, the developers generally started work on the back-end processes, giving us a little more time to get the user facing designs ready.

A side benefit to this approach was that the engineers had a much better idea of what they were going to be building, since some design was done, and they could give better time estimates for their tasks and make better decisions about product architecture. On the other hand, since they didn't need to wait for a "full" design to be done, they could begin development in parallel with design and speed up the whole process by a matter of several weeks.

Development

Your next question might very well be, “What good is a designer during the development phase?” Well, if you've started your design phase only a sprint or two early, the chances are that your product is not completely designed. And it shouldn't be! Things change during the development process that require design changes, especially when you're constantly getting feedback from real users on the work in progress.

Things also change during the design process that have nothing to do with users, of course.

Example: I was working on a fairly large, important feature that had a lot of user interaction. We were at the point where most of the back end work was done, and we had created a well-tested user interface and visual design that engineers were busy implementing.

Every so often, an engineer would come to me and say, “This thing you’re asking me to do will take 3 weeks. Is it worth that?” When that happened, I could sit down with the engineering team and go through our options. Sometimes the piece was so critical to the experience that we had no choice but to forge ahead with it. Other times, we were able to find a solution that was almost as good (and sometimes better) that would take significantly less time to build.

The most important thing was that having a designer on the team at all times allowed us to make those choices with a full knowledge of how they would affect the final product. It let us make smart, fast changes to the design without losing the overall vision.

QA

I know that QA isn’t always its own separate phase of the product cycle, but at some point, I hope that somebody’s looking for problems in your product before your customers find them. Having an embedded designer at this phase is extremely helpful, since whoever is looking for bugs needs to know what’s a bug and what isn’t, and the person who is best suited to answer that question is the person who designed the thing in the first place.

Example: I can’t tell you the number of times I’ve had a bug that an engineer marked as “as designed” that I've had to bounce back to them with the comment, “Well, I sure as hell didn’t design it that way.”

Deployment

If there’s ever a time it seems like you could do without those pesky designers embedded on a team, deployment is that time. After all, what is there left for a designer to do?

Plan for the next version, of course! Once the product is in the hands of your customers, it’s the best time to start gathering metrics, observing users, and coming up with the next set of features for the product.

Also, keeping your designers embedded on a particular product over several versions or product cycles allows them to really get to know the users, the product, and the projected feature roadmap. Designers can help make sure that any new updates, features, and bug fixes remain consistent with the core product and don’t contribute to drift.

You Missed Something

“Hang on,” you’re thinking, if you’re paying attention. “Didn’t you miss the part where you test your designs?” I did not. In my mind, iterative testing is part of all the phases of the project. You should be constantly getting feedback of one sort or another on your designs to make sure they’re usable to your potential customer. User feedback isn’t a separate phase. It’s built into all of them.

Time Commitments

One question that often comes up is whether having an embedded designer means that that person has to be dedicated full time to one, single project. Well, it really depends, but in most cases I’d say no. The design workload on a project tends to wax and wane, depending on the phase.

Obviously, the research and design phases can be resource intensive, but the others often require much less time, which can free the designer up to work on other projects simultaneously. I’ve often been on a project full time during the research and design phases, and available for meetings and questions for several hours a week during the development and QA phases.

The most important thing is that there should always be an informed design resource available to handle the questions and tasks that can come up at any point in the project lifecycle.

Tell me your stories about agile or non-agile design in the comments.

Enjoy the post? Follow me on Twitter!

A Depressing Example of Non-Agile Design

2010-03-19T12:41:00.000-07:00

I recently had a really depressing experience. I was asked to work on a very cool application that focused on my favorite subject. That’s not the depressing part. That part was awesome.

Despite a lot of misgivings, I agreed to use the waterfall development process and design a lot of stuff up front, because that’s what the client wanted. Don’t get me wrong. They were doing some development work on the back end, but the goal was to figure out the entire application UX and feature set, wireframe it, and get a visual design done before starting work on anything customer-facing.

The application was quite large and had several different areas. I specifically designed it so that many of the features were modular, which meant that the development team could leave out whole chunks of the UI without ruining the application. They could then add more feature modules in as they finished them, which would theoretically allow them to launch as soon as they had just a few modules finished. Even though we were using a waterfall method, I didn’t want to design something that had to be absolutely 100% finished and built before it was usable. Basically, I was giving them the option to be agile in their development process, if they chose to, even if everything was designed up front.

After a few months of design, we had a great product roadmap, a list of features we ideally wanted in version 1.0, a list of features that would be acceptable for 1.0 (still FAR more than a MVP), lots of fully interactive wireframes, a gorgeous visual design (not done by me, of course), and a lot of data ready to go on the back end. We had also burned through our design budget, so I turned everything over to the developers and moved on to other things.

After several months, the very first version of the product shipped. This is the depressing part. It included exactly 0% of the original design. That’s right. Not a single element I’d designed made it into the first version of the product. Unfortunately, because all the design work had been done up front, I had no idea why the company made the decisions they had made. Of course, if I had been involved with the project all the way through, I could have stripped down the design in a way that at least tried to stay true to the original vision of the product. Instead, it came across as something that had never really been designed at all.

Oddly, there’s a part of me that’s happy that they did what they did. After all, they ended up releasing a MUCH smaller version of the product than we had originally envisioned. I’m not sure if it’s a Minimum Viable Product – only time will tell if it’s viable, after all – but it has many fewer features than we originally envisioned.

But, in another way, I’m sad. If I could have convinced them to go this small at the very beginning of the project, I could have designed their Minimum Viable Product in a lot less time, and, I think, with a lot better result than they got doing it on their own. More importantly, we could then have worked together to build out new features based on learning from the MVP, and we probably still would have had some design budget left to eventually implement that gorgeous visual design.

I guess my lesson from all of this is that I need to be stricter with companies who want a giant, fully fleshed-out product all designed for them at the beginning of the process. I need to be clearer that they’d be better off designing something very small and immediately implementable and then working from there. And I need to encourage them to spread the design budget out over a longer period of time so that they are never left without design resources. Either that, or I need to let go of my designs once they’ve been handed over to the client and acknowledge that they get to do (or not do) anything they want with them. I just find that really depressing.

Why Your Customer Feedback is Useless

2010-03-15T11:14:00.000-07:00

Here’s the scenario: You have a minimum viable product. You’re talking to your users about it. You’re asking them questions, and they’re answering. But for some reason, it’s just not turning into usable information.

You wonder what’s going on. You imagine that perhapsyour users suck at giving good feedback or else they don’t have anything useful to say. Maybe, you think in a moment of hopeful delusion, your product is so perfect that it can’t be improved by customer feedback.

While these are all possibilities, the reality is that it’s probably not your customers’ fault. So, if you don’t seem to be getting any good data, what IS the problem? Probably one of the following things:

You’re asking the wrong questions

I’ve written before about asking customers the wrong questions, but in summary, customers are very good at giving you certain kinds of information and very bad at other kinds. For example, users are great at telling you about their problems. They can very easily tell you when something isn’t working or interesting or fun to use. What they suck at is telling you how to fix it.

Customers are great at:

Complaining about problems
Describing how they currently perform tasks
Saying whether or not they like a product
Showing you parts of a product that are particularly confusing
Comparing one product to another similar product
Explaining why they chose a particular method of doing something

Customers are bad at:

Predicting their future behavior
Predicting what other people will like
Predicting whether they’ll pay for something
Coming up with innovative solutions to their own or other people’s problems
Coming up with brand new ideas for what would make a product more appealing

To take advantages of users’ strengths, have them describe things like, “Tell me about your most recent experience using the product and how that went for you.” Or ask them questions like, “What about [competitor product] do you particularly enjoy? What do you hate?” You can even ask questions like, “Of the following 5 features, which would you prefer?” You’ll want to be a lot more careful about listening to their answers to overly broad questions like, “What brand new feature would you like to see implemented?” or “What would make this product more fun to use?”

You’re asking the right questions the wrong way

It takes practice to ask good questions. Sometimes you’re too close to your product to be objective, and other times you don’t have very good moderation skills. Whatever the problem, you need to make sure that you’re being a good interviewer and not biasing the data.

One of the best ways to improve at this (aside from reading the above posts or spending years as a user researcher), is to have somebody objective and honest give you feedback on your interview skills. Get somebody else in the room who will watch you interview and tell you if you’re asking questions well. Make sure they read the above posts first though, so they’ll know what to look for.

You believe everything customers say

Why bother asking customers questions if you’re not going to listen to them? Well, you are going to listen to them, but you’re also going to verify their answers. Users lie. They don’t necessarily mean to, but they do, often because they simply don’t remember most of what typically happens when they use your product.

The easiest solution is to watch them using your product in addition to talking to them about it. Also, follow up with metrics whenever possible. Maybe they all say that they log into your website every day, but their customer history might tell a very different story. As a bonus, by watching people use your product and checking metrics, you will see your customers’ usage habits, which may vary significantly from what you expect.

You think you already know the answers

You know that saying, “when all you have is a hammer, everything looks like a nail (or a human head, depending on how sick your friends are)?” When you already have a great idea for a feature in mind, everything a customer says may lead you to believe that the feature is a good idea, even when there might be a much better solution.

It doesn’t matter what people tell you if you think you already know what you’re going to hear. When asking for feedback, you need to stay as neutral as possible, and, if you can’t, get somebody else to do the interviewing for you. Also, having more people in the room always helps, since you can compare notes about what everybody else thought the user said.

You haven’t fixed any of their old problems yet

Sometimes, you just hear the same old problems over and over and over. Do you know why that is? It’s because you haven’t fixed some very big problems! I know this sounds obvious, but I’ve worked with enough companies who completely ignored their data to know that it bears repeating. If customers keep complaining loudly about the same things, YOU SHOULD FIX THOSE THINGS. Otherwise, you’ll soon need to get some new customers.

Once you’ve fixed the big problems that are really annoying your users, you’ll be able to have a lot better discussion with them about the other issues they may be having.

You can't turn information into action items

Ok, this one’s hard. I’ve written a bit about how to improve the ROI on your user research, but there’s more to be said about this topic. The real problem is that a user test can generate hours of video and pages of notes, and it can be tricky to distill that down into a task that can be put on your scrum board and implemented by your engineers.

Here are a few suggestions to improve this:

Hold more targeted interviews, tests, surveys, etc. For example, concentrate one series of customer development entirely on your Registration process or your Purchasing flow, and only gather data on that. By focusing, you will simply have less data to comb through, and you’ll be able to go deeper into all the problems with a particular system.
Have several people observe interviews in real time and jot down the 5 most important things they heard during the session. Then quickly discuss everybody’s lists after the session to see if there’s consensus. This eliminates the long, costly process of going back after a series of interviews and digging through all those notes and videos.
Use A/B testing wherever practical to answer concrete questions, like which content improves customer conversion rates or which page layout works better. This means you won’t have to spend time gathering and sifting through certain types of data, and you can focus on areas where qualitative data is more helpful. Unsurprisingly, I have already written about how to more successfully integrate your qualitative testing with A/B testing to maximize efficiency.

You’re not asking them anything!

Of course, the number one reason that people don’t get good data from their customers is that THEY’RE NOT ASKING FOR IT. All of the above techniques require that you be committed to connecting with your customers on a regular basis and getting their feedback and opinions to make your product better. If you’re not doing that, you’re ignoring a huge amount of nearly free information, and your product will likely suffer for it.

I want to hear from you!

Have you tried all of these things and still feel like you’re not getting the information you want? Is there a problem that I’ve missed that has kept you from getting good feedback? How did you learn how to do good customer development? Let me know in the comments.

Like what you read? Want to read more like it? You should really follow me on Twitter.

Seven More Ways People Suck at Customer Development

2010-03-08T11:43:00.000-08:00

I’ve spent many years talking to users about how to improve products. It’s a big part of the job of any UX professional. With the recent emphasis on lean customer development and MVP, talking to customers is no longer limited to a couple of people in an organization. These days, everybody is learning from customers, and that’s a great thing for the usability of products.

Unfortunately, it does cause a few new problems. The main one is that most people who are new at talking to users kind of suck at it.

I’ve written a bit about the mistakes that inexperienced user researchers make when talking to users. However, there are several other serious problems that are far more likely to occur when the people doing the interviewing are deeply invested in the product. If you’re a founder, an engineer, a designer, a product owner, or anybody else with strong, emotional ties to your product, and you’re trying to learn by talking to users, you need to make sure you’re not falling into any of the following traps.

The Big Sale

Often, when you’re sitting down with a potential customer to talk them about your product, your first impulse is to make as good an impression as possible. This is perfectly natural. The problem is, it can really get in the way of getting good information from the potential customer about how to make your product better.

If your goal is to understand what’s not working about your product or what’s preventing people from buying it, stop trying to sell your product to the potential customer. Don’t read off a list of features that are in the product or that will be in the product. Don’t tell the customer how awesome the product is or how good it is at solving all their problems. And, for the love of God, don’t tell them about your Vision for the product.

What should you do? Shut up and let the customer tell you about their problems and how they’d like to solve them. Listen to the customer tell you about his perception of your product and what he sees as wrong with it. Your goal isn’t to convince the customer that your product is great. Your goal should be to let the customer help you make your product great.

The Guided Tour

Talking to users or potential customers can be a little nerve wracking, and a normal reaction to those nerves can be to start a conversation by giving a guided tour of the product. After all, if you’re showing the user a new feature or a brand new product, sometimes they need a little context for what they’re about to see.
What starts out as a quick intro can turn into a long-winded demo describing the product and all of its fabulous uses, which can leave the participant bored, annoyed, and with nothing left to discover on his own.

What should you do? When you’re talking to somebody about your product, it’s often necessary to give a little background, but you need to be careful you’re not giving them way too much information. To help control the flow of info, spend some time before the interview, and write down everything that a user would be likely to know if she were to discover the product or feature or on her own or that she absolutely must know for the product or feature to make sense. Read the intro to the customer, and then don’t say anything else about the product.

The Best Defense

It can be hard to hear people say mean things about the product that you love so much. I know. I’ve been there. I’ve also been present in usability tests where people angrily defended their product against any sort of criticism rather than listening to the feedback and using it to improve things.

Remember, the people savaging your product are telling you important things that you need to know. They’re telling you their perceptions of the product, and if they think it sucks, it’s important for you to find out why they think that rather than convincing them that they’re wrong. Arguing with them will only get them to go away thinking you’re a jerk and won’t do a damn thing to make your product better.

What should you do? Suck it up, cupcake. I don’t care how angry you are at that person for hating your product, you must never, ever show it. This can be tricky, because defensiveness can come across not just verbally but physically. Glowering, rolling your eyes, and sighing deeply are all very, very bad for establishing a comfortable rapport with your customer. If you absolutely can’t be neutral when listening to criticism, have somebody else run the interview and make sure you sit out of the line of sight of the customer.

The Shut Down

Being defensive isn’t the only way to shut down somebody’s opinion. You can also do it by not drawing out the participant in the right way. If you find that you’re getting a lot of yes/no answers or only hearing exactly what you expect to hear, you may be shutting down your interview participants in a more subtle way.

What should you do? Don’t ask for or settle for one word answers. If somebody says something is “cool” or “neat” or “pretty,” follow up. Ask WHAT about the product makes it cool, even if you think already know the answer. Also, smile, connect, make a little small talk, and otherwise do your best to put the person at ease so that they feel comfortable telling you things even if you didn’t ask. Somebody who feels like they’re having a comfortable chat with you will volunteer information, but a person who feels like they’re being grilled by a homicide detective will shut down and give as little information as possible.

The I’m Right, You’re Wrong

Sometimes even good methods can be used for bad reasons. Repeat after me, “User research is not about proving somebody else wrong.” Your goal for a customer interview can NOT be to prove a point to one of your colleagues who thinks differently than you do about a feature, no matter HOW stupid they’re being. When you talk to customers, your goal should be to find out how they feel, NOT to prove that the customers agree with you.

What should you do? If you find yourself saying (or thinking) something along the lines of, “well, we’ll just test it, and the users will like my solution better than so and so’s,” then you need to take a deep breath and let go of your preconceptions. You also probably need to find somebody else to run the interview, since you may be biased beyond all hope of neutrality.

The Helping Hand

It’s great to be a helpful person, and I can completely understand how hard it can be to watch somebody struggle at something that you already know how to do. Sadly, this can go a little too far. I’ve been in usability tests where observers have literally grabbed a mouse from a participant to show them how to perform a task.

But sometimes you need to let people fail so that you can understand how to make things easier for them. After all, you won’t be there in the room with every customer who is struggling to use your product, so the only way to help them all is to let the person you’re talking to fail miserably.

What should you do? Sit on your hands, duct tape your mouth closed, count to one hundred…anything to keep yourself from jumping in to help as soon as a customer or test participant starts failing to use your product correctly. Watch your customer use your product the way she uses it, and don’t try to make her use it the way you think she should use it, even if your way is better. If you have a helpful tip that you really want to share, save it until the end of the interview, and then let her know.

The Complete Denial of Reality

None of these tips will help you if you refuse to listen to what your customers are saying to you. If you go into a customer interview already knowing what the customer is going to say, that’s most likely what you’re going to hear. It’s called confirmation bias, and it is incredibly hard to avoid.

What should you do? Probably the best thing you can do is to simply be aware that the bias exists and work very hard to go into discussions with as open a mind as you can have. Try specifically to listen for evidence contrary to what you already believe. You can also help combat it by having other people in the room with you, preferably ones who are less invested in the outcome than you are or even people who disagree with you about the product. After the interview, discuss what everybody “heard” and figure out if you are all taking away the same message. If everybody gets wildly different messages from the interview, it may be time to bring in a neutral third party.

Why Is This So Hard?

Learning from customers can be tough. If you concentrate on staying relaxed, neutral, engaging, and quiet, you’ll be most of the way toward being successful at customer development. If you’re wondering how you’re doing, try having somebody sit with you in a session and tell you afterward whether you were committing any of the above sins. But, above all, learning from users gets easier with practice, so get out there and start listening!

Want to learn more? You should follow me on Twitter.

How to Make Your Product as Awful as a Corporate Intranet

2010-03-04T11:39:00.000-08:00

Recently I was dealing with intranets for a couple of largish companies, and I realized something. In every company where I've worked or contracted, intranets have been a problem. For some reason, the vast majority of intranets turn into huge, unwieldy, out-of-date, unusable link farms with bad search results. I’m told there are exceptions, but I’ve never seen them. Even small companies have big, awful intranets.

I started to figure out why intranets are so bad, and I came up with a list of problems that all of the ones I’ve seen seem to share. That’s when I noticed that these problems are in no way unique to intranets.

Many products I’ve used or worked on have had at least some of these problems.

None of the products I’ve used has ever managed to combine all of these problems into one enormous, unholy mess in quite the same way that a bad intranet can, but that doesn’t mean you shouldn’t give it a try! To help you out, here is a list of horrible things you can do to make your product just as bad as a typical intranet. Good luck!

Never throw anything away

The first key to really ruining your product’s usability is to never throw anything out.

Have a new feature, link, or module? Throw it onto the main screen along with everything else you’ve ever released! If even one person thinks it’s important or useful, you must be sure to support it until the end of time. This will ensure that your interface is so cluttered that nobody will be able to use the 10% of your product that is actually interesting to 90% of your customers.

Don’t put anybody in charge

It is very important to make sure that no single person is responsible for your overall product. If you want a really terrible product, make sure to enforce this total lack of responsibility and ownership.

After all, if you were to put somebody competent in charge, they might do something rash like develop a vision for the product and make sure that only relevant, useful features got added while unused features were removed. They might also get to know the users of the product and start coming up with new product ideas that would please paying customers. Even worse, they could be held responsible for changes in important metrics and expected to understand what features were affecting trivial things like revenue.

Let everybody change everything

For the love of God, don’t restrict who gets to make changes to any part of the product! With more companies using wikis instead of regular intranets, we’ve learned an important lesson: everybody should be changing everything all the time.

Remember, within your company, everybody’s opinion is, of course, equally valid, and everybody should be independently making decisions about product direction. If you were to actually restrict who got to make changes and when, you might end up with a consistent look and feel to your product. You might even have somebody who actually knew everything that was being added to your product, which would ruin the surprise when one of your customers asks, “what the hell is this new feature, and why does it work like this?”

Ignore anything that doesn’t generate revenue DIRECTLY

Intranets almost never sell anything to anybody, so they don’t directly bring in money. Similarly, there are whole swaths of your product that don’t contribute in an absolutely direct way toward separating your users from their cash. For example, very few people actually pull out their credit card during registration, so it’s very important that you not spend ANY resources improving your registration process, no matter how many potential users give up in disgust. Only a very small percentage of them would have ended up giving you money anyway, so your time is probably better spent rebuilding the credit card submission process for the 37th time or creating a complicated promotion.

Also, you shouldn’t believe people, particularly interaction designers or user researchers, when they tell you that happy, unconfused users are far more likely to pay for or continue using a particular service. Those people are lying to you for fun.

Don’t invest in improving infrastructure

Once a product or intranet is built, it is critical that you NEVER EVER change any of the underlying infrastructure. Do not tear out or rewrite old code. Do not optimize queries. And, I am begging you, do NOT change anything to improve scalability. Changing infrastructure takes time and resources, and all it provides is a more stable platform and a better, less buggy experience for your users. Instead, you should just keep hacking new bits onto the original shaky framework until the whole thing collapses in a heap.

Use as many out of the box platforms as possible

As a corollary to the above, make sure to save time by integrating dozens of different out of the box solutions that almost do what you want them to, rather than taking the time to research and buy or build exactly what you need. There has never been a case where a pre-built, open-source solution was hard to install or maintain. They always do exactly what they’re supposed to, and, because your engineering team doesn’t build them, they are 100% bug and hassle free!

Never do a full UX overhaul

Really, this is key. Whatever happens, do not, under any circumstances, do a complete sweep of your product in order to make sure that the end to end experience is consistent, usable, attractive, and intuitive. If each individual feature seems to make sense, there’s absolutely no reason why hundreds of them wouldn’t all work together seamlessly.

Don’t Test It!

Why on earth would you want to observe people using your product? You might see the places that they’re having problems or getting confused. You might come up with ideas about how to improve the user experience. If you want a truly horrible product, you must never watch a customer fail to use it. Trust me. That will only make it less awful.

Or Do the Opposite…

Of course, if you don’t want your product to be a bloated, confusing mess, you can always do the opposite of the above things. It doesn’t guarantee that your product will be perfect or make you rich, but at least it won’t be as bad as most of the intranets I’ve seen lately.

Enjoy the post? Follow me on Twitter!

6 Ways You May Be Failing at Customer Development

2010-02-26T16:56:00.000-08:00

Listening to users can be difficult for a small company. Most start ups, especially lean ones, don’t have a dedicated person or team whose only goal is to connect with users, gather feedback, test products, or design new features. Often these roles are being filled by founders, engineers, or product owners. This isn’t necessarily a bad thing. In fact, having all sorts of employees connecting with users can be great.

The biggest problem I’ve noticed in situations like this is that the people who are talking and listening to users aren’t really very good at it.

You see, learning from customers can be hard. Sure, people will tell you it’s as easy as sitting down and watching somebody use your product or asking a few questions– and don’t get me wrong, that alone can be valuable! But actually getting the right information from customers and turning it into a product can take some training and practice.

What You Are Probably Getting Wrong
Let’s take a look at some of the most common and costly mistakes I’ve seen when people are trying to do their own customer development without much experience or guidance.

Bad Interviewing Technique
There are a whole host of problems that people commonly have when they first start moderating studies or interviewing users about products. I covered five of the biggest issues in this post for the Sliced Bread Design blog, but there are dozens of ways to screw up an interview.

But why does interviewing technique matter at all? Because that’s how you’re going to get information from your customers! Good technique makes it a lot more likely that you’ll be able to elicit open, honest, actionable feedback from your users. Bad technique means you may not learn anything useful or, even worse, that you may bias the user enough so that you only hear what you expected to hear.

Needless to say, it you’ve never run a user discussion session before (or even if you have), it can’t hurt to brush up on techniques like not giving a guided tour of the product, letting the user fail, not leading the witness, asking open ended questions, letting the user explore, and shutting the hell up. There is a lot more to being a successful interviewer, but fixing those few common mistakes will make getting good user feedback a whole lot easier.

Not Turning Data into Action
Once upon a time I did some work for a company that boasted of being committed to customer development. They brought people into the office weekly and chatted with them. They solicited customer opinions in surveys and forums. They did everything right! Except, when the time came to make product decisions, those discussions with customers were often conveniently forgotten, and the company just implemented features with very little regard for the data they had so painstakingly collected.

Customer feedback can be a wonderful thing, but only if it turns into actionable items that can be incorporated into your development cycle. Don’t collect data just to have it. Go out and get the data that you need to improve various parts of your product. If you’re hearing the same problems over and over again for months at a time, it may mean that you’re not turning your feedback into useful features for users.

Fixing the First Problem You See
This may seem like I’m contradicting the above section, but please don’t run out and fix the very first problem reported by a user. I’ve seen this happen quite a bit, especially when engineers are running or observing interviews. What happens is that an engineer sees a user having a problem and thinks, “Hey, I can fix that!” and is automatically off and running to code a solution. It’s not the engineer’s fault! After all, it’s a great thing to want to fix a problem that somebody is having.

The real issue is that, while it may be a problem for one or two customers, it may not be the biggest problem that needs solving, and running off to fix it can distract you from finding bigger issues. Spending a little time to follow up on the issue with other customers can unearth an underlying problem that could be fixed or a model that needs changing. By investigating reported problems, you can often save yourself a lot of work by fixing several related problems with one elegant solution.

Not Researching Early (Or Often) Enough
User research and customer development are not things that are done at a single point in the development process. They should be done constantly. Too many companies think that user research is limited to one task like market segmentation, usability testing a new product, focus grouping a product idea, or sending out periodic surveys about what new features customers would like to see.

But user research and customer development encompass all of those things and a lot more. Good customer development is an ongoing process that needs to be built into your company’s entire development and release cycle. At some point in the future I’d like to write up the various different types of outreach that are useful depending on the phase of your product, but for now, assume that if you’re not reaching out to interact with a customer this week in some way, you’re doing something wrong.

Not Having a Product Vision (or Not Committing to Changing It)
Perhaps the biggest challenge for customer development is knowing when to say, “I’m going to ignore that piece of user feedback.” I know, I know. It’s incredibly important to listen to your users and add features they need. But the fact is, a product can’t be all things to all people. Adding feature after feature to your product just because customers have asked for them often results in cluttering up the interface and making your product impossible to use.

Sure, listen to your customers and implement new features for them, but if a particular requested feature doesn’t fit comfortably with the rest of your product vision, either don’t implement it or commit to changing your product vision pretty significantly to incorporate it cleanly. And yes, that may mean killing other features that other customers have asked for!

Giving Users What They Say They Want
I’ve covered this before on the Sliced Bread Design blog, but it’s worth repeating. Customers don’t always tell the truth about what they want. They don’t mean to lie to you. They just can’t always answer the question, “what do you want this product to do.”

Sometimes, if you’re not particularly experienced in customer development, it can be tough to make good decisions about what you should really be giving customers. After all, if you can’t trust what they tell you, how are you supposed to make them happy?

The simple answer is to listen to your customers’ problems, not their solutions. There’s more to it than that, but it’s almost always better to listen to a lot of problems and come up with a good solution than to simply implement the fixes or features suggested by users.

What Next?
I really hope that this didn’t scare anybody off from talking to their customers and learning all of the important things that they have to teach you. While getting good customer feedback and acting on it isn’t always easy or intuitive, it’s something that can and should be practiced by people at all levels of your company.

What Do You Think?
Did I miss any common problems that you have had trying to do customer development on your own? I’m interested in hearing people’s stories, good and bad, about how they have managed to learn from their users.

Interested in hearing more about what I think? You should follow me on Twitter.

Which Metrics Equal Happy Users?

2009-12-03T18:47:00.000-08:00

This post originally appeared on the Sliced Bread Design blog.

One of the greatest tools available to me as an interaction designer is the ability to see real metrics. I’m guessing that’s surprising to some people. After all, many people still think that design all happens before a product ever gets into the hands of users, so how could I possibly benefit from finding out what users are actually doing with my products?

Well, for one thing, I believe that design should continue for as long as a product is being used by or sold to customers. It’s an iterative process, and there’s nothing that gives me quicker, more accurate insight into how a new product version or feature is performing than looking at user metrics.

But there’s something that I, as a user advocate, care about quite a lot that is really very hard to measure accurately. I care about User Happiness. Now, I don’t necessarily care about it for some vague, good karma reason. I care because I think that happy users are retained users and, often, paying users. I believe that happy users tell their friends about my product and reduce my acquisition costs. I truly believe that happy users can earn money for my product.

So, how can I tell whether my users are happy? You know, without talking to every single one of them?

Although I think that happy users can equal more registrations, more revenue, and more retention, I don’t actually believe that this implies the opposite. In other words, there are all sorts of things I can do to retain customers or get more money out of them that don’t actually make them happy. Here are a few of the important business metrics you might be tempted to use as shorthand for customer happiness – but it’s not always the case:

Retention

An increase in retention numbers seems like a good indication that your customers are happy. After all, happier customers stay longer, right?

But, do you mean retention or forced retention? For example, I can artificially increase my retention numbers by locking new users into a long contract, and that’s going to keep them with me for awhile. Once that contract’s up, they are free to move wherever they like, and then I need to acquire a customer to replace them. And, if my contract is longer than my competitors’, it can scare off new users.

Also, the retention metric is easy to affect with switching barriers, which may increase the number of months I have a customer while making them less happy. Of course, if those switching barriers are removed for any reason – for example, cell phone number portability – I can lose my hold over long time customers.

While retention can be an indicator of happy customers, increasing retention by any means necessary doesn’t necessarily make your customers happier.

Revenue

Revenue’s another metric that seems like it would point to happy customers. Increased revenue means people are spending more, which means they like your service!

There are all sorts of ways I can increase my revenue without making my customers happier. For example, I can rope them into paying for things they didn’t ask for or use deceptive strategies to get them to sign up for expensive subscriptions. This can work in the short term, but it’s likely to make some customers very unhappy, and maybe make them ex-customers in the long run.

Revenue is also tricky to judge for free or ad-supported products. Again, you can boost ad revenue on a site simply by piling more ads onto a page, but that doesn’t necessarily enhance your users’ experience or happiness.

While increased revenue may indicate that people are spending more because they find your product more appealing, it can also be caused by sacrificing long term revenue for short term gains.

NPS – Net Promoter Score

The net promoter score is a measure of how many of your users would recommend your product to a friend. It’s actually a pretty good measure of customer happiness, but the problem is that it can be tricky to gauge accurately. It generally needs to be obtained through surveys and customer contact rather than simple analytics, so it suffers from relying on self-reported data and small sample sizes. Also, it tends to be skewed in favor of the type of people who answer surveys and polls, which may or may not be representative of your customer base.

While NPS may be the best indicator of customer happiness, it can be difficult to collect accurately. Unless your sample size is quite large, the variability from week to week can make it tough to see smaller changes that may warn of a coming trend.

Conversion to Paying

For products using the freemium or browsing model, this can be a useful metric, since it lets you know that people like your free offering enough to pay for it. However, it can take awhile to collect the data after you make a change to your product because you have to wait for enough new users to convert to payers.

Also, it doesn’t work well on ad-supported products or products that require payment upfront.

Most importantly, it doesn’t let you know how happy your paying customers are, since they’ve already converted.

Conversion to Paying can be useful, but it is limited to freemium or browsing models, and it tends to skew toward measuring the free part of the product rather than the paid product.

Engagement

Engagement is an interesting metric to study, since it tells me how soon and often users are electing to come back to interact with my product and how long they’re spending. This can definitely be one of the indicators of customer happiness for ecommerce, social networking, or gaming products that want to maximize the amount of time spent by each user. However, increasing engagement for a utility product like processing payroll or managing personal information might actually be an indicator that users are being forced to do more work than they’d like.

Also, engagement is one of the easiest metrics to manipulate in the short run. One time efforts, like marketing campaigns, special offers, or prize giveaways can temporarily increase engagement, but unless they’re sustainable and cost effective, they’re not going to contribute to the long term happiness of your customers.

For example, one company I worked with tried inflating their engagement numbers by offering prizes for coming back repeatedly for the first few days. While this did get people to return after their first visit, it didn’t actually have any effect on long term user happiness or adoption rates.

Engagement can be one factor in determining customer happiness, but this may not apply if you don’t have an entertainment or shopping product. Also, make sure your engagement numbers are being driven by actual customer enjoyment of your product and not by artificial tricks.

Registration

While registration can be the fastest metric to see changes in, it’s basically worthless for figuring out how happy your users are, since they’re not interacting with the product until after they’ve registered. The obvious exception is products with delayed (i.e. lazy) registration, in which case it can act like a lower barrier-to-entry version of Conversion to Paying. When you allow users to use your product for awhile before committing, an increase in registration can mean that users are finding your product compelling enough to take the next step and register.

Registration is only an indicator of happy customers when it’s lazy, and even then it’s only a piece of the puzzle, albeit an important one.

Customer Service Contacts

You’d think that decreasing the number of calls and emails to your customer service team would give you a pretty good idea of how happy your customers are. Unfortunately, this one can be manipulated aggressively by nasty tactics like making it harder to get to a representative or find a phone number. A sudden decrease in the number of support calls might mean that people are having far fewer problems. Or, it might mean that people have given up trying to contact you and gone somewhere else.

Decreased Customer Service Contacts may be caused by happier customers, but that’s not always the case.

So which is it?

While all of these metrics can be extremely important to your business, no single one can tell you if you are making your customers happy. However, looking at trends in all of them can certainly help you determine whether a recent change to your product has made your customers happier.

For example, imagine that you introduce a new element to your social networking site that reminds users of their friends’ birthdays and then helps them choose and buy the perfect gifts. Before you release the feature, you decide that it is likely to positively affect:

Engagement – every time you send a reminder of a birthday, it gives the user a reason to come back to the product and reengage.
Revenue – assuming you are taking a cut of the gift revenue, you should see an increase when people find and buy presents.
Conversion to Paying – you’re giving your users a new reason to spend money.
(Lazy) Registration – if you only allow registered users to take advantage of the new feature, this can give people a reason to register.
Retention – you’re giving users a reason to stay with you and keep coming back year after year, since people keep having birthdays.

Once the feature is released, you look at those numbers and see a statistically significant positive movement in all or most of those metrics. As long as the numbers aren’t being inflated by tricks or unsustainable methods (for example, you’re selling the gifts at a huge loss, or you’re giving people extra birthdays), you can assume that your customers are being made happy by your new feature and that the feature will have a positive impact on your business.

Of course, while you’re looking at all of your numbers and metrics and analysis, some good old fashioned customer outreach, where you actually get out and talk directly with users, can also do wonders for your understanding of WHY they’re feeling the way they’re feeling. But that’s another post.

Interested? You should follow me on Twitter.

6 Reasons Users Hate Your New Feature

2009-11-13T18:46:00.000-08:00

This post originally appeared on the Sliced Bread Design blog.

You spend months on a new feature for your existing product: researching it, designing and building it, launching it. Finally, it’s out in the world, and you sit back and wait for all those glowing comments to come in about how happy your users are that you’ve finally solved their biggest problems. Except, when the emails, forum posts, and adoption data actually come in, you realize that they hate it.

There is, sadly, no single reason why your new feature failed, but there are a number of possibilities. The failure of brand new products is its own complicated subject. To keep the scope narrow, I’m just going to concentrate on failed feature additions to current products with existing users.

Your Existing Product Needs Too Much Work

Ah, the allure of the shiny new feature! It’s so much more exciting to work on the next big thing than to fix bugs or improve the user experience of a boring old existing feature.

While working with one company, I spoke with and read forum posts written by thousands of users. I also used the product extensively myself. One of the recurring themes of the complaints I heard was that the main product was extremely buggy and slow. The problem was, fixing the bugs and the lagging was really, really hard. It involved a significant investment in infrastructure change and a serious rewrite of some very tricky code.

Instead of buckling down and making the necessary improvements, management spent a long time trying to build new features on top of the old, buggy product. Unfortunately, the response to each new, exciting feature tended to be, “Your product still crashes my computer. Why didn’t you make it stop doing that instead of adding this worthless thing that I can’t use?”

Now, you obviously don’t need to fix every last bug in your existing offering before you move on and add something new. You do, however, need to be sensitive to the actual quality of your product and the current experience of your users before adding something new. You wouldn’t build a second story on a house with a shaky foundation. Don’t tack brand new features onto a product that has an unacceptably high crash rate, severe usability problems, or that runs too slowly for a significant percentage of your users.

Before you add a new feature to a product, ask yourself, “Have I fixed the major bugs, crashes, and UX issues that are currently preventing my users from taking advantage of core features?”

Your Product Interface Is a Giant Mess

Remember the old Saturday Night Live spoof commercial that advertised, “It’s a floor wax! It’s a dessert topping!”? It’s not as funny when it’s true. Products cannot do everything, and when they try, they end up with interfaces far too complicated for the average user to navigate.

I see this happen all the time, especially with startups looking for a way to make their product appeal to more people. Instead of improving their core product and adding features that enhance that experience, they add unrelated feature after unrelated feature, often stolen directly from more successful companies with larger user bases. Their goal is to find something that makes them blow up huge, but they just end up with an overly complicated product that tries to do too many things and doesn’t do any of them well.

It’s not just startups that suffer from this. Products that have been around for many years often get bogged down with feature after feature, all of which have to be supported because some fraction of the user base still uses it. These products then become vulnerable to new challengers with more focused, easy to use interfaces and smaller feature sets.

Of course, there are times when companies have to take their products in a new direction. For example, Flickr started as a set of tools for an MMORPG called Game Neverending. The game has ended, but Flickr lives on as an entirely different business. PayPal began as a way to make PDA to PDA mobile payments, but that feature got killed years ago when they realized that web payments were a far better business model.

When you do find that killer feature that’s going to change your whole business model, commit to it and make it a serious focus rather than burying it under dozens of less popular features. Don’t try to be all things to all people, or you will end up with nothing but a giant, unusable mess.

Before adding a new feature, ask yourself, “Does this enhance my current product experience or just add to an already confusing and cluttered interface?” And, if it doesn’t fit with your current product offering but you still want to do it, ask, “Am I prepared to cut other features to make this part of my core offering and simplify my experience?”

You Didn’t Build What They Asked For

Let’s face it, sometimes your priorities are different from your users’. For whatever reason, be it a new business partnership, a need for a new revenue stream, or the desire to attract a different group of users, sometimes you’re going to build something that your current users don’t want and didn’t ask for.

This isn’t always a bad thing. For example, something that annoys your current non-paying users but attracts a whole slew of new, paying users is worth a few nasty emails to your customer service department. Just make sure that the new feature is really going to do what you think it will. It sucks to piss off your current, paying customers to build a feature that never really fulfills its initial promise. Trust me on this.

Before building a feature that potentially has more benefits to your company than your current users, ask yourself, “Am I prepared to deal with the fact that this is going to annoy some of my customers, and what is the real likelihood that I will get more out of this than I will lose?”

You Built EXACTLY What They Asked For

I know. It doesn’t seem fair. They’re angry if you don’t do what they want, and they’re angry if you do what they want.

The truth is that users will often ask you for a solution when it would really be more helpful to tell you that they have a problem. I’ve written more extensively about when you shouldn’t be listening to your users, but the upshot is that users aren’t great predictors of which brand new features will be big hits. Sometimes users will tell you that they want a toaster in their car, when what they really mean is that they don’t have time to make breakfast in the morning.

Before building a new feature that your customers are demanding, ask yourself, “What known user problems is this solving, and is this the best way to solve it for everybody?”

The Feature’s Not Finished

Now, I’m all for building the minimum viable product, getting it out in front of users, and then iterating on it to improve it, but some features just aren’t ready for prime time. By launching a half baked feature without key functionality, you’re running the risk of turning a lot of people off on the idea before they ever get a chance to really try it out.

Remember, your customers aren’t in the conference room with you when you come up with your grand vision. They don’t know where you’re going with this neat new idea. They’re judging the feature based on their first experience with it. Make sure that the first version is at least usable and hopefully that it’s far enough along that users can see the same promise that you do.

Also, good enough to ship doesn’t necessarily mean good enough to remain in your product long term. A big part of shipping early is continuing to improve the feature once it’s been out for awhile. One company I worked with had the tendency to ship early versions of features and then let them just sit there gathering dust, rather than iterating on them until they were truly high quality. What they ended up with was an enormous product that all seemed about half finished and a lot of unhappy customers who didn't believe features would ever improve past version 1.0.

Before shipping a new feature, ask, “Is this good enough that users will get why they should care about it? And, if they do care about it, am I committed to improving it?”

The Feature’s Not Finishable

At many of the companies I’ve worked with, features have tended to evolve before they even get built. What generally happens is this: you have an idea based on something you’ve heard from users; that idea gets brainstormed and grows based on internal input; UX and visual designers spec out the whole idea, often expanding on the original idea; then engineering gets involved and gives a time estimate of how long the feature will take to build; finally the whole thing gets cut back by about 80% based on the estimates.

Unfortunately, the 20% you end up implementing may not solve the original problem. That means, when you finally announce your great new feature, users who originally asked to have that particular problem solved are justifiably upset.

Before drastically cutting your new feature back, ask yourself, “Does this still solve the original problem I was trying to solve?” If it doesn’t, ask, “Can this problem be solved with a reasonable level of investment?” There’s no shame in answering no to either or both of these questions, as long as you decide not to go forward with the new feature.

The Secret to Making Everybody Love Everything You Make

I’m joking. There’s no secret. The truth is that it’s almost certainly impossible. But by asking yourself the right questions during your feature development phase, you can dramatically cut back on time spent creating things your users hate.

And never forget, when you do build something they hate, acknowledge it, apologize for it, and fix it. It will go a long way toward making your users happy again, and it might even get them to like that neat new feature you just shipped.

Interested? You should follow me on Twitter.

Is Continuous Deployment Good for Users?

2009-11-04T18:44:00.000-08:00

This post originally appeared on the Sliced Bread Design blog.

The recent release of Windows 7 got me thinking about development cycles. For those of us who suffered through the last 2+ years of Vista, Windows 7 has been a welcome relief from the lagging, bugs, and constant hassle of a failed operating system. Overall, as a customer, I’m pretty happy with Windows 7. But, at least on my part, there is still some latent anger - if Windows 7 hadn’t been quite as good as it seems to be, they would have lost me to Apple. They still might.

A big part of my unhappiness is the fact that I had to wait for more than two years before they fixed my problems. That’s a lot of crashes and frustration to forget about.

One approach that many software companies have been adopting to combat the huge lag time built into traditional software releases is something called continuous deployment. This sort of deployment means that, instead of having large, planned releases that go through a strict process and may take months or years, engineers release new code directly to users constantly, sometimes multiple times a day. A “release” could include almost anything: a whole new feature, a bug fix, or a text change on the landing page.

I worked with a software development organization that practiced continuous deployment on a very large, complicated code base, and I can definitely say, the engineers loved it. From the point of view of the employees, continuous deployment was a giant win.

But how was it for the users? The fact is, some decisions that seem like they only affect engineering (or marketing, business, PR, etc.) can actually have a huge impact on end users. So, whenever organizations make decisions, they should always be asking, “how might this affect my customers, and how can I make it work best for them?”

Is Continuous Deployment Good For Users?

As with so many decisions, the answer is yes and no. Continuous deployment has some natural pros and cons for the customer experience, but knowing about them can help you fix the cons and benefit even more from the pros.

Big Customer Wins

Fast Bug Fixes

Perhaps the biggest win for users is that bugs can get addressed immediately. Currently, even Microsoft releases patches for some of its worst security holes, but there is certainly a class of non-critical, but still important bugs that have to wait until the next major release to get addressed. That means weeks, months, or even years of your users dealing with something broken, even if the fix is simple. In continuous deployment, a fix can be shipped as soon as it's done.

Fast Things vs. Slow Things

Similar to the first point, continuous deployment lets you get everything, not just bug fixes, to users as soon as they’re ready to go. Small features, easy changes, and UI tweaks don’t have to wait for larger, unrelated features to be released to customers. After all, should a new design for the splash screen really have to wait on the implementation of a whole new payment system?

More Opportunities for Community Involvement

If you’re having a constant dialog with your customers (you are, right?), they’re probably making some pretty good suggestions about problems they’re having or ways to improve the product. A by-product of the first two benefits is that those users are going to feel even more involved in the development process when their suggestions or concerns are dealt with quickly, rather than if they have to wait months or even years for the next major release.

(Mostly) Avoidable Customer Problems

As with anything, continuous deployment can also cause some problems for users. Of course, some of these problems can exist in big, staged deployments as well, but these are a few things in particular to watch for.

Constant Change

Imagine if every day, the layout on your car changed, sometimes slightly, other times drastically. You want to drive to the store, but the steering wheel is on the other side and you’ve suddenly got an extra pedal. It would make it a lot harder to get where you were going, wouldn’t it?

Well, presumably your users are using your product to get something done, and they’ve got a certain way that they’ve learned to do it. Continuous deployment can mean that the interface for your product can change at any moment, even several times a day. If features constantly appear and disappear, it can be very disruptive to your users’ process.

There are a few things you can do to minimize the disruption. First, make sure that you’re testing your biggest changes on small cohorts of people. Iterate on a subset of your user base, rather than hitting every single user with every single change. This will limit the change that any individual sees while still giving you the benefit of constantly pushing code to customers. In fact, use this as an opportunity to do the A/B testing you should be doing anyway.

Also, and this should be obvious, try to limit your truly disruptive user interface changes so that things don’t feel like they’re in constant flux. You can still change things, but be aware of how frequently you’re making major changes and stay in contact with your users to make sure they’re not feeling dizzy.

Inconsistent UIs

When a big new release is planned, often there is a comprehensive design phase where all the new changes are mapped out and discussed. This means that any inconsistencies in the UI can be found and addressed.

In continuous deployment, different pieces of the product are getting built and shipped to customers all the time, and there is rarely a time when the entire UI is reevaluated as a single entity. This means that sometimes UI standards can tend to…oh, let’s say evolve.

This problem can be controlled by having a UX team member embedded in the development team and constantly working with the engineers to enforce standards before things ship to customers. It can also be improved by providing wireframes, visual designs, and tools like templates to developers so that the look and feel of the product doesn’t shift too dramatically over time.

Avoiding QA

I’m certainly not claiming that every single thing in a traditional release process gets a full QA pass, but I do find that continuous deployment makes it easier for code to slip out to users without any human testing at all. Any time you give engineers the ability to ship code directly into production, you’re tempting fate. Somebody’s going to say, “oh, it’s just a tiny change,” and it’s going to get out without any testing. I’m not naming any names, but you only have to have a tiny change break the entire product once before you realize that there’s no such thing as a tiny change.

Also, continuous deployment can make certain types of testing much less likely to happen. While large release cycles tend to have a code freeze and weeks or months devoted to testing, hopefully including regression, unit tests, and end to end testing, continuous deployment doesn’t necessarily have that baked into the process.

You can always add periodic end to end testing of the product to your own process, of course, and it can be quite helpful in improving code quality, especially when your engineers occasionally slip through a “tiny change.”

Communication Issues

When you’re constantly releasing new features and bug fixes, communication with your users can be a challenge. You don’t have one big release with new help docs, a big roll out plan, and an advertising campaign. Instead, stuff is coming out all the time, and users can get overwhelmed by keeping up with the changes.

Context sensitive help and inline information for each feature can help users get quickly oriented. Also, clearly marking new features as alpha or beta can let users know when a feature is still being developed so they can set their expectations accordingly.

Documentation can also easily get out of date when things are getting released constantly. Big, staged development cycles often have a built-in time for creating documents. Typically, manuals or help documentation and FAQs go through QA sometime after code freeze and before release. But since you’re not necessarily doing a big, monolithic release with continuous deployment, you can end up with this material never getting any sort of end to end editing to make sure they stay current. Make time for this. It’s good for both your customer experience and your customer support team.

Frequent Downloads and Updates

While continuous deployment is quite natural with web applications, even downloaded products can be constantly updated. However, you should always be aware of the burden you’re placing on your user base. If you’re forcing people to download a large file and go through an installation process too often, you’re going to annoy people. As a personal example, iTunes appears to have a new version every week, and I’ve started to flinch every time I open it.

There are a few things you can do to make downloads easier on your users. First, you can ask the user to allow the update to download in the background and install automatically the next time the product opens. This means that the updating happens with very little user annoyance. Also, it’s best to keep the update quick by making the downloads incremental. For example, your virus protection software probably updates its virus information daily without asking you to reinstall the entire product every time.

So, Is It Good for Users or Not?

Continuous deployment can be done in a way that’s good for both engineering teams and users, but you do need to take some precautions. By taking care when you introduce changes that your users will really notice and making sure that you make time for important processes like QA, you can get features out faster and constantly improve your product. And that is very good for users.

Interested? You should follow me on Twitter.