What Makes UX Lean - My Talk from SXSW

If you couldn’t make it to SXSW this year, there was a fantastic, all day lean startup track with talks from lots of lean startup experts. I was lucky enough to be asked to be on the Lean UX panel, along with the always awesome Janice Fraser, Ian McFarland, and Dan Martell.

I gave a short talk on what makes Lean UX Lean. Since I’m a blogger at heart, I wrote down pretty much everything I was going to say first, which means I can now publish a draft of the talk here! If you didn’t get to hear the panel, or if you did and want a quick refresher, please enjoy!

I’ve been a user experience designer for a lot of years, and I’ve worked with a lot of lean startups, which is part of the reason why I got a call last year from Manuel Rosso, the CEO of Food on the Table.

Now, Food on the Table is a very lean startup here in Austin. Because they’re a lean startup, they measure absolutely everything. And because they measure everything, Manuel knew immediately when the product developed an activation problem.

The whole project has been written up in a post for Eric’s Startup Lessons Learned blog, and I strongly recommend that you go read it if you haven’t already. It has a lot of tips about how to incorporate design into your startup that you’ll hopefully find helpful.

But today, I want to go a little deeper into what made that project a good example of Lean UX. Because, during that project, we did a lot of things that you might do in any sort of a UX project for any sort of a company.

For example, we did qualitative user research to understand why users were having a problem. We made sketches and built interactive prototypes, and we tested and iterated on them.

These are wonderful, helpful things to do, but they’re not unique to Lean User Experience Design. They’re part of User Centered Design, which I’m a huge fan of, and I’ve done all of those things in waterfall projects at giant companies that were anything but lean.

So, what are a few things that made this a lean ux project and not just a regular old redesign?

Integrating Quantitative Research

I think the first hallmark of Lean UX is using quantitative metrics to both drive and validate design changes. What does that mean? Well, it means that the reason we were working on the first time user experience was because a specific metric, activation, wasn’t as high as the team wanted it to be.

Quantitative metrics didn’t tell us exactly why we had a problem - we needed to do our qualitative research to understand that - but it did tell us what our most immediate problem was, which helped us to understand where we should start improving our user experience.

In that way, the metric drove the product decision.

Quantitative metrics also meant that we knew, at the end of the project, we’d be validating our work with an a/b test against the original design. That quantitative validation of design really helps improve the design process over the long run, because we can see what sorts of changes have the biggest positive impact on our end user experience. That lets us improve the ROI on future design projects.

When Is a Design Done?

I was talking with a designer about Lean UX. I was explaining that one of the hallmarks of Lean UX is to get a good, but not complete version of a product or feature designed and built and then iterate on it later. She thought this sounded like an interesting approach, but then she asked, “When do you know you’re done?”

Figuring out when you’re “done” is tricky for any design or redesign project, unless you’re a consulting agency, of course, in which case the answer is, “when the client runs out of money.” But I realized that, in Lean UX, figuring out when you’re done is actually incredibly easy.

You’re done when your metrics tell you you’re done.

Let me explain. No product is ever actually “done.” There is always something you could do to improve it. However, projects can certainly be done. The trick is that you have to choose your projects correctly.

What’s the correct way to choose a Lean UX project? Every Lean UX project should be chosen based on a metric.

This may piss off a lot of designers who want to make wonderful, exciting, super cool designs just for the sake of design or user happiness, but when it comes down to it, unless you’re independently wealthy, every design change you make should move a number that is important to your business.

Now, it is a lucky break for those of us who care deeply about our users that improving the overall user experience of the product frequently improves some number that the business people care about. But not every single thing you can do to make a user happy has the same ROI for the business. And not every improvement makes the right people happy at the right time.

That’s why the UX projects you choose should be based on metrics.

Let me give you an example. Whoever it is at your startup who is in charge of running the business should have a pretty good idea of what your various metrics have to be in order for you to all retire and buy yachts. For example, your Activation number may have to be 20% and your Retention may have to be 70%. (Please note, I made these numbers up. Your metrics may vary.)

They pick these numbers because they know that having, for example, a 99% retention rate and a 1% activation rate may lead to retaining 3 incredibly happy users forever, which is suboptimal from a business perspective.

So, if your activation number is at 10%, your business folks may come to you and say, “we need to turn more of our acquired traffic into regular users because we have identified this as the most important problem to solve at this moment.” You respond, “Great! How many more do you need?” They explain that you need to get activation from 10% to 20%.

You will notice that the metrics are not driving your design decisions. Nor are they driving your feature requirements or any other product changes. They are simply telling you what your biggest business problem currently is.

Now, it’s up to you as a designer or product owner to figure out what is keeping the activation number low and then come up with some ideas of how to fix it. You do this with what I like to call “research and design” or alternately “that thing you are paid to do.”

You may have dozens of wonderful ideas for how to fix the problem, and you may love and believe in all of them. You may not, however, actually execute every single one of them.

This is where the Lean part comes in.

Ideally, you will design and execute as many of the fixes as necessary in order to move the number to where you want it to be. Maybe you’re awesome (or awesomely lucky), and you move that activation number on the first try with a very small bug fix.

Does that mean you never get to implement the super sweet, but somewhat complicated, feature that you know will make users incredibly happy and improve activation even more? No! Unfortunately, you may not get to implement it just yet.

You see, once you got your activation number to where it needed to be, it stopped being the most important problem to solve. Now, maybe you need to work on getting retention higher or improving revenue or referral.

On the flip side, maybe you redesign the first time user flow and improve activation, but not by enough. That means you should continue working on it. Figure out why your changes didn’t have as big of an impact as you thought they would, and then try some new things.

You’re not “finished” until your metrics are where you want them to be.

Why is this important? Startups have a ridiculous number of things to do, and they typically have limited resources. It can be incredibly difficult to prioritize when to keep working on a feature or an area of the product, and when to move on.

By setting the goals ahead of time based on metrics that are critical to the business, it becomes much easier to know when you’re “done,” and when you should keep optimizing or redesigning.

Like the post? Follow me on Twitter!

Like Lean UX but hate reading? I'll be on the UX panel at the Lean Startup track at SXSW. You should come see it and then say hi to me afterward.

Qual vs. Quant: When to Listen and When to Measure

I have written about qualitative vs quantitative research before, but I still get a lot of questions about it. To answer some of those questions, I want to do a bit of a deeper dive here and give a few examples to help startups answer the key question.

To be clear, that key question is “when should I use qualitative research, and when should I use quantitative research for the best results?” Another way of looking at this is, “when should I be listening to users, and when should I just be shipping code and looking at the metrics?”

The real answer is that you should do both constantly, but there are times when one is significantly more helpful than the other.

I will continue to repeat my cardinal rule: Quantitative research tells you WHAT your problem is. Qualitative research tells you WHY you have that problem.

Now, let’s look at what that actually means to you when you’re making product decisions.

A One Variable Change

When you’re trying to decide between qualitative and quantitative testing for any given change or feature, you need to figure out how many variables you’re changing.

Here’s a simple example: You have a product page with a buy button on it. You want to see if the buy button performs better if it’s higher on the page without really changing anything else. Which do you do? Qualitative of quantitative?

That’s right, I said this one was simple. There’s absolutely no reason to qualitatively test this before shipping it. Just get this in front of users and measure their actual rate of clicking on the button.

The fact is, with a change this small, users in a testing session or discussion aren’t going to be able to give you any decent information. Hell, they probably won’t even notice the difference. Qualitative feedback here is not going to be worth the time and money it takes to set up interviews, talk to users, and analyze the data.

More importantly, since you are only changing one variable, if user behavior changes, you already have a really good idea WHY it changed. It changed because the CTA button was in a better place. There’s nothing mysterious going on here.

There’s an exception! In a few cases, you are going to ship a change that seems incredibly simple, and you are going to see an enormous and surprising change in your metrics (either positive or negative). If this happens, it’s worth running some observational tests with something like UserTesting.com where you just watch people using the feature both before and after the change to see if anything weird is happening. For example, you may have introduced a bug, or you may have made it so that the button is no longer visible to certain users.

What Does Your User Know?

You and your customer have very different sets of information. This shouldn’t come as a surprise to you, since by now it should be obvious that you are not your user.

Sometimes it can be very hard to distinguish what you know about your product or industry from what your user knows. But it’s important! Making assumptions about your user’s knowledge can lead to products that are impossible for normal people to use.

You Know...

The Details of How Your Product Works

You know all the technical and implementation details of your product. Your user doesn’t even know what those things mean.

One company I worked with had a feature that allowed users to mark off all the items they owned from a list. The engineers knew that, when a user made a selection, that selection was sent to the server in an AJAX request, so the account was kept constantly up to date.

The users didn’t know this. During testing, several users went through, marked off all their selections, and then searched in vain for a Save button. They assumed that they would have to save their work, since they had no idea about the AJAX request that was happening in the background.

The solutions to this were either to allow the users to explicitly save with a button or to give them a tiny amount of feedback while the item was being saved via a very brief wait spinner.

Once we implemented the latter solution, users immediately understood that each click was actually saving the item automatically, and they no longer looked for that Save button.

The take away: Don’t make any assumptions that your users understand what you’re doing for them unless you’re explicitly telling them in some way.

Every Single Feature and Its Purpose

You designed and built every feature in your product, so you know exactly what each of them does and where to find it in your product. Your user knows only what she finds during her time using your product.

One company I worked with had a very useful feature. When I interviewed users about new features they’d like to see, many of them requested the feature, even though it was already in the product! They simply had no idea that it even existed.

The take away: It’s not enough to create fabulous new features. You have to make sure that your features are discoverable by normal users.

Specialized Knowledge About the Product Space

Sometimes we build products to help people do hard things more easily.

Tax preparation software is a fantastic example of this. It is safe to say that most of us who use tax preparation software do not know nearly as much about tax preparation as the people who are building the software. At least, I hope they know more than I do!

The problem arises when we lose touch with exactly which parts of the space the users understand well. We can start to use jargon or terminology that makes perfect sense to us because we hear it all the time. We can design processes that seem completely reasonable if the user already knows the goal.

Unfortunately, this creates complicated, confusing products that assume a much higher level of understanding than the user has.

The take away: When you’re trying to help users accomplish a complicated goal, you need to work even harder to keep the interface simple.

Of course, your customer knows some stuff you don’t know...

How She’s Used to Doing Things

If you’re creating a product that is meant to help a users do something they already do (again, tax preparation or any business software), your goal is to create a generic experience that will satisfy as many people in your core demographic as possible.

But remember, each of those users already does things slightly differently. For example, if you’re helping people who sell things on eBay, you have to understand that each seller already has a process that she follows - from pricing to listing to shipping to dealing with customers.

Asking your users to change too many of their behaviors in order to use your product creates a huge barrier to acceptance.

If you only talk to a few potential customers (or, even worse, none at all), you run the risk of creating something that isn’t broadly usable by all sorts of different users.

The take away: Understanding the variations in user behavior will help you deliver something that is usable by a larger segment of your user base.

Two Stupid Reasons for Complicated Products

I frequently get asked by startups to simplify products. In general, companies are fantastic at coming up with great feature ideas, but they tend to find it harder to either kill underperforming features or properly integrate new ideas that got tacked on as an experiment or pivot.

Because I get called in when a company already has a product that new users find confusing, I see a lot of the same mistakes repeated. I also hear the same excuses for those mistakes.

Often, when I’m looking at a new product, I’ll find very similar features in different parts of the interface.

For example, one social product had three completely different ways of searching for friends the user might know. Now, I don’t mean that there were different criteria you could use - like email address or interests or user name. That would have been fine.

I mean there were three completely different places the user could go in the product to find three completely different features that were meant to help people search for their friends. There was huge functionality overlap among the three features, but they were all slightly different.

There’s a Reason For That

Of course, I mentioned that it seemed odd and confusing to have three different places to go to do essentially the same thing. And the product owner patiently explained to me that there was a reason for that.

The product owner then launched into a detailed description of how the first one had been built by the team as an experiment. A few months later, since the experiment went well, but was a little slow, the team migrated to a different technology for search, and built a second version of the feature alongside the first one to see if it could be faster.

Since the product owner hadn’t given them any requirements for the new version other than “go faster,” and the new technology made some of the old functionality tricky, the second version didn’t have exactly the same capabilities as the first. So, the team decided it wasn’t an adequate replacement for the old version. They released it anyway, since they’d already built it, and it was faster.

The third version of the feature had been built as part of a larger feature, but the rest of that larger feature had been killed, and only the search part remained. It had some neat new functionality that users liked, but the team felt it didn’t really replace either of the other two versions.

Moreover, the product owner explained to me that different types of users liked the various different versions of friend search, so he was hesitant to kill any of them, because whichever one he killed would upset somebody.

So, he was stuck supporting three different variations of the same feature, and new users were overwhelmed by choice for where they were supposed to go to find their friends.

And here’s the thing: users don’t actually want this sort of choice. This sort of choice is confusing. It makes navigating the product cumbersome, and it’s unpleasantly surprising to constantly find new, slightly different ways to do the same thing.

There’s a Big Difference

This isn’t the only way that the problem manifests. Sometimes when I point out similar features in different places, the product owner reassures me that “there is a big difference” between the features.

This happened with a client when I pointed out that there were two different types of quizzes in one section of the product, but they had different names and placements. I asked why they couldn’t be combined into one section, so that the clutter on the page would be reduced and the user could always know where to go to take quizzes.

He assured me that there was a big difference between the two types of quizzes. When I asked him to elaborate, he went into detail about how the types of quizzes were different on the back end, and how one was often used as a business development tool while the other was user generated.

In other words, both of the “big differences” were things that were only different to the company. They were the sorts of differences that a user would never notice. All a user would notice would be that the quizzes were sometimes in one place and sometimes in another, which would be confusing and frustrating if she was looking for that feature.

How to Avoid This

Take a hard look at your product. Are there any similar features? You’ll be surprised at how often the answer is yes. If there are, ask yourself what your reasons were for building the different variations.

If your reasons for building different versions of the same feature are based entirely on technology or business development (in other words, things that only matter to YOU and not your users), you’ve got a problem.

The most customer friendly way of dealing with it is to try to come up with a superset of the best functionality from all the versions and improve one of the versions until it satisfies as many user stories as possible.

But even if you don’t have the time or resources to consolidate them into one great version of the product, just killing the duplicated features will ultimately simplify your product and make it more useful and understandable for all of your customers.

Like the post? Follow me on twitter!

Lean UX - A Case Study

For those very, very few (ok, none) of you who read my blog but don't read Eric Ries's blog, Startup Lessons Learned, I have some exciting news for you. But first, why the hell aren't you reading Eric's blog? You really should. It's great.

I've a written a guest post that now appears on the Startup Lessons Learned blog. It's a case study of a UX project I did with the lean startup Food on the Table.

If you're wondering whether design works well with lean startups, I answer that question in the post. Spoiler alert: The answer is 'yes'.

Testing Whether Your Users Will Buy

As you all know by now, I’m a huge proponent of qualitative user testing. I think it’s wonderful for learning about your users and product.

But it’s not a panacea. The fact is, there are many questions that qualitative testing either doesn’t answer well or for which qualitative testing isn’t the most efficient solution. I cover some of them in my A Faster Horse post.

The trick is knowing which questions you can answer by listening to your users and which questions need a different methodology.

Unfortunately, one of the most important questions people want answered isn’t particularly well suited to qualitative testing.

If I Build It, Will They Buy?

I get asked a lot whether users will buy a product if the team adds a specific feature. Sadly, I always have to answer, “I have no idea.”

The problem is, people are terrible at predicting their future behavior. Imagine if somebody were to ask you if you were going to buy car this year. Now, for some of you, that answer is almost certainly yes, and for others it’s almost certainly no. But for most of us, the answer is, “it depends on the circumstances.”

For some, the addition of a new feature - say, an electric motor - might be the deciding factor, but for many the decision to buy a car depends on a lot of factors, most of which aren’t controlled by the car manufacturer: the economy, whether a current car breaks down, whether we win the lottery or land that job at Goldman Sachs, etc. There are other factors that are under the control of the car company but aren't related to the feature: maybe the new electric car is not the right size or isn't in our price range or isn't our style.

This is true for smaller purchases too. Can you absolutely answer whether or not you will eat a cookie this week? Unless you never eat cookies (I'm told these people exist), it’s probably not something you give a lot of thought to. If somebody were to ask you in a user study, your answer would be no better than a guess and would possibly even be biased by the simple act of having the question asked.

Admit it, a cookie sounds kind of good right now, doesn’t it?

There are other reasons why qualitative testing isn't great at predicting future behavior, but I'm not going to bore you with them. The fact is, it's just not the most efficient or effective method for answering the question, "If I build it, will they come?"

What Questions Can Qualitative Research Answer Well?

Qualitative research is phenomenal for telling you whether your users can do x. It tells you whether the feature makes sense to them and whether they can complete a given task successfully.