Design Hacks - The Talk

I write a lot about user research - generally tips and tricks for people who don't have much experience with it. The reason for this should be obvious. Understanding your user, by any means necessary, is always the first step in creating a compelling product.

Seriously, you can't build a product without understanding the problem you're solving and the people for whom you're solving it. Various forms of research are the best way of understanding people who aren't you. It's really as simple as that.

But I've also seen another common problem. A whole lot of folks have learned how to go out and listen to their customers and understand problems, but they still make bad, hard to use products that don't really solve a problem. It turns out that, while learning your users problems is a necessary first step, it's not the only step.

You also have to be able to create something that people understand and want to use, and you don't do that by simply trying random ideas until one of them sticks unless you have an infinite number of monkeys and typewriters. If you are constrained with respect to monkeys, typewriters, or VC funding, you might want a little guidance on what to do once you understand the problem.

Getting your design closer to right on the first, second, or third try will speed things up considerably. It's hard to learn anything from a badly designed, unusable product other than the fact that people hate badly designed, unusable products. And believe me, that lesson has been learned. Kind of a lot.

That's why I'll be giving a talk at Lean Startup Circle on Wednesday, March 20th. It starts at 6:30.  There will be other interesting speakers, as well. You can sign up here: http://sanfrancisco.leanstartupcircle.com/events/102633722/

I'll be talking about Design Hacks. These will include a few general tips on producing a good design. It will also include some (free) resources for getting good design ideas. Time permitting, it will include an example or two of how to think about new features for your product in a way that makes them easier to design.

This talk is NOT for design experts. Sorry, you'll be bored out of your minds.

This talk is perfect for founders and engineers who don't have experience with turning what they know about their users into useful designs. And, as always, I'll be hanging around afterward to answer specific questions about your products.

Once again, to see the talk, sign up here: You can sign up here: http://sanfrancisco.leanstartupcircle.com/events/102633722/

If you want to hear about future events where I'll be speaking, you can follow me on Twitter.
If you like to read about things like Design Hacks, the book is available for pre-order.

Don't Make Your Users Feel Like Idiots

I’m a smart person. I’ve been using the Internet since the early 1990s. I know how to program. I only feel the need to point this out, because I’m about to share with you a story in which I come across as a complete, blithering idiot, and I’m feeling a little defensive about it.

I got an email from an event that I won’t name, but I’m guessing a few of you are getting emails of your own. If you didn’t make the same mistake, then bask in the glory of being better at computers than I am. If you did make the same mistake, welcome to the club. You’re not alone.

The email I received was several paragraphs long and told me all the places and times where I could pick up my badge for the event. It also said that they were introducing something new this year called a QuickCode. The email instructed me to bring my photo id and my QuickCode to pick up my badge.

Then it had the following line:
Laura Klein’s QuickCode:

That’s it. After that, it went on to give me more badge-related information.  “Aha, I thought. The automated system has failed to print my QuickCode.”

I immediately wrote back and said that I didn’t get my QuickCode. To the credit of the organization, I was immediately written back to by a very polite person who explained that the QuickCode was an image and even gave me instructions on how to turn on images in my email, in case I didn’t know how.

I was, as you might imagine, embarrassed. I mean, of course I know how to show images in an email. I just want to make that clear, because I’m coming off as enough of an idiot without you thinking I can’t use Gmail.

What I didn’t know was that the QuickCode was an image. Because I’ve never seen a QuickCode. Because a QuickCode wasn’t a thing to me until an hour ago. Because a QuickCode is just a name that somebody made up for a bar code that they’re using to help with their badging system.

Obviously the people writing the email knew what a QuickCode was, so it wasn’t at all surprising to them that you’d have to turn on images to see one. For those of us (ok, me) who had never heard of a QuickCode, this wasn’t immediately obvious. A QuickCode could just as easily have been a string of numbers and letters that could have been printed in the email. Of course, when I went back and re-read the email, the first paragraph did mention “scanning” the quick code, so I might have figured out what it was, but there were a lot of paragraphs in the email that I quickly skimmed. This is not unusual user behavior.

The interesting thing is that they could have avoided my acting like an idiot and subsequently having to deal with my support email by just including the phrase, “If you don’t see your QuickCode, try turning on images in your email.” They could have made that the alt text for the image so only people who didn’t have images turned on would see it.

Why am I telling you all this? I’m telling you this because we make assumptions of this sort in our interfaces every day. We assume people know that a QuickCode is an image, even though they’ve never heard of a QuickCode. We assume people know what our products do, even though they’ve never heard of our product. We assume people know where to go within our products to find the things they’re looking for, even though they weren’t in the meeting where we determined our product structure.

We are almost always wrong.

The moral of this story is not (just) that your users are going to do stupid things sometimes. It’s not even that they’re probably only going to skim our very long emails. The moral is that we constantly need to be asking ourselves what we really expect a user to understand about our product, and we need to have ways to preemptively help them in places where we’re presenting new concepts or unfamiliar terminology.

Users don’t know our slang. They don’t know our jargon. They don’t know our product. If we want them to use our products successfully, we need to teach them what they need to know without making them feel like idiots.

Combining Qualitative & Quantitative Research

Designers are infallible. At least, that’s the only conclusion that I can draw, considering how many of them flat out refuse to do any sort of qualitative or quantitative testing on their product. I have spoken with designers, founders, and product owners at companies of all sizes, and it always amazes me how many of them are so convinced that their product vision is perfect that they will come up with the most inventive excuses for not doing any sort of customer research or testing. 

Before I share some of these excuses with you, let’s take a look at the types of research I would expect these folks to be doing on their products and ideas.

Quantitative Reserach

When I say quantitative research in this context, I’m talking about a/b testing, product analytics, and metrics - things that tell you what is happening when users interact with your product. These are methods of finding out, after you’ve shipped a new product, feature, or change, exactly what your users are doing with it. 

Are people using the new feature once and then abandoning it? Are they not finding the new feature at all? Are they spending more money than users who don’t see the change? Are they more likely to sign up for a subscription or buy a premium offering? These are the types of questions that quantitative research can answer. 

For a simple example, if you were to design a new version of a landing page, you might run an a/b test of the new design against the old design. Half of your users would see each version, and you’d measure to see which design got you more registered users or qualified leads or sales or any other metric you cared about.

Qualitative Research

By qualitative testing, I mean the act of watching people use your product and talking to them about it. I don’t mean asking users what you should build. I just mean observing and listening to your users in order to better understand their behavior. 

You might do qualitative testing before building a new feature or product so that you can learn more about your potential users’ behaviors. What is their current workflow? What is their level of technical expertise? What products are they already using? You might also do it once your product is in the hands of users in order to understand why they’re behaving the way they are. Do they find something confusing? Are they getting lost or stuck at a particular point? Does the product not solve a critical problem for them? 

For example, you might find a few of your regular users and watch them with your product in order to understand why they’re spending less money since you shipped a new feature. You might give them a task in order to see if they could complete it or if they got stuck. You might interview them about their usage of the new feature in order to understand how they feel about it. 

Excuses, Excuses

While it may seem perfectly reasonable to want to know what your users are really doing and why they are doing it, a huge number of designers seem really resistant to performing these simple types of research or even listening to the results. I don’t know why they refuse to pay any attention to their users, but I can share some of the terrible excuses they’ve given me. 

A/B Testing is Only Good for Small Changes

I hear this one a lot. There seems to be a misconception that a/b testing is only useful for things like button color and that by doing a/b testing you’re only ever going to get small changes. The argument goes something like, “Well, we can only test very small things and so we will test our way to a local maximum without ever being able to really make an important change to our user experience.”
This is simply untrue.

You can a/b test anything. You can show two groups of users entirely different experiences and measure how each group behaves. You can hide whole features from users. You can change the entire checkout flow for half the people buying things from you. You can test a brand new registration or onboarding system. And, of course, you can test different button colors, if that is something that you are inclined to do.

The important thing to remember here is that a/b testing is a tool. Itʼs agnostic about what youʼre testing. If youʼre just testing small changes, youʼll only get small changes in your product. If, on the other hand, you test big things - major navigation changes, new features, new purchasing flows, completely different products - then youʼll get big changes. And, more importantly, you’ll know how they affected your users. 

Quantitative Testing Leads to a Confused Mess of an Interface

This is one of those arguments that has a grain of truth in it. It goes something like, “If we always just take the thing that converts best, we will end up with a confusing mess of an interface.”
Anybody who has looked at Amazonʼs product pages knows the sort of thing that a/b testing can lead to. They have a huge amount of information on each screen, and none of it seems particularly attractive. On the other hand, they rake in money.

Itʼs true that when youʼre doing lots of a/b testing on various features, you can wind up with a weird mishmash of things in your product that donʼt necessarily create a harmonious overall design. You can even wind up with features that, while they improve conversion on their own, end up hurting conversion when they’re combined. 

As an example, letʼs say youʼre testing a product detail page. You decide to run several a/b tests simultaneously for the following new features:

• 
customer photos

• comments
• ratings

• extended product details

• shipping information

• sale price

• return info

Now, letʼs imagine that each one of those items, in its own a/b test, increases conversion by some small, but statistically significant margin. That means you keep all of them. Now youʼve got a product detail page with a huge number of things on it. You might, rightly, worry that the page is becoming so overwhelming that youʼll start to lose conversions.

Again, this is not the fault of a/b testing – or in this case, a/b/c/d/e testing. This is the fault of a bad test. You see, itʼs not enough that you run an a/b test. You have to run a good a/b test. In this case, just because the addition of a particular feature to your product page improved conversions doesn’t mean that adding a dozen new features to your product page will increase your conversion. 

In this instance, you might be better off running several a/b tests serially. In other words, add a feature, test it, and then add another and test. This way you’ll be sure that every additional feature is actually improving your conversion. Alternatively, you could test a few different versions of the page with different combinations of features to see which converts best. 

A/B Testing Takes Away the Need For Design

For some reason, people think that a/b testing means that you just randomly test whatever crazy shit pops into your head. They envision a world where engineers algorithmically generate feature ideas, build them all, and then just measure which one does best.

This is just absolute nonsense.

A/B testing only specifies that you need to test new designs against each other or against some sort of a control. It says absolutely zero about how you come up with those design ideas.

The best way to come up with great products is to go out and observe users and find problems that you can solve and then use good design processes to solve them. When you start doing testing, youʼre not changing anything at all about that process. Youʼre just making sure that you get metrics on how those changes affect real user behavior.

Letʼs imagine that youʼre building an online site to buy pet food. You come up with a fabulous landing page idea that involves some sort of talking sock puppet. You decide to create this puppet character based on your intimate knowledge of your user base and your sincere belief that what they are missing in their lives is a talking sock puppet. Itʼs a reasonable assumption.

Instead of just launching your wholly re-imagined landing page, complete with talking sock puppet video, you create your landing page and show it to only half of your users, while the rest of your users are stuck with their sad, sock puppet-less version of the site. Then you look to see which group of users bought more pet food. At no point did the testing process have anything to do with the design process. 

Itʼs really that simple. Nothing about a/b testing determines what youʼre going to test. A/B testing has literally nothing to do with the initial design and research process. 

Whatever youʼre testing, you still need somebody who is good at creating the experiences youʼre planning on testing against one another. A/B testing two crappy experiences does, in fact, lead to a final crappy experience. After all, if youʼre looking at two options that both suck, a/b testing is only going to determine which one sucks less.

Design is still incredibly important. It just becomes possible to measure designʼs impact with a/b testing.

There’s No Time to Usability Test

When I ask people whether they’ve done usability testing on prototypes of major changes to their products, I frequently get told that there simply wasn’t time. It often sounds something like, “Oh, we had this really tight deadline, and we couldn’t fit in a round of usability testing on a prototype because that would have added at least a week, and then we wouldn’t have been able to ship on time.” 

The fact is you don't have time NOT to usability test. As your development cycle gets farther along, major changes get more and more expensive to implement. If you're in an agile development environment, you can make updates based on user feedback quickly after a release, but in a more traditional environment, it can be a long time before you can correct a big mistake, and that spells slippage, higher costs, and angry development teams. Even in agile environments, it’s still faster to fix things before you write a lot of code than after you have pissed off customers who are wondering why you ruined an important feature that they were using. 

I know you have a deadline. I know it's probably slipped already. It's still a bad excuse for not getting customer feedback during the development process. You're just costing yourself time later. I’ve never known good usability testing to do anything other than save time in the long run on big projects.

Qualitative Research Doesn’t Work Because Users Don’t Know What They Want

This is possibly the most common argument against qualitative research, and it’s particularly frustrating, because part of the statement is quite true. Users aren’t particularly good at coming up with brilliant new ideas for what to build next. Fortunately, that doesn’t matter. 

Let’s make this perfectly clear. Qualitative research is NOT about asking people what they want. At no point do we say, “What should we build next?” and then relinquish control over our interfaces to our users. People who do this are NOT doing qualitative research. 

Qualitative research isn’t about asking people what they want and giving it to them. Qualitative research is about understanding the needs and behaviors of your users. It’s about really knowing what problem you’re solving and for whom.

Once you understand what your users are like and what they want to do with your product, it’s your job to come up with ways to make that happen. That’s the design part. That’s the part that’s your job.

It’s My Vision - Users Will Screw it Up

This can also be called the "But Steve Jobs doesn't listen to users..." excuse. 

The fact is, understanding what your users like and don't like about your product doesn't mean giving up on your vision. You don't need to make every single change suggested by your users. You don't need to sacrifice a coherent design to the whims of a user test. You don’t even need to keep a design just because it converts better in an a/b test. 

What you do need to do is understand exactly what is happening with your product and why. And you can only do that by gathering data. The data can help you make better decisions, but they don’t force you to do anything at all.

Design Isn’t About Metrics

This is the argument that infuriates me the most. I have literally heard people say things like, “Design can’t be measured, because design isnʼt about the bottom line. Itʼs all about the customer experience.”

Nope.

Wouldnʼt it be a better experience if everything on Amazon were free? Be honest! It totally would. 

Unfortunately, it would be a somewhat traumatic experience for the Amazon stockholders. You see, we donʼt always optimize for the absolute best user experience. We make tradeoffs. We aim for a fabulous user experience that also delivers fabulous profits.

While itʼs true that we donʼt want to just turn our user experience design over to short term revenue metrics, we can vastly improve user experience by seeing which improvements and features are most beneficial for both users and the company.

Design is not art. If you think that thereʼs some ideal design that is completely divorced from the effect itʼs having on your companyʼs bottom line, then youʼre an artist, not a designer. Design has a purpose and a goal, and those things can be measured.

So, What’s the Right Answer?

If you’re all out of excuses, there is something that you can do to vastly improve your product. You can use quantitative and qualitative data together. 

Use quantitative metrics to understand exactly what your users are doing. What features do they use? How much do they spend? Does changing something big have a big impact on real user behavior?

Use qualitative research to understand why your users do what they do. What problems are they trying to solve? Why are they dropping out of a particular task flow when they do? Why do they leave and never come back.

Let’s look at an example of how you might do this effectively. First, imagine that you have a payment flow in your product. Now, imagine that 80% of your users are not getting through that payment flow once they’ve started. Of course, you wouldn’t know that at all if you weren’t looking at your metrics. You also wouldn’t know that the majority of people are dropping out in one particular place in the flow.

Next, imagine that you want to know why so many people are getting stuck at that one place. You could do a very simple observational test where you watch four or five real users going through the payment flow in order to see if they get stuck in the same place. When they do, you could discuss with them what stopped them there. Did they need more information? Was there a bug? Did they get confused?

Once you have a hypothesis about what’s not working for people, you can make a change to your payment flow that you think will fix the problem. Neither qualitative nor quantitative research tells you what this change is. They just alert you that there’s a problem and give you some ideas about why that problem is happening. 

After you’ve made your change, you can run an a/b test of the old version against the new version. This will let you know whether your change was effective or if the problem still exists. This creates a fantastic feedback loop of information so that you can confirm whether your design instincts are functioning correctly and you’re actually solving user problems. 

As you can hopefully see from the example, nobody is saying that you have to be a slave to your data. Nobody is saying that you have to turn your product vision or development process over to an algorithm or a focus group. Nobody is saying that you can only make small changes. All I’m saying is that using quantitative and qualitative research correctly gives you insight into what your users are doing and why they are doing it. And that will be good for your designs, your product, and your business.


Like the post? 

Follow me on Twitter!

Pre-order the book! 

Make Meetings Less Awful

Meetings are the worst. I mean, my God, they suck. The vast majority of meetings are simply awful. 

But they don’t have to be!

If you’ve ever been in a meeting where you felt like your soul was being sucked out of your body through your eyes, I have a few tips that will make future meetings more tolerable. If you implement them correctly, they might even make some of your meetings useful! Imagine that. 

Write It Down Ahead of Time

Agendas. You should have one. Well, this seems painfully obvious, doesn’t it? But seriously. How many meetings do you attend where there isn’t a single person who knows exactly what you’ll be talking about in the meeting beforehand? 

Here’s a simple solution for making meetings wildly more productive. The person who is in charge of the meeting needs to make an agenda and send it out to all the attendees before the meeting. A full day is great, especially if there are things that people might want to research in preparation for the meeting. Even a few hours is helpful. It’s best if the person in charge reaches out to attendees early to see if they have anything they’d like to see on the agenda. 

The corollary to this is that the meeting attendees must actually read the agenda, understand what will be discussed, and come to the meeting prepared to discuss and make a decision on any of the agenda items they care about. 

And, of course, if they don’t care about any of the agenda items, they probably shouldn’t attend the meeting. 

Another, slightly more spontaneous, method is the box on the whiteboard. We used to do this in engineering meetings at IMVU. Before the weekly eng meeting started, people could add topics they wanted to discuss to a list on the whiteboard. Once the meeting started, someone drew a box around the list. Nothing could be added to the list once we started, and nothing was discussed that wasn’t in the box. As a bonus, it encouraged people to get to the meeting early if they had a topic to discuss. 

Everything Has a Next Step 

Meetings are not open ended discussion forums. They’re not group therapy sessions. Meetings are for making decisions. Every single thing you discuss in a meeting should have an decision and a deliverable. 

Here’s an example. Once, I was in a meeting to talk about a change somebody wanted to make to a product’s design. We sat together for half an hour discussing the types of research she could do to figure out whether the design would work or whether it was small enough just to ship. At the end of about 30 minutes, she announced, “Well, I don’t think we’re going to decide this now.” To which I responded, “Why the hell not?”

Stop having discussions just to have discussions. Refusing to make a decision in this meeting just ensures that you need to have another meeting later, and nobody wants that. Make sure that all agenda items at meetings have outcomes. Sometimes the outcome will be, “Susan is going to go off and investigate these three questions and report back so that we can make a more informed decision.” Sometimes the outcome will be, “Laura is in charge of building a prototype and will pull in whomever she needs to help.” Sometimes the outcome will be, “We’re shipping this damned thing as soon as we leave the room.” I kind of wish that were always the outcome.

The outcome will never be, “Well, we need to think more about this.” The problem with this statement is that it’s too vague. There is nothing actionable about this. Nobody is assigned to do anything, so nothing will really get done, and the next time the point comes up, you’ll have to have the whole conversation over again. Everything from a meeting needs a specific next step and somebody who is assigned to take it. 

Fewer Attendees

Meetings become far less productive after about four people, so whenever possible, keep meetings as small as you can. Obviously you sometimes need to have more folks, but really ask yourself whether everybody needs to be in the meeting, or if somebody would do just as well with a quick report after the fact.

If there are people who routinely aren’t contributing to the meeting in any way - no agenda items, no adding to the discussion, no making decisions, no deliverables after the fact - then they are great candidates for not getting an invitation next time. Presumably you’re paying these people, and I have to imagine there is something more productive they could be doing than sitting in a meeting checking their email.

Every Meeting Has a Leader

Someone has to be in charge of the meeting. Always. 

The person in charge of the meeting has a lot of responsibilities. The leader must make the agenda, keep everybody on track, mediate disputes, ensure that everybody who has a contribution gets to make that contribution, make sure that all the deliverables and next steps are being captured, and follow up on the things that come out of the meetings. 

I was in a meeting once that was led by a particularly ineffective PM. We were discussing what the priorities would be for her product (don’t even get me started on why engineers and designers were discussing this when it was so clearly her job). We were each giving our opinions about what should be done first, and the discussion began to get heated. 

Instead of stepping in and guiding the discussion or just deciding what order we’d build things in, the PM sat back and let everybody scream at each other. The meeting ended with someone in tears (unsurprisingly, this person wasn’t me) and no decision made about prioritization. 

Unless somebody is in charge, meetings just meander and go on for three times as long as they need to with nobody who is willing or able to say, “Right. We’re done here. Let’s go do something productive.” Having someone whose job it is to end discussion and assign tasks makes things go much more smoothly and quickly. 

Besides, if we actually expected some work from the people who call all those meetings, maybe they’d call fewer damned meetings. 

No Broadcast Meetings

I’m going to assume that everybody working for your company is literate. If this is true, please stop having meetings where you read things to them. You’re not in kindergarten. This is not story time. 

I have been to too many meetings where a PM or CEO or somebody else who should know better shows a slide deck and then proceeds to read all the slides to the audience for an hour. 

Here’s an idea: send the deck out the day before. Tell people to read it for themselves and come up with questions. At the meeting, spend no more than five minutes summarizing the most important things about the slide deck (“We made more money this month than last month! Yay!”), then take questions from the audience about the rest of the deck. 

If you are concerned that people will miss critical information because they are failing to read important emails, that’s really something that you need to address separately. I’ve found that reducing meeting times by a few hours a week gives people far more time to read their email or to do something actually productive. 

More Discussions, More Working Sessions, Fewer Meetings

You know what I like more than meetings (besides everything)? I like discussions. Discussions are things that happen between two or three people who are all interested in and informed about a particular topic. They tend to happen in hallways and they often help disseminate important information to the people who need it. 

I also like working sessions, in which a few people all work together on something like a design or code. Working sessions generally involve a lot of writing on whiteboards or pair programming or gathering around somebody’s screen to try different variations of a particular wireframe. Working sessions are better than even good meetings because by the end of the working session, you’re often done with whatever it was you were going to just talk about in the meeting. 

And maybe that’s the most important point here. Meetings are not conducive to DOING. They are conducive to TALKING. Talking is the enemy of doing. By making a few small changes in the way you conduct your meetings, you can turn them into places where things get done rather than just talked about. And that will make meetings suck a whole lot less. I promise. 

To Kill or Not to Kill

After my rant last week about product managers, the excellent Joshua Porter (@bokardo) made a great point about it. He said, “In my own experience the hard part is knowing when to kill something vs. when to give it more breathing room, as sometimes a really new idea can’t really be tested in low fidelity.”

As much as I’d love to send my pageviews soaring by starting a flame war with somebody popular on the internet, I have to admit that he’s 100% right. Killing a feature or product is exceptionally difficult. It’s tough to know when to do it. It’s tough to figure out if you made the right decision. And it’s tough emotionally to let go of something you really thought was going to be great.

First, let’s talk a bit about why you kill products or features. You kill them because they’re not succeeding or because you don't expect them to succeed. That could mean that they’re not getting enough traction or because you’ve determined that they’re never going to turn into an important part of your business. You kill them because the ROI isn’t high enough to justify investing more resources in them. You kill them because they are using resources that would be better spent in other places.

They’re hard to kill precisely because you never know whether they’re just a few days away from taking off and turning into everything you thought they’d be. After all, every successful product went through some period of time before everybody found about it.

So, let’s look at a few questions to ask yourself before killing a product or feature. For the purposes of this post, I'm just going to talk about killing existing features or products. I'll probably address how to decide to kill things before you build them in a future post.

These questions won’t make killing easy, but hopefully, they’ll make it possible.

Why Isn’t Everyone Using It?

There are four reasons people don’t use your product or feature. Yep. That’s right. There may be thousands of reasons that people do use a product, but there are really only four basic reasons that they don’t.

People don’t use your product because:

• They don’t know about it 
• It doesn’t solve a problem for them 
• They don’t understand that it will solve a problem for them 
• The problem it solves isn’t worth the investment of time, money, or effort

Before you kill your existing product or feature, figure out why it’s not popular. For example, if you’re simply not getting any traffic to your page, it means not very many people know it exists. On the other hand, if you’re getting tons of traffic, but none of it is converting or engaging, then your problem is one of the last three. People are finding your product, but they don't want it or understand it enough to convert.

You can find out if the product solves a serious problem for people by talking to the types of people you expect to have that problem. Develop a persona that represents the sort of person who might suffer from this problem, and interview them about that portion of their lives.

Don’t just ask, “do you suffer from x problem?” Have them tell you stories about their real life experiences in situations where they might have experienced the problem. For example, if you’re testing to see if people need turn by turn navigation on their phones, you might ask them to tell you about the last time they were trying to get somewhere and they got lost. Then you might ask how often that happens. If it’s extremely rare, they probably don’t have the problem you’re trying to solve.

If you’re trying to figure out if they understand the problem your product or feature solves, you can do that by showing them your product or feature (or a mockup or prototype) and asking them to tell you about it. Don’t prompt or prep them. Just show them the product and say, “Tell me what this does. Who do you think it’s for?” You will be shocked by how often your perfectly crafted prose and imagery cause nothing but blank stares.

When determining whether or not your product is worth getting, don’t forget that money isn’t everything to your potential users. Sometimes there are switching costs that they’ll have to deal with or just the cognitive load of learning how to use a new product. I can’t tell you how often I’ve seen people stick with a completely suboptimal solution to a problem, just because that’s what they’re used to.

Regardless of which it is, determining the reason people aren't responding positively to your product will go a long way toward telling you whether to kill it or keep it. 

Who Is Using It?

So, once you’ve determined that there are people who are using your product or who you expect will use it because it solves a serious problem for them at a price they’re willing to pay, it’s time to look at who those people there are and how many of them exist.

A great company with a very engaged group of users recently killed a feature. Unsurprisingly, there was a huge outcry. They got many, many complaints telling them how sad users were that the feature was going away. If they had gone entirely by the comments on the blog post about removing the feature, they would have been justified in thinking that they were making a huge mistake.

Luckily, they didn’t do that.

It turned out that the number of people using the feature was an incredibly small percentage of their user base. More importantly, the people using the feature were not, by and large, paying customers. In other words, a couple percent of very vocal users who didn’t earn a cent for the company were upset by the removal.

While it’s always best to avoid making your users angry, there are certainly users that it’s safer to anger than others. Keeping a feature or product that is disproportionately useful to people who aren’t benefitting your business in some real way means that you have fewer resources to devote to things that might make you some money. 

The other thing to consider here is how many people you might reasonably expect to have use this product if everybody knew about it. Unless there is a huge potential market for your feature or the small market that exists is willing to pay quite a lot to use it, you may want to consider killing it.

Note: for those few people who inevitably write to me and complain that “it’s not all about money,” I would like to point out that it very frequently does have to be about money or you will go out of business. If you want to keep your 10 free users super happy, you go right ahead. I’m going to cater to the large number of folks who pay me. 

And yes, I do understand the difference between long term and short term gains, and I expect my readers do, as well. Assume I'm optimizing for lifetime value here and not simply what makes the most money right this second.

What Is the Actual Cost of Keeping It?

Now that we’ve determined that people are using your product or feature, we should figure out how much it costs to keep your product or feature alive.

Of course, if you’re talking about your whole product, this math is relatively easy. The only hidden cost to keeping your product alive is the opportunity cost of building something else. If you’re working on your current product, you can’t be working on something more promising.

However, if you’re talking about a piece of your overall product, sometimes it can be harder to figure out how much it costs to keep a feature alive. Obviously there is the cost of the engineers or customer support people or sales people, but often they’re working on other things as well, so it’s not clear that cutting any particular feature will really save you any money. If even a few people are using a feature and it’s already built, why not just let it hang around indefinitely?

Well, consider some of these hidden costs to keeping a feature:

• Bug fixes 
• Customer support 
• A more complicated code base 
• A more complicated user interface (more features means more cognitive load on your new users) 
• Server and infrastructure costs 
• Additional work if you decide to do a site redesign or visual refresh

These may seem like small things, and in some cases they are, but don’t ever think that a feature is free just because you no longer are actively building it.

Of course, there's another alternative, which is to continue to iterate on the feature or product. This obviously adds hugely to the expected costs. Let's say that you create a new search feature for your product, and very few people end up using it. The actual cost of that feature needs to include all the iterations and changes that you're willing to try before people start using it or you give up on it. 

What Is the Actual Cost of Killing It?

In a similar vein, sometimes things can cost more to kill than you think. Unhappy users can cause trouble in forums or for support staff.

Of course, I did mention above that it’s sometimes acceptable (and inevitable) to annoy some of your users, but don’t underestimate the work that it will take to keep that unhappiness from spreading throughout your entire community.

There are engineering costs to turning off a feature, as well. Either you need to pull it out of your code base or leave it there to rot. Neither of those options is free, although both can be ultimately cheaper than maintaining the feature.

If you’re killing your whole product, you are often throwing away a huge percentage of your customer base. Just because you’re pivoting doesn’t mean that all of your users will pivot with you.

And the same goes for your employees, if you’re lucky enough to have any. Killing a product or significant feature can be absolutely terrible for morale. Obviously you’re not going to keep a failing product just to make your employees happy, but make sure that you are prepared for the fallout – and possibly resignations - when you do decide to make a major change.

So, Should You Kill It?

In the end, it really comes down to the expected ROI, and the future is notoriously difficult to predict. Good customer development techniques can help you get a clearer idea of the eventual potential of a product or feature that seems to be failing. An honest assessment of real costs can help you determine the investment that you're really making into the product.

But it's an art, not a science. In the end, you're still going to have to make the decision. And it may be the toughest decision you ever make as an entrepreneur. Good luck!

Like the post? Here's what you should do next:

• Follow me on Twitter
• Pre-order the book
• Attend a workshop

Expect More From Product Managers

I talk to a lot of startups, and I've noticed a really troubling trend at some of them. A huge percentage of Product Managers at startups suck.

Not you, obviously. I mean, you're a product manager at a startup, and you're the exception. Good for you. I mean all the other ones.

I mean the Product Managers who are failing to do the following obvious things:

Understand their Users

I was visiting a company, and I asked about what sort of ongoing user research they were doing. It turned out that they had an intern for a few weeks who was doing a giant research study to try to understand how people viewed the product. The Product Managers would occasionally attend a session, but basically they seemed uninvolved in the research. They knew nothing about the planning. They didn't monitor to see how things were going.

The research involved talking to a lot of people who had heard about the product but who had never used it before to see why they weren't using it. The Product Managers' main job with regard to research seemed to be to read a report produced by the intern at the end of the few weeks.

Meanwhile, nobody was doing targeted usability research on whether people were confused by the product. Nobody was doing any sort of inquiry into how big customers were using the product. Nobody was calling people who had stopped using the product to find out why. Nobody was following up with new users to understand their initial experiences.

Unsurprisingly, none of the PMs in charge of deciding what to build next had any real idea about usability problems, what separated their biggest customers from everybody else, or why people tried the service once and never returned. And that meant that they weren't very good at figuring out how to build their product, so when they did add a feature, it very rarely improved any of their key metrics.

The single most important thing you can do as a Product Manager is understand your users. This is the foundation for making every single decision about your product that you will ever have to make. If you don't know who your customers are and what motivates them, you can't consistently deliver features that will make them happy. It really is that simple.

Know Exactly How their Product Works

I was talking to a different startup about their onboarding flow, and I asked the product manager to explain how it currently worked for certain types of users under specific circumstances. The product manager said he'd have to check with the designer to get all the details.

The designer, of course, knew exactly how the product worked, since she was the one who had designed it. She also knew all about the user needs and the issues with the current flow and what she wanted to try next. All of this led me to wonder why on earth she wasn't the one in charge of the product.

When I attended meetings with both of them, the product manager constantly had to refer to the designer when talking aout what things they had tried, what had worked, and what users were currently seeing. Whenever an engineer asked a question about how something was supposed to be built, the PM would simply pass the question along to the designer and then relay the answer back to the engineers.

As far as I could tell, the PM was nothing more than a buffer between the engineers and the designer. Once the designer started working directly with the engineers, the PM had almost nothing to do except track the project schedule.

If you don't know everything about how your product works and why it works that way, how are you supposed to make good decisions about it? The role of a product manager isn't to be a passive conduit for information. A great PM needs to understand the product well enough to make important decisions about how to change it.

Validate Assumptions Before Building

I'm not going to tell a specific story about this one, because I've seen it happen absolutely everywhere.

Essentially, Product Managers seem to fall in love with ideas. Maybe they get the ideas from meetings or from their bosses or the ideas appear to them in a flash of light one morning in the shower. But regardless of how it happens, today's idea is always the Big Idea that is going to change everything.

Once they've got the idea, of course, they jump straight to the question of how to get it built. Typically, they get the feature on the schedule, break it down, get a designer in to mock it up, and get the engineers working on it. At no time before all this happens do they stop and ask the question, "How can I tell if this idea is any good before I spend a lot of time and money on it?"

No matter how good your instincts are, you don't really know that adding this new feature is going to be an enormous hit with your users. Sometimes you're right, and the feature is a game changer, but just as often, an unvalidated feature has a negative ROI.

Good product managers do as much work as possible ahead of time to figure out if they're spending their resources on the right stuff. Maybe they devise a small experiment to test whether people will use the feature. Maybe they do a very small version of the feature first. Maybe they do a concierge version of the feature. Hell, maybe they even sell the feature before they build it.

Whatever their strategy, good product managers validate their features before they build them, and that's why their ideas are so much more likely to improve the bottom line of the company. They don't necessarily have better ideas. They just kill the bad ones before spending too much time on them.

Prioritize Changes Based on Business and Customer Needs

Everything can't be a top priority. That's not how priority works. But you'd never know that to talk to some product managers. I can't tell you how often I've seen product managers try to build everything at once, because they simply couldn't make the decision about what was most important to either the business or to users.

Unfortunately, this tends to cause problems as engineering gets spread too thin and the team gets confused about what they should be working on.

Without clear prioritization, nothing really gets done well or quickly. Everybody ends up working on different projects, and all the projects move much more slowly than they would if everybody worked together.

A good product manager can make priorities clear without micromanaging. She makes the decision about what to work on based on things like expected ROI and the outcome of early validation tests. She balances features that are great for the business with features that are great for the users, and she always tries to find features that are good for everybody. She looks at things like long term vs short term pay off and makes sure that features are delivering value to all the different types of customers - new users, power users, business users, etc.

She clearly says that the entire team's goal is to improve a particular key metric and then makes sure that the team understands what that means. She then prioritizes which features should be delivered first while monitoring the metrics, and she keeps the team motivated to follow up by making progress toward the goal very clear.

Keep Engineers from Getting Jerked Around

How many of you have been working on something when a product manager has suddenly told you to stop working on that and work on something entirely different? How often does that have to happen before you become completely unmotivated? Not very often, right?

Of course, sometimes things change. Bugs are found that need to be dealt with. Business people make new deals. Company priorities get shifted around. Organizations get shuffled.

A good product manager protects her team from as much of this as she possibly can. She does this by pushing back on pressure from above to change priorities midstream. She does this by making sure that what her team is working on is important enough that she has a good reason to keep them on it, even if her boss suddenly wants something new and shiny. Sometimes she does it by making sure that there is somebody who can triage high priority bugs in order to keep the whole team from getting pulled off their work in the event of an emergency.

A good product manager manages up as much as she manages down.

Do Jobs that Aren't Technically Product Management

Just because the word manager is in the title doesn't mean you're off the hook for actually building things yourself. This is especially true at startups, where there are very few people trying to do a whole lot of work.

When I was working as a product manager, I frequently wrote copy, answered customer support questions, touched up images, built prototypes, consulted with the CEO on strategy, ran scrums, responded to people on Twitter, and made user research calls. Oh, and about hundred other jobs that had nothing to do with management.

I did those things because they had to be done, and there was nobody else to do them, but I also did them because the act of doing them made me a better product manager. By answering customer support questions, I learned how users felt about the product and where they got confused. By touching up images, I learned how much time it took to do the job so that I could gauge how efficient other people were when they were doing it. By monitoring Twitter, I learned who was talking about my product and what they were saying. These things were all critical to doing my job well, plus it meant that we didn't have to hire a bunch of specialists to do these things.

If you're not doing things outside of your normal job description, take a good hard look at whether you're really doing the most important parts of the job. I'll give you a hint, if what you're doing involves hours of meetings every week, there are more important things you could be doing.

This Sounds Hard

This sounds hard because it IS hard. Product Management should be hard. You're in charge of creating something. You have to make important decision affecting a huge number of people all the time. If you think it's all just going to meetings and making schedules for other people to stick to, you're doing it wrong.

Like the post? Follow me on Twitter!
Want more info on Lean Startup? Check out the Official Lean Startup Workshops Series I'm working on.

A Perfect Use for Personas

I was reading Dave McClure's post about changes to menus (and its not always flattering Hacker News thread), and I found myself both violently agreeing and disagreeing with both. I kept thinking something along the lines of, "That would be great! Except when it would be incredibly annoying!"

That's when I realized what was missing for me: personas.

 First off, apologies to Dave, who certainly doesn't need me to defend or improve his ideas. This is just meant to be an explanation of the process I went through as a designer and researcher to understand my weird, ambivalent reaction to his product suggestions. Here are the problems that Dave listed in his post that he was solving for:

• Too many items, not enough pictures, simpler & more obvious recommendations. 
• Not online, no order history, no reviews, no friends, no loyalty program, no a/b testing. 
• Have to wait forever for waiter to order, re-order & pay. 
• Nothing to do while I'm waiting. 

Then he presented reasonable solutions to these problems. All of the suggestions seemed geared toward making restaurants quicker, more efficient, and lower touch. Interestingly, both the Hacker News complaints and my own seemed to be from the point of view of people who do not have these problems. They were saying things like, "this would make restaurants awful!" but what they really meant was, "I, as a potential user, don't identify with that particular problem you're trying to solve, so your solution does not really apply to me."

In other words, Dave's suggested solutions might be great for people who have these problems but might not appeal at all to people who don't have these problems.

 So, then I started to think about the types of people who would have those types of problems. I put together a few rudimentary personas of people who likely would benefit from things like recommendations, entertainment while waiting, a more efficient order process, and a faster way to pay.

As a note, these personas are behavioral, not demographic. This means that you might sometimes fit into one of them and at other times you wouldn't. It depends more on what you do than who you are.

The Business Person

Imagine that you're on a business trip to someplace you've never been. You're quite busy, and it's likely that you'll have to eat a few meals on your own, possibly on the way to or from a meeting or the airport. You're not a fan of fast food, so you'd rather be able to find something you like at an interesting local place than at a big national chain.

 In this instance you might LOVE having things like recommendations from people you trusted, pictures on the menu of unfamiliar dishes, and a quick, efficient ordering and payment system that guaranteed you wouldn't hang around for twenty minutes waiting for a bill. You might also really enjoy some entertainment so that you'd have something to do that wasn't stare creepily at the other patrons.

The Barfly

Now imagine that you're at an incredibly crowded night spot. You are desperate for a bourbon, but you don't want to queue up five deep at the bar to try to get someone's attention. You manage to get a table, but now you have to decide whether to leave it to flag down one of the few waitresses or or just wait it out.

 In this instance you would almost certainly be excited to be able to order and pay directly from your table using some sort of tablet. You'd also be able to quickly order your second, third, and (dare I say it) fourth rounds without having to go through the whole process again or count on the waitstaff knowing exactly when to ask if you want a refill.

The Group Luncher

Last one for now, I promise. You're out to lunch with eight of your coworkers. You need to get back to the office in 45 minutes for another stupid meeting. You don't want to spend 10 of those minutes just for a waiter to make it to your table and take your orders. Also, you really don't want to be the one in charge of figuring out how to split the bill, especially since three of your coworkers always get booze, one of them never eats more than a salad, and two of them order the most expensive thing on the menu.

In this instance, you'd be thrilled to be able to just sit down, punch in your order (and your credit card!), get your food delivered to you quickly, and get to spend more time chatting with that cute new person in accounting rather than negotiating who forgot to figure in tax to the amount they owe on the bill. 

And the rest...

There are probably a half dozen other hypothetical persona groups, all of which would obviously need to be validated (or invalidated) with various forms of user research and quantitative testing.

 The persona groups that aren't on this list are also important. Many of these types of innovations might make things worse for the types of folks who are enjoying the experience of being in a restaurant as an event. For example, a romantic dinner for two at a high end restaurant is not improved by shaving thirty minutes off the wait between courses. Other people might enjoy the personal exchange with the waiter or a consultation from a sommelier more than reading about items on a tablet.

That's ok. These products aren't necessarily going to be for every type of restaurant all at once. There's no need to worry that suddenly Manresa is going to be putting pictures on the menu like Denny's.

 The reason I bring this up is that it often helps me to evaluate product ideas through the eyes of the people I expect to use the product. When I find myself saying things like, "Driving sucks! I'm going to fix driving!" I have to step back and realize that driving (like eating in restaurants) is an almost universal activity that has a constellation of problems, many of which are not shared by all types of drivers (or eaters). If you think your startup has a brand new product that's going to solve all the driving problems for stock car drivers, commuters, and truck drivers, I think you're probably wrong.

Instead of arguing back and forth whether or not these problems exist, it's very easy to identify particular types of people for whom these problems MIGHT exist and then do some simple qualitative research to see if you're right. After all, we know at least one person (Dave) has these problems that he wants solved. Presumably Dave (or the companies he invests in) are doing the sort of research necessary to make sure that there are enough people like Dave to make a profitable market. That market might not include you, but there are lots of wildly successful products you don't like.

 So. Long story short: personas, yay!

Postscript

For those of you who notice these things, you're right, I didn't include the personas for the other side of the equation: the restaurant owners. Whenever your customers (the people who give you money) and your users (the people who actually use your product) are different, you're in a much more complicated space from a user experience point of view. I'm assuming that, if we can make a specific type of end user happy enough it will make the types of restaurant owners who cater to those users interested in purchasing the product.

 That's just another hypothesis, and all hypotheses need to be validated, not assumed to be facts.