The Dangers of Metrics (Only) Driven Product Development

When I first started designing, it was a lot harder to know what I got right. Sure, we ran usability tests, and we looked generally at things like page counts and revenue before and after big redesigns, but it was still tough to know exactly what design changes were making the biggest difference. Everything changed once I started working with companies that made small, iterative design changes and a/b tested the results against specific metrics.

To be clear, not all the designers I know like working in this manner. After all, it's no fun being told that your big change was a failure because it didn't result in a statistically significant increase in revenue or retention. In fact, if you're a designer or a product owner and are required to improve certain metrics, it can sometimes be tempting to cheat a little.

This leads to a problem that I don't think we talk about enough: Metrics (Only) Driven Product Development.

What Is Metrics (Only) Driven Product Development?

Imagine that you work at a store, and your manager has noticed that when the store is busy, the store makes more money. The manager then tells you that your job is to make the store busier - that's your metric that you need to improve.

You have several options for improving your metric. You could:

• Improve the quality of the shopping experience so that people who are already in the store want to stay longer
• Offer more merchandise so that people find more things they want to buy
• Advertise widely to try to attract more people into the store
• Sell everything at half off
• Remove several cash registers in order to make checking out take longer, which should increase the number of people in the store at a time, since it will take people longer to get out
• Hire people to come hang out in the store

As you can see, all of the above would very likely improve the metric you were supposed to improve. They would all ensure that, for awhile at least, the store was quite busy. However, some are significantly better for the overall health of the store than others.

A Review of UserTesting.com

A lot of people recently have asked me about the new crop of user testing tools available on the internet. One specific tool that comes up a lot is usertesting.com, and I’d like to talk a little bit about my experience with it.

Frankly, when I first saw the site, my initial reaction was, “Well, that’s not going to be very useful.” I’ve heard a similar gut reaction from several user researchers. Once I’d given it a try, my reaction changed to, “Oh, shit. This could seriously cut into my income.”

Having used it several times now, I can happily say that neither of these reactions was correct.

The Cons:

Let’s start with the things that I originally noted about usertesting.com that made me think it wouldn’t be very useful.

There is no moderator.

Not having a moderator  means that there isn’t a human being running the test who can ask follow up questions and delve deeply into issues that come up naturally during a session.  Good moderators don’t just follow a script; they ask the right questions to really understand why users are doing what they’re doing.

It’s less useful for testing incomplete interactive prototypes.

I have not yet found a good way to test prototypes with usertesting.com. When I design, I create very sketchy, incomplete mockups of products. Typically, these only run in one browser, they have no visual design, and large parts of them won’t really work or will use fake data.

I can deal with all of these issues in a one-on-one session by explaining that the prototypes are not real, giving the participant some help in areas where the prototype isn’t perfect, or gently reminding them not to fixate on the visual design. This isn’t really possible without a human moderator. 

Pie-Jacking and Other Tips for Making Engineers Tolerate UX

Obviously, I find UX to be incredibly important. And these days, I’m finding more and more people who agree with me. Unfortunately, in many organizations, there are people who still feel that user research and interaction design slow things down or aren’t really necessary.

Sometimes, those people are engineers. Not always, of course. I’ve worked with lots of engineers who are very excited about the idea of good design and getting user feedback. But occasionally you run into groups of engineers who have yet to be convinced.

In the interest of achieving harmony between the Engineering and UX departments at your company, here are a few tips for convincing people (especially engineers) of the value of user research and design.

To be clear, I am in no way suggesting that you trick engineers or lie to them or manipulate them in any way. I’m working on the assumption that engineers are frequently extremely logical people who just need evidence that things are useful before they buy in.

Involve Them in the User Research

The number one way to get anybody excited about user research and UX is to get them involved with it. Too frequently, the only thing about user research that engineers get to see is a thirty page paper detailing all of the problems with their product. This is boring, painful, and easily ignored.

In my experience working with dozens of companies, the single most powerful tool for getting engineers in touch with users was having them sit in on a few usability sessions. The sessions didn’t have to be formal. I’d just put the engineers in chairs in the back of a small conference room and have a participant use the product in front of them.

Without fail, the engineers started to understand the pain that their users were feeling. And, since engineers are not monsters, they wanted to help those people. They would fix bugs they observed during the sessions. They would ask for advice on how to improve screens that they had previously thought were just fine.

Most importantly, the engineers would suddenly understand how different they were from their users! Suddenly, it became much harder for the engineers to believe that they had all the answers to their customers’ problems, since they had seen that they were really quite different.

6 Incredibly Important Lean UX Lessons

A project I’ve been working on recently should really be featured in all books on agile and lean design. I’ve found that many projects and clients have similar issues with their design processes, but it’s rare that a single project clearly demonstrates so many incredibly important tenets of lean UX. 

So far, over the course of about a month, this one has driven home all of the following lessons:

• You shouldn’t be a slave to your metrics, especially when they’re telling you something surprising
• Qualitative testing is necessary for understanding why your metrics are what they are
• Iteration in the design process is critical
• Very small changes can make a huge difference
• You need to understand the actual experience your user is having with your product
• Don’t give up on your vision too early!

So, what happened?

I’m working with a very cool startup that is absolutely devoted to metrics. They A/B test and measure everything. When they first brought me on, the team went over some history of the product.

They originally had a particular flow for the product, but when they did qualitative testing, it became clear to them that users expected things to behave differently. The team designed a new version of the product that they felt addressed the issues that testers were having. They then built and released the new version.

It bombed. Well, perhaps that’s a bit strong, but it performed significantly worse than the original design in an A/B test.