The Dangers of Metrics (Only) Driven Product Development

When I first started designing, it was a lot harder to know what I got right. Sure, we ran usability tests, and we looked generally at things like page counts and revenue before and after big redesigns, but it was still tough to know exactly what design changes were making the biggest difference. Everything changed once I started working with companies that made small, iterative design changes and a/b tested the results against specific metrics.

To be clear, not all the designers I know like working in this manner. After all, it's no fun being told that your big change was a failure because it didn't result in a statistically significant increase in revenue or retention. In fact, if you're a designer or a product owner and are required to improve certain metrics, it can sometimes be tempting to cheat a little.

This leads to a problem that I don't think we talk about enough: Metrics (Only) Driven Product Development.

What Is Metrics (Only) Driven Product Development?

Imagine that you work at a store, and your manager has noticed that when the store is busy, the store makes more money. The manager then tells you that your job is to make the store busier - that's your metric that you need to improve.

You have several options for improving your metric. You could:

• Improve the quality of the shopping experience so that people who are already in the store want to stay longer
• Offer more merchandise so that people find more things they want to buy
• Advertise widely to try to attract more people into the store
• Sell everything at half off
• Remove several cash registers in order to make checking out take longer, which should increase the number of people in the store at a time, since it will take people longer to get out
• Hire people to come hang out in the store

As you can see, all of the above would very likely improve the metric you were supposed to improve. They would all ensure that, for awhile at least, the store was quite busy. However, some are significantly better for the overall health of the store than others.



The same thing happens all the time when designing products. If your assigned goal is to increase the number of active users, there are lots of different design changes you could make, but not all of them will be equally effective for improving the actual goal, which is probably increasing the number of people who use the product and generate revenue.

How Does This Happen?

I think the biggest reason that this happens is that people fixate on metrics without understanding the reason behind the numbers. Designers and product owners are then pressured to move a number that represents a particular metric rather than focusing on improving the product.

One company I talked with had this problem with acquisition. The person who was responsible for acquiring new customers was simply given a budget and told to get as many users as possible for that amount of money. Unfortunately, the users that were cheapest to acquire were the least likely to spend money on the site. If, instead of trying to maximize the number of users, he had concentrated on maximizing the number of users who were likely to spend money, he would have acquired fewer people and missed his metric, but he would have increased revenue.

Another company had a design problem. They wanted to redesign their Invite a Friend feature to encourage people to invite more friends. Unfortunately, the "most effective" method of getting people to invite friends was to forcibly spam users' Facebook feeds and make it easy for users for accidentally invite everybody in their address books. While this resulted in more invitations sent, it also vastly increased the number of unhappy customers and decreased the percentage of invitations that were accepted. It also caused the company to be banned from Facebook and put on the spam list of several ISPs. It sure improved that invitation metric, though.

How Should You Avoid It?

There are three ways to avoid this problem, and you should use all of them.

Make sure that you're measuring the right metric.

If you care about revenue (and you should), measure revenue. If you care about retention, measure retention. If you care about page views, you're probably doing something wrong.

Unfortunately, it can be difficult to immediately see the impact of a particular design change on things like revenue and retention, which makes it tempting to use substitutes for the important number.

If you are using a substitute - for example, if you're using something like "customers returning once" as a shorthand for "becoming an active customer" make sure that the link is actually causal. In other words, if you can increase the number of people who come back once, make sure that that really does lead to an increase in people who become an active customer.

Make sure that you're not gaming the metrics.

Paying people to come back to your site may result in more returning customers, but it doesn't necessarily result in more customers paying you. If you cut your prices in half, you may end up selling twice as many items, but you're not making any more money. Make sure that, if you're moving a metric, you understand the second (and third and nth) order effects of whatever change improved the metrics.

Make your customer experience better.

This may seem obvious, but pissing off your customers is a terrible long term strategy, even if it briefly moves a metric. On the other hand, improving your customers' overall experience leads to happy, contented customers who stick around and continue to pay for your service. Over time, that's going to improve all of your metrics.

And always remember, metrics are just shorthand for real customer behaviors that are important to your business. They are a tool to help you understand your product, not a goal to be met at any cost. 

Like the post? Follow me on Twitter.