Check out my guest blog post for KISSmetrics: Why Website Test Results Don’t Always Add Up & What To Do About It!
Here's a little sample:
If you do enough A/B testing, I promise that you will eventually have some variation of this problem:
You run a test. You see a 10% increase in conversion. You run a different, unrelated test. You see a 20% increase in conversion. You roll both winning branches out to 100% of your customers. You donʼt see a 30% increase in conversion.
Why? In every world Iʼve ever inhabited, 10 plus 20 equals 30, right? Youʼve proven that both changes youʼve made are improvements. Why arenʼt you seeing the expected overall increase in conversions when you roll them both out?
Read the Rest at KISSmetrics.