A/B testing has become a popular way to evaluate everything from website design to online offers to product descriptions. But many managers don’t design these experiments carefully, relying on software that lets them track hundreds of metrics. The problem is that if you’re looking at such a large number of metrics, you’re at risk of seeing what statisticians call spurious correlations. In other words, the more things you’re measuring, the more likely it is that you’ll see random fluctuations. You aren’t asking a useful question like “What’s happening with this specific variable?”; you’re asking “What general changes am I seeing?” So design a better test by deciding on the metrics you’re going to look at before you execute an experiment. And limit yourself to a few metrics. If you know exactly what variable you’re interested in and how you’ll measure it, you have a much better chance of seeing (and reporting) significant results.
Adapted from "A Refresher on A/B Testing," by Amy Gallo
