A simple example of this is to send out a small batch of email with different subject lines, and wait to see how many people open the email with one subject line versus the other. Then you use the winning subject line for the email that goes to the rest of the list.
If you have an enormous mailing list, that is a valid thing to do.
But if you have a smaller mailing list, it may be meaningless to do an A/B test because someone (cough, cough, your boss) thought it was a good idea. Why? Statistics.
There’s a whole science behind choosing a meaningful sample size. (Data science: aaagh!) Here’s a quick example. One of the mailing lists I manage has 3000 subscribers. If I do an A/B test with ten percent of my list, I’m pretty sure I don’t have enough of a sample size to make a data-driven decision.
There are other variables to consider too, such as:
- how much time to wait before you call the test finished.
- controlling other variables. Let’s say your test send went out Monday morning so you decide to send out the final batch the next Monday morning. But what if one of those Mondays was Super Bowl Monday?
- testing the right thing. If you’re testing two subject lines, you can measure their impact on open rates. Can you measure their impact on click-through rates? Not directly or in a meaningful way.
I am very far from being an expert on statistics, but Optimizely published an article on sample size and even offers a sample size calculator that you can use to either ensure that your next A/B test is valid or—admittedly my preference——talk your boss out of making you run an A/B test.
