Trending Now

How do you know how long to A/B test your website changes?

Everyone wants to be data-driven these days, so A/B testing is all the rage. Too often, however, I run into clients that run dozens of tests a month and come back with the disheartening results: Not enough repetitions to know the right answer.

This has probably happened to you. You have a critical page on your website that has a high exit rate. It is close to an important conversion page, so you know it will be valuable to optimize that page for clickthrough to that conversion. So, you run an A/B test. You have a few ideas of how to make things better, but after a couple of months, you still aren’t sure you have enough repetitions to know whether to believe the test.

How can you know?

One way is to go for statistical significance. That makes sense, but it isn’t always easy when you have low page view rates. How much is enough? It usually depends on how far apart the A and the B tests are. If they are wildly different, you don’t need as many repetitions. But that still begs the question.

Another way to think about this problem is not with an A/B test, but with an A/A test. There will be some random noise in any test. One way to detect how many repetitions you might need for statistical significance is to run an A/A test–where you use an A/B testing tool to run the exact same experience against itself. Once you get enough repetitions, the A/B test tool should tell you that there is no winner. The results should get closer and closer together as more repetitions occur.

You might be disappointed to find out that it will take months to declare a winner on some of your low-trafficked pages. If you have high-value conversions, you might want an answer sooner. Enter machine learning. It’s possible for trained machine learning tools to predict the conversion rate, bounce rates, and other statistics before your page is even published. Doing so allows you to make changes to at least get the page up to “average” before the hard work of A/B testing starts. If you have high-value pages on a low-trafficked website, you will be glad you did.

Mike Moran

Mike Moran is a Converseon, an AI powered consumer intelligence technology and consulting firm. He is also a senior strategist for SoloSegment, a marketing automation software solutions and services firm. Mike also served as a member of the Board of Directors of SEMPO. Mike spent 30 years at IBM, rising to Distinguished Engineer, an executive-level technical position. Mike held various roles in his IBM career, including eight years at IBM’s customer-facing website,, most recently as the Manager of Web Experience, where he led 65 information architects, web designers, webmasters, programmers, and technical architects around the world. Mike's newest book is Outside-In Marketing with world-renowned author James Mathewson. He is co-author of the best-selling Search Engine Marketing, Inc. (with fellow search marketing expert Bill Hunt), now in its Third Edition. Mike is also the author of the acclaimed internet marketing book, Do It Wrong Quickly: How the Web Changes the Old Marketing Rules, named one of best business books of 2007 by the Miami Herald. Mike founded and writes for Biznology® and writes regularly for other blogs. In addition to Mike’s broad technical background, he holds an Advanced Certificate in Market Management Practice from the Royal UK Charter Institute of Marketing and is a Visiting Lecturer at the University of Virginia’s Darden School of Business. He also teaches at Rutgers Business School. He was a Senior Fellow at the Society for New Communications Research and is now a Senior Fellow of The Conference Board. A Certified Speaking Professional, Mike regularly makes speaking appearances. Mike’s previous appearances include keynote speaking appearances worldwide

Join the Discussion

Your email address will not be published. Required fields are marked *

Back to top Back to top