Trending Now

Statistical significance is overrated

I’m a big advocate of measuring the success of your site, but many marketers find the statistics intimidating. Many marketers are more comfortable with the dreaded “anecdotal evidence” than they are with numbers—that’s changing, but slowly. I am wondering if the intimidation might be caused by statisticians themselves.


One of the most intimidating parts of the numbers game is statistical significance. Many marketers struggle with this concept, even those normally comfortable with numerical analysis.
To the uninitiated, when a statistician says that a number is not statistically significant, they interpret that as meaning that no conclusions can be drawn from that number. But that’s wrong. Let’s take an example.
If you tested three different e-mail messages to see which one has the highest click rate, the statistician (or your computer statistics program) might tell you that the third version was significantly worse than the first two, but that the difference between the first two was insignificant. What do you do?
You could run another test. You could eliminate the third version and retest the first two. Your numbers person will tell you that you need many more e-mail recipients to achieve statistical significance, because the two e-mails are quite close to each other in effectiveness.
If you want to “do it wrong quickly,” then just send out the one that tested better, regardless of whether it is a statistically significant difference. Why? Because the two e-mails are close enough to each other that sending either one is probably OK. And chances are that the one that tested slightly better really is better, even if you don’t know for sure.
To understand why this is true, you need to understand what “statistical significance” means—it means that you are 95 percent sure that your conclusion is correct. That means that even when you achieve statistical significance, you have a one in 20 chance of being wrong.
A different statistical methodology is gaining traction, called Bayesian probability, which takes a different approach, based on the persuasiveness of the data. Bayesian aficionados argue that when you know where you are starting from, such as the conversion rate for your shopping cart page, you can persuade yourself that a new page design is “working” with a very small number of successes. That small sample size is not statistically significant, but it is probably the right conclusion for your decision.
If you can’t stop yourself from aiming for statistical significance, remember that the bigger the change, the smaller the sample size you need. So try to go after things that raise your conversion rate ten percent rather than one percent. That might sound obvious, but often we need to make a concerted effort to “think big” rather than shooting only for small improvement.
Better yet, don’t get hung up on statistical significance at all. It’s better to make ten decisions in a row with 70 percent confidence than just one that you are 95 percent sure of. By making frequent changes, the ones that turned out to be wrong will be found out soon enough.
If you’re still thinking that you need statistics to prove that your results aren’t random, just remember the words of comedian Dick Cavett: “Just think of all the billions of coincidences that don’t happen.”

Mike Moran

Mike Moran is a Converseon, an AI powered consumer intelligence technology and consulting firm. He is also a senior strategist for SoloSegment, a marketing automation software solutions and services firm. Mike also served as a member of the Board of Directors of SEMPO. Mike spent 30 years at IBM, rising to Distinguished Engineer, an executive-level technical position. Mike held various roles in his IBM career, including eight years at IBM’s customer-facing website, ibm.com, most recently as the Manager of ibm.com Web Experience, where he led 65 information architects, web designers, webmasters, programmers, and technical architects around the world. Mike's newest book is Outside-In Marketing with world-renowned author James Mathewson. He is co-author of the best-selling Search Engine Marketing, Inc. (with fellow search marketing expert Bill Hunt), now in its Third Edition. Mike is also the author of the acclaimed internet marketing book, Do It Wrong Quickly: How the Web Changes the Old Marketing Rules, named one of best business books of 2007 by the Miami Herald. Mike founded and writes for Biznology® and writes regularly for other blogs. In addition to Mike’s broad technical background, he holds an Advanced Certificate in Market Management Practice from the Royal UK Charter Institute of Marketing and is a Visiting Lecturer at the University of Virginia’s Darden School of Business. He also teaches at Rutgers Business School. He was a Senior Fellow at the Society for New Communications Research and is now a Senior Fellow of The Conference Board. A Certified Speaking Professional, Mike regularly makes speaking appearances. Mike’s previous appearances include keynote speaking appearances worldwide

Join the Discussion

Your email address will not be published. Required fields are marked *

Back to top