Multivariate testing was the talk of this month’s Emetrics Summit in Washington. Everywhere I looked, people were singing its praises. And before the conference ended, Google announced a tool you can use to do simple multivariate testing for free. So if you don’t know what multivariate testing is, you need to check out this month’s Biznology newsletter.
In a past newsletter, we’ve discussed the “Do It Wrong Quickly” concept—you try something, see whether it works, and then try something else until you home in on the best approach. But how do you know you’re improving? You test and check the metrics. This month, we’ll look at multivariate testing, a powerful testing approach that dramatically increases what you learn from each test.
Direct response marketing pioneered the idea of the A/B test, where you test two versions of a direct mail piece, or a catalog entry, to see which version causes more response. Direct marketers use small test mailings to decide between version A and version B before they mail the better one to their entire mailing list. A/B testing is a bedrock practice for test marketing, but it assumes that either version A or version B is actually the best approach. Usually, neither one is.
Japanese engineer Genichi Taguchi improved on A/B testing, describing a process called multivariate testing. His insight was that testing just two possibilities against each other was very slow when you had many different variations to test. Taguchi showed how you can test several variables at the same time by carefully selecting different combinations so that the total test reveals which version of each variable is best. For example, if you want to test five versions of the copy and five versions of the image on your mailing piece, you can use different combinations of each so that you can test all 25 combinations at the same time, getting your answers for both variables at once.
But even Taguchi might be blown away by what computers can do with his original idea. Because computers can combine dozens of variations for dozens of variables, it’s multivariate testing on steroids. Computers can assemble every permutation of a page—dozens of copy and image variations for a dozen places on your Web page. The possibilities can be in the billions! Even a site that has heavy traffic can’t test even a thousand variations for a page, unless you use multivariate testing software.
Multivariate testing software can select which version of each variable should be tested on each page and can boil down those billions of possibilities to a few dozen. After testing those few dozen, you can run a second multivariate test with a dozen versions, then get it down to three or four for the last wave before you decide the right variants for every variable.
One interesting case study I saw came from Time-Life, the publisher of books and other media that spends lots of money on Direct response TV ads. Time-Life decided to use multivariate testing to improve the sales of their compilation album called “70s Music Explosion.” They found that their ads worked great to get traffic to their Web site, but few people added the album to their cart and even fewer actually purchased. They engagedOptimost to design a multivariate test that could yield a better design of this part of their Web site.
With Optimost, Time-Life identified ten variables they wanted to test, with several different variations of each. They had different copy they wanted to test, different images, and different offers. They wanted to know if showing a seal with their “money back guarantee” caused more sales (it did) and whether the order of the content mattered (it did).
Over a period if a couple of months, Time-Life took literally a billion possibilities and tested in a few waves to identify two different winning designs that improved the number of shoppers that dropped the album in their carts by 75% and increased actual sales by 5%.
Another case study was offered up by monster.com, in which it tested a two alternatives for each of seven different variables in a software download experience using Offermatica. In less than three weeks, the best design was chosen, which resulted in a 35% increase in downloads.
But the biggest news is that Google is offering free multivariate testing, using a beta tool called Website Optimizer. The beta test is by invitation only, so no one has enough experience with Google’s offering to compare it against existing competitors in the market (such as Optimost and Offermatica).
Don’t be left behind in the optimization race. Multivariate testing is your best ticket to higher conversions for your Web site.
