Last week, I wrote about an up-and-coming player in multivariate testing, SiteSpect, which allows you to test almost any aspect of your user experience, unlike the standard JavaScript testing approaches. SiteSpect also eliminates the need to modify your site at all, speeding the test process so you can Do It Wrong (more) Quickly. I conducted an e-mail interview with one of SiteSpect’s happy customers, Seth Moore, the Manager of Marketing Analytics and Testing at Overstock.com.
Me: What’s your background, Seth? How did you reach the position you are in today?
Seth: I came to Overstock two years ago from the U.S. Senate and have a background in political science and quantitative methods.
Me: How long has your company been doing A/B or multivariate testing? Did they use other vendors before SiteSpect? What were the results?
Seth: We have been doing testing for years. We tried most of the major platforms and couple of in-house platforms. All the testing platforms we have used have added value to the company. The difficulty for us was always the agility and adaptability of the platform. We always wanted to avoid development time where possible, and SiteSpect has been the best for quickly deploying tests without development changes to the Web site code.
Me: Are senior managers aware of the testing that you do? What kind of persuasion was required to get this kind of testing in place? What kind of communication do your top execs get about testing now?
Seth: A high-level involvement in testing has never been a problem for us. The CEO follows test results like some follow the NBA playoffs. In fact, there is a healthy amount of competitive banter about whose site ideas are performing the best, so everyone watches the results like a spectator sport.
Me: Why is testing so important to your company? What has been the impact to your business?
Seth: Testing is critical to making sure that Web site optimization is not “two steps forward and one step back.” Often, we find that the results are counterintuitive to what we had assumed, so now we hesitate to make site changes without testing. The Web site has, of course, improved in efficiency throughout our company’s history, but testing has accelerated our rate of improvement markedly.
Me: How did you hear about SiteSpect? What attracted you to their offering?
Seth: When SiteSpect first made their sales pitch to us, we were drawn to the idea that we could test without altering the Web site code. If development has to insert extra JavaScript on the page or make other code alterations, it creates a bottleneck in the site optimization process. SiteSpect also gives us access to the raw data from tests allowing us to do deep statistical dives and segmentation rather then just looking at aggregated data.
Me: How easy is SiteSpect to begin using? How long did it take after you signed a deal to start your first test?
Seth: SiteSpect uses regular expressions heavily, so that was the bulk of the learning curve. We were able to deploy SiteSpect quickly and had a test up immediately thereafter.
Me: How has SiteSpect lived up to what you expected? What features distinguish it from the competition?
Seth: We have been pleased with the results from SiteSpect. It is highly adaptable, easy to integrate with Omniture and other web analytics vendors, and provides a lot more statistical insight into our test results then most platforms.
Me: Is there anything about the way you do testing that my readers would like to know about that I haven’t asked about? What would you advise people who aren’t conducting tests? What would you advise people who are conducting tests but are not using SiteSpect?
Seth: All online business boils down to one simple equation: traffic x conversion rate x conversion value = revenue. The more money you invest in advertising, the more you ought be investing in making that traffic count. SiteSpect offers a sophisticated tool at a good price.
Me: Thanks so much for sharing your experience with my readers.
