Reader Anna Green challenged me to write my own success story to illustrate just what kind of entries I am expecting for my new Do It Wrong Quickly contest. So here goes.
You Can’t Plan a Guess
Back in 1999, I was given the assignment of improving the site search facility for a Fortune 10 Web site and found that the search results were worldwide, even though we knew that searchers wanted results primarily from the country site they lived in (that information was in their own language and sold products using their currency).
Many searchers looked for product information and got back the wrong stuff. Imagine that you are looking for a brand name product but get back information in a language you don’t speak, priced in money you don’t have, that uses an electrical plug you can’t use. You’re likely to shop somewhere else.
The simple solution would be to show the pages for the right country at the top, showing more generic pages lower in the list (in case the country pages don’t answer the question). The problem was that the pages were not tagged with the country they were from, so changing the search engine to look for pages tagged with the right country would eliminate most of the right answers from the search results. (If you tell the search engine to find pages tagged with a certain country, but 80% of the pages that really belong to that country are not tagged, those 80% won’t be found by search when they should be.)
Both of my predecessors in the job decided the best way to fix this problem was to embark on a company-wide page tagging campaign. With a couple of million pages, this was a difficult task and a slow one. Neither person had come anywhere near succeeding, and they were considered failures at the job of improving search when they left.
Many meetings had ensued over how to tag the pages, but no real consensus emerged. Small countries that had only a few hundred pages said they had only one person responsible for the Web site and had no time to spend days tagging pages. Large countries with dozens or hundreds of Web employees said they had no time to encode tens or hundreds of thousands of pages. No one could figure out want way to write a program to accurately tag the pages. Studies, task forces, and meetings continued to search for the right answer, but no one could agree on it.
I resolved to find an easier and faster way of solving the problem than hand-tagging two million pages. I first looked at the URL patterns of the pages, hoping to see that all country pages used a similar pattern. Many did, but many did not. Dozens of different patterns existed for pages within the same country, with some important country pages that had no real pattern. And we had no pattern at all for US pages—there was no way to tell a “global” page from a US country page. It seemed hopeless to come up with a set of patterns that would work perfectly.
Tagging the pages was the right way to do it, but slow. Coming up with an all-encompassing set of URL patterns would be a right way too, but seemed impossible. I decided to do it wrong quickly.
I convened a meeting of the representatives of the Web sites for each country. I showed them the patterns that I had discovered and my team created a tool with which they could add more patterns for their countries. I set up a test search server that respected those country patterns and they quickly saw that the search results were better, even though Web pages from the wrong countries still often showed up in the test search results.
Each time someone found an erroneous result in the test, we added that pattern to its proper country list. So if we were searching within France pages and found one from Canada, we added the pattern of that page to the definition for Canada. Over a period of a few weeks, the results became remarkably good, as we continually updated the now complex set of patterns each time we saw an error.
That still left the US. We decided to create a similar set of patterns to define all “global” pages (pages that were considered to be good answers for every country), and to treat as US pages every page that did not fit any of our other patterns. So, any page not identified as a country page or a global page was treated as a US page.
We launched this new search facility within a few months and left the feedback mechanism in place, so that country representatives could forever maintain the URL patterns to tweak their country results whenever they saw an error. Our customer searchers loved the new precision of finding just the pages within their countries, with surveys telling us that 5% more searchers found what they were looking for—this for a minimal investment in people’s time, almost no technology, and no costly content changes.
Although it seemed like a dumb idea to some at the time because it was not the “right” way to fix the problem, doing it wrong quickly made a huge difference. The patterns that made up each country’s URLs were not planned, so we couldn’t plan how we were going to guess what they were. We just had to experiment—we had to do it wrong quickly and then fix it.
—————
So that’s my example of a “do it wrong quickly” success story for my new Do It Wrong Quickly contest. Yours doesn’t have to be this long, but it should have some of the same elements. You should go after a real problem that requires experimentation to be solved really well and you should show that it worked. Now your problem could be as simple as having sales lower than you wanted—it need not be as specific as mine. Thanks in advance for your submissions.
