Where business and technology create a winning customer experience.

A while back, I wrote about online panels, a kind of focus group on steroids that companies are using to both lower their research costs and to scale survey data to be more quantitative than typical focus groups. These online panels allow more participants than focus groups, offer better representation of your target market, and scale as easily as surveys, usually at lower cost. What’s not to like?

Nothing, if you use online panels properly.
My worry, perhaps unwarranted, is that old-time marketing folks will mistake an online panel’s ability to answer the “why” question for an ability to answer the “what” question. In other words, I mistrust people to tell you what they will really do in all situations, but I do trust them to tell you why they would do something.
We know that people don’t always do in real life what they say they will do when questioned in an artificial situation. They may be embarrassed to say they don’t know what they’d do, or feel uncomfortable saying what they would really do, or perhaps they don’t know. As an example, no one surveyed would ever tell you that they prefer to click on more pages to get the right answer, but in reality they sometimes do. We know from tracking real users that people prefer several “easy” clicks—ones that require little thought—to one very painful click where they must read a lot and then take a guess at the right thing to do next. But they would always tell you on a survey that they prefer fewer clicks; we know what they really do only by watching clicks from real users on live sites.
So use online panels for what they are good for—brainstorming ideas or getting to the “why” behind what you know people do. But use real activity tracking to see “what” people do. Let your metrics system prove what people do in real life and then use online panels to find out why they do—between them, you’ll have the information you need to identify areas for change and change them. Because in the old days we had no mechanism to find out “what” customers were doing, I don’t want us to sieze upon online panels as the only information we need—let’s use all the information available: both activity and survey data.

Tweet about this on TwitterShare on FacebookShare on Google+Pin on PinterestShare on LinkedInEmail this to someone
Mike Moran

About Mike Moran

Mike Moran has a unique blend of marketing and technology skills that he applies to raise return on investment for large marketing programs. Mike is a former IBM Distinguished Engineer and a senior strategist at Converseon, a leading social consultancy. Mike is the author of two books on digital marketing, an instructor at several leading universities, as well as a Senior Fellow at the Society for New Communications Research.

2 replies to this post
  1. Mike, I’m with you on this and perhaps even a bit more harsh on the current use of surveys online. This is an area where “lame practices” seem to be all around us.
    In the “old days” at HSN, we did an annual customer satisfaction tracking study. In this case, we had the mechanism for “what” and surveyed many segments with known behavior, then tracked their behavior over time.
    We found in fact many people:
    1. Have no idea what they do, example, they say they have made 5 purchases in the last year when they have really made 10
    2. Do the opposite of what they say they will do, example, say they will continue to purchase when in fact they do not purchase again
    This is why surveys that are not tied to actual behavior are dangerous.
    If your end game is to affect behavior, segment the behavior *first* and then do your surveys so any information / results of actions taken can be tracked against actual results.
    Likewise, make sure you understand who you are surveying from a behavioral perspective so you can really understand the survey results.
    For example, let’s say you do a survey that provides a particular direction and you act on that direction. Only later do you find out the survey audience contained primarily light users and the action you have taken impacts heavy users negatively.
    Clearly an area where a lot of work needs to be done, given how much hype goes into “feedback”.
    I’m waiting for the first study that proves how different survey results and actions taken are, wonder who will be brave enough to publish it, given all the hype? I’m not holding my breath though; the problem is people are heavily invested in this stuff and a giant “mea culpa” is not on their schedule…

  2. This is great feedback, Jim. I am writing a post today to help people see how they can use online panels to get good survey feedback–the “why” not the “what.”

Leave a Reply



Yesterday, our author Chris Abraham  presented our latest Biznology webinar about how to grow your social media influence.  Whether you’re...