The potential fallacy of polls
I have become something of a skeptic when it comes to polls and surveys. While they are the theoretical best way to get aggregate viewpoints of a consumer or user base, people’s inclinations toward self-perception make them inherently flawed.
Take, for example, this Accountemps survey on online shopping for the 2009 holidays.
“Most workers,” the release says, “plan to browse on their own time, a new Accountemps survey shows. Nearly four out of five (77 percent) professionals surveyed said they are not planning to shop online while at work.”
Isn’t that a nice statistic for Accountemps to give back to its hiring firms? Staff are hard-working and focused!
Except it’s probably false. For the survey, Accountemps had a separate firm interview 455 employees–a number of whom were probably concerned that their employers might be listening or getting access to the data, so they lied about it to sound good.
Other respondents probably said to themselves, “Well, I’ll probably browse a bit here and there, but I won’t actually shop until I get home.” So they got to say no, which made them feel better about their dedication to their jobs. And some people just didn’t want to admit that they goof around at work, so they answered the same way.
It’s impossible to determine from my desk what those subsets do to the data, but at the least, they make the original results much more suspect.
The perpetuation of self-interest and positive self-perception is a common theme in polls. Ai recently received client research that said a celebrity spokesperson had only marginal effect on the opinions of the people polled. To which I thought: well, of course the respondents say this.
Few people want to admit to a curious stranger that Roger Federer’s mug on TV is the reason they considered Credit Suisse for their asset management. But that image certainly influences people, even those who won’t explicitly acknowledge it. (Nate Silver’s marvelous poll aggregation during the 2008 election cycle reinforces some of this.)
Polls and surveys aren’t going away, and the insights they contain are often valuable and impossible to otherwise discern. But the questions they seek to answer may not always be fully answerable by a conscious group of respondents.