Why surveys suck
It’s because of people like me:
So you get a machine that dials phone numbers at random, so that there is no selection bias baked into your survey from the get go. You call a thousand random people, ask them some questions, record their answers, and you’re done. Easy, right?
In reality you always face a problem: some of the people that you call choose not to participate in the survey. In fact, quite a large proportion of the people you call do not participate. This introduces nonrandomness into your survey — any systematic difference between the kind of person who does respond and the kind of person who does not will bias your results.
And there’s no really good way to counteract that fact, either:
Are the nonresponders different from the responders in a way that is going to systematically bias the results? This is not a question that can be answered scientifically, and I think a lot of the techniques that are used to compensate for the nonrandomness that nonresponders introduce into the survey are pretty shoddy.
My understanding is that survey companies will try to weight the people who do respond to attempt to better represent the proportion of the population that their demographic makes up.
And in applying these handy fudge factors, they damage the randomness, hence the ultimate validity, of the data they so laboriously gathered. Not that the people paying for the survey really care, I suspect: you tell J. Hack Politician that 51 percent of some population somewhere does not actually want him tarred and feathered, and he’s happy.
Still, there is a feedback method that works, sort of:
Political polls, especially the ones reported the closest to the time of voting, have a very powerful feedback mechanism: once the vote is counted up, we can very easily evaluate the accuracy of each poll as a predictor of the outcome. If, at the time of the vote, Gallup is showing candidate A with 60% and candidate B with 40%, and B wins in a landslide, that makes Gallup look pretty inept.
If you can remember when Dewey defeated Truman, you can see how pollsters do have some incentives to improve their methods.
In the meantime, I avoid all telephone surveys and most Internet surveys. (I do complete the Consumer Reports questionnaire every year, mainly because it doesn’t ask for much in the way of opinions; either the Model PX-1107A broke during the preceding twelve months, or it didn’t.) And I expect to continue to do so.
Adam »
15 November 2009 · 9:52 am
Thanks for the double link!
I’d be interested in corroborating this, but according to Doug Rivers the response rate thirty years ago was something like 70%, while today it has gone down to an abysmal 15-20%. I found that pretty shocking–how can they trust their data at all at that point? Rivers claims that it is partly because telephone surveys today have to compete with a lot of telephone marketing, so people have less patience for any of them. Also, technology is such that a growing number of people don’t even have landline phones, and don’t expect to be called by anyone they don’t know on their cell phones.
I think Rivers’ technique of picking at random from a consumer database and then actually asking an individual with the same characteristics who has opted into their survey voluntarily is an interesting one. I don’t know if it solves the problem in areas where you don’t have a strong feedback, but it apparently has performed well for the few election cycles it’s been put through in its short history.
CGHill »
15 November 2009 · 10:13 am
Not to mention that you can no longer tell where a person is from his phone number, what with number portability and changes in the allocation of number blocks. Cell phones don’t even pay attention to area codes; for a while, I had a cell in area code 919, which is a long way away.
But I’d buy that phone-overload business: I get at least ten calls a week from credit-card types wanting to sell me additional services to increase the take. Not that I ever answer them.
Adam »
15 November 2009 · 10:25 am
I work at a startup, and when they were first ramping up hiring in July 2008, I had the misfortune of ending up with the phone that, for whatever reason, all the telemarketers called when they wanted to try and sell our business something. The upshot was that I was given full authorization to tell them off.
Francis W. Porretto »
15 November 2009 · 10:50 am
We can do better than Dewey/Truman. On the basis of a telephone poll it conducted, Literary Digest called this election for Alf Landon.
CGHill »
15 November 2009 · 11:38 am
Back then, elections had consequences: Literary Digest, having screwed the pooch, subsequently bought the farm. Then again, credibility actually meant something in those days.
McGehee »
15 November 2009 · 11:46 am
While everyone wants their opinion counted, I think most people no longer believe their opinions will be fairly reported.