Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

July 30, 2008
By: Kevin Drum

PROBLEMS WITH POLLING....Taegan Goddard gives us a preview of The Opinion Makers, by David W. Moore:

The author — a former senior editor of the Gallup Poll — says that today's opinion polls misfire due to an intrinsic methodological problem: survey results don't differentiate between "those who express deeply held views and those who have hardly, if at all, thought about an issue."

This is disturbing. Either Moore managed to find a publisher for a book thesis about as obvious as "college students like to drink," or else Moore's thesis actually isn't as bog obvious as I think it is. I'm not sure which is worse.

Or there's a third option: his thesis really is as obvious as I think it is, but everyone keeps pretending not to know it anyway. Which means it's worth a book. Good luck, David!

Kevin Drum 3:01 PM Permalink | Trackbacks | Comments (21)

Bookmark and Share
 
Comments

survey results don't differentiate between "those who express deeply held views and those who have hardly, if at all, thought about an issue."

It occurs to me that both kinds of folks get to vote ...

Posted by: Tim Morris on July 30, 2008 at 3:08 PM | PERMALINK

*

Posted by: mhr on July 30, 2008 at 3:10 PM | PERMALINK

The big danger is not in the horse-race numbers of who would win an election. It's in the "cross-tabs" and secondary questions that reporters love to claim is the "real" reason they watch the polls. Let's say, for example, that 10% of Americans say they won't vote for Obama BECAUSE he's Muslim, or to be bi-partisan, won't vote for McCain BECAUSE he's been brainwashed by his Vietnamese captors. So the pollster and the reporter merrily report this as if this is the unvarnished truth. But if we delved deeper -- or even just thought for a minute -- we'd realize that virtually anyone who cites these as their "reasons" is unreachable anyway. So we spend a lot of time worrying about a troubling poll result, when really all the poll is telling you is that 10% of people who respond to polls act like jerks.

Posted by: tom veil on July 30, 2008 at 3:17 PM | PERMALINK

Will someone please call Nurse Ratched for me?
I agree with mhr and Orwell.

Posted by: thersites on July 30, 2008 at 3:34 PM | PERMALINK
Nuff said.

Posted by: Orwell

Well thank God for that.
.

Posted by: Grand Moff Texan on July 30, 2008 at 3:35 PM | PERMALINK

bet ya all those statisticians and pollsters have thought endlessly about that exact question and written scores of papers on it and how to control for it.

bet ya.

Posted by: alex on July 30, 2008 at 3:58 PM | PERMALINK

In the social sciences, particularly in economics, theories often include assumptions that may be false, and nobody cares as long as the theories work.

It is obvious that most polls do not distinguish between "those who express deeply held views and those who have hardly, if at all, thought about an issue." What is not so obvious is whether it is actually important for pollsters to make that distinction. Some polls do ask respondents questions designed to elicit information about how strongly the respondent feels about something. Currently, pollsters don't do this very often, presumably because pollsters and their clients have concluded, based on past experience, that it's not worth the trouble.

Posted by: Kenneth Almquist on July 30, 2008 at 4:08 PM | PERMALINK

I think your third possibility is probably the right one.

What the Democrats need to do hire Marlee Matlin's character from The West Wing to do all of their polling. She not only gets this point right, but she also figured out that many people give superficial answers to poll questions without revealing their true feelings about a subject. That's a lesson Democrats -- who have hired a million Mark Penns and Bob Shrums to read polls at face value -- could well learn.

Posted by: AMP on July 30, 2008 at 4:10 PM | PERMALINK

bet ya all those statisticians and pollsters have thought endlessly about that exact question and written scores of papers on it and how to control for it.

bet ya.

Maybe not for a variety of reasons, one being this:

It is difficult to get a man to understand something when his salary depends upon his not understanding it. -- Upton Sinclair

It might be their dirty little secret....

Posted by: jerry on July 30, 2008 at 4:11 PM | PERMALINK

The problem is that members of both groups get 1 vote each.

Dukakis lost for many reasons, one of which was a significant group of voters who saw the Boston Harbor ad and voted for Bush as the better environmental candidate.

Posted by: snoey on July 30, 2008 at 4:19 PM | PERMALINK

Now, why didn't Moore read this article?

Blumer, H. (1948). Public Opinion and Public Opinion Polling. American Sociological Review 13, 242-249.

Posted by: lurker on July 30, 2008 at 4:31 PM | PERMALINK

I think mhr is confusing "was once ahead in the polls" with "predicted... would defeat." The final Gallup poll had Roosevelt 55-44 over Landon, Bush 56-44 over Dukakis, Bush-Kerry tied at 49% each. If I remember right, getting the result wrong with Truman prompted Gallup to examine their methods to correct their mistake, something that mhr could learn from.

http://www.gallup.com/poll/9442/Election-Polls-Accuracy-Record-Presidential-Elections.aspx

Posted by: tomeck on July 30, 2008 at 4:57 PM | PERMALINK

lurker, Moore probably did read that article, but since it's 60 years old it doesn't say anything about how pollsters are conducting themselves today. I'll be interested to find out more about the book.

Regarding Kevin's original puzzler, I suspect that it's a combination of option 3 and a lack of uniform methods for distinguishing between deeply held beliefs and off-the-cuff answers (and disagreement over how important it is). As Kenneth Almquist said, it's not simple to do, especially when, as in an election year, each outfit may be doing a lot of polling; they want to streamline the process as much as possible, because polling is expensive.

Posted by: editer on July 30, 2008 at 4:58 PM | PERMALINK

I seem to remember something about TV; people saying they watch the ads; some new fangled set top box saying they don't; TV execs then not liking the results one bit; so the company collecting the data stopped collecting that bit of data; so everybodys head is buried in the sand and liking it!

Posted by: landofid on July 30, 2008 at 5:31 PM | PERMALINK

As a researcher, you either confirm the common sense, in which case you wasted tax money on something everybody already knew, or you debunk the common sense, which only shows that you have no common sense and the university wasted tax money on you.

Posted by: ogmb on July 30, 2008 at 5:38 PM | PERMALINK

Rule of 4, change the equation, test what moves.

Posted by: anonymous on July 30, 2008 at 7:36 PM | PERMALINK

The point is obvious that people hold views at differnt levels of deepness, but so what? In election polls, it seems of little consequence and, if important, the pollster can poll how deeply the views are held.

We are way over polled and the media makes way too much of them. But we are stuck with them because they are cheap and easy "news" stories.

Posted by: brian on July 30, 2008 at 8:33 PM | PERMALINK

OK, then the next question is "how long have you supported/opposed whatever"? I would imagine, for presidents, that anything under a month is soft support, over a month is hard support.

Posted by: orion on July 30, 2008 at 11:36 PM | PERMALINK
It occurs to me that both kinds of folks get to vote ...

No doubt, which is why such a differentiation may matter little in interpretation of polls conducted immediately before elections. OTOH, in interpretation of the meaning of polls conducted long before an election, identifying those difference is important. And, in fact, some polls that I've seen (or, OTOH, been polled for) do as questions to identify these kinds of differences, though even when they do the relation of those to the support for particular candidates is never reported in mainstream media reports of the poll results, and I don't recall ever seeing a media-sponsored poll that asked that kind of question. The problem is, it would seem to me, more of a media problem than anything.

Posted by: cmdicely on July 31, 2008 at 6:47 PM | PERMALINK

Very nice site! cheap viagra

Posted by: John369 on June 19, 2009 at 4:26 PM | PERMALINK

Very nice site!

Posted by: John369 on June 19, 2009 at 4:27 PM | PERMALINK




 

 

Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM



buy from Amazon and
support the Monthly