Political Animal

Blog

December 27, 2012 10:16 AM Pollsters’ Regrets

By Ed Kilgore

If you spent a tenth the time I did staring at and trying to understand polls during the 2012 election cycle, then you owe it to yourself to read Steven Shepard’s National Journal piece on the post-election debate over polls and poll aggregation.

The big data point everyone’s trying to digest is that so many pollsters—particularly Big Dogs like Gallup—missed or underestimated Barack Obama’s eventual four-point popular vote margin. And that in turn has the polling industry (notably those—again like Gallup—who conduct expensive live interviews) a-fearing that their media consumers will increasingly turn to aggregators who simply average and at most massage polling data generated by others.

Shephard offers a useful brief history of the aggregators:

Real Clear Politics began the practice of averaging polls before the 2002 midterm elections. RCP was joined by Pollster.com—which is now part of The Huffington Post—four years later. “Pollster started in 2006, and we were really building on what Real Clear Politics did,” founding Coeditor Mark Blumenthal said. The statistician Nate Silver began a similar practice in 2008, and his site, FiveThirtyEight, was acquired by The New York Times shortly thereafter. More recently, the left-leaning website Talking Points Memo started its PollTracker website before the 2012 election.
Each of these organizations differ in their approaches. Real Clear Politics does a more straightforward averaging of the most recent polls. TPM’s PollTracker is an aggregation involving regression analysis that uses the most recent polls to project a trajectory for the race. FiveThirtyEight and HuffPost Pollster use polls, adjusting them for house effects—the degree to which a survey house’s polls lean consistently in one direction or another. FiveThirtyEight also uses non-survey data to project the election results.
All four of these outlets underestimated Obama’s margin of victory. Both Real Clear Politics and PollTracker had Obama ahead by only 0.7 percentage points in their final measurements. HuffPost Pollster had Obama leading by 1.5 points, while FiveThirtyEight was closest, showing Obama 2.5 points ahead of Romney in the last estimate. The aggregators that came closest to Obama’s overall winning margin were the ones that attempted to account for pollsters’ house effects.

Aside from “house effects,” aggregators are also vulnerable to distortions created by pollsters who are especially active in any particular election:

Part of that problem, at least when it comes to the national presidential race, were the daily tracking polls from Gallup and automated pollster Rasmussen Reports. Both firms reported results that were biased in favor of Romney this cycle, but by publishing a new result every day, their polls could be overrepresented in the averages. “The one sort of Achilles’ heel of the regression trend line that we’ve done classically on our charts, there are two pollsters that contribute most of the data points,” said Pollster’s Blumenthal. “Not only does that make the overall aggregate off, it can also create apparent turns in the trend line that are [because] we’ve had nothing but Gallup and Rasmussen polls for the last 10 days.”

Gallup and Rasmussen, of course, were the two big polling outfits whose final surveys projected a Romney win. But it’s not as though you can necessarily just pick a “good” pollster and stick with its results: one of those which seemed to nail the final results, Pew Research, was all over the place late in the campaign, producing real panic among Democrats with an October poll showing Romney up by four points. So for the time being, the aggregators would seem to be the most reliable barometers of where a given contest is and may be headed.

Now some people will read this or read Shepard’s piece and say: “Screw the polls.” That’s an understandable attitude, but not particularly helpful; candidates and elected officials are going to look at polling data even if you and I don’t, so we might as well keep up. More generally, the answer to bad or questionable data is more and better data, not less. So as the next election cycle takes shape, perhaps we will see a gradual evolution towards better understanding of polls so voters are less inclined to be fooled by idiot newsreaders or hackish spinners touting a particular survey as definitive.

Ed Kilgore is a contributing writer to the Washington Monthly. He is managing editor for The Democratic Strategist and a senior fellow at the Progressive Policy Institute. Find him on Twitter: @ed_kilgore.

Comments

  • c u n d gulag on December 27, 2012 10:34 AM:

    I've written about this before here, so I'll be bried (or, as brief as I can be).

    I wish polling was done only for the politicians, and the results were not released to the MSM. Or, only released once a month - until the last month, when they would again return to 'internal use only.'

    All the MSM has covered in the last few national elections, is the polling data, and not the positions of the politicians running for office.

    It's as if, at a racetrack, all you could see were the latest odds.
    No information on the horses prior races, or what the parentage was - or was he/she a good "mudder," or only did well on dry tracks?

    On a rainy day, you'd almost have to be an idiot to take the horse with the worst odds - but, had you known that this horse did great on muddy tracks in prior races, while the rest of the horses didn't, then you might think he/she was worthy of at least a small bet.

    Most times, the odds don't make the horse. The horse makes the odds.

    And our MSM, because it only looked at the odds, was unable to see that Mitt had problems with women, single women especially, and that Obama did well with single women - AND "mudders."

  • boatboy_srq on December 27, 2012 10:51 AM:

    Now some people will read this or read Shepard’s piece and say: “Screw the polls.”

    Isn't this what Multiple Position Mitt's people - indeed, nearly the entire Teahad - did with the last election: plug their collective ears whenever negative poll results came in, and coccoon themselves in la-la-la (I can't hear you) land to convince themselves of the Rightness of their Cause and the Inevitability of their Victory?

    Using more data points is a better method than using fewer ones (or none at all): it makes the aberrations easier to wash out, and it makes for more rational handling of the issue/campaign/whatever. Using less information - or less "outside" information - inevitably skews not only one's approach, but one's entire viewpoint. One look at the near-meltdown amongst the Teahad after their Assured Triumph is enough to prove that.

  • ieiueous on December 27, 2012 11:12 AM:

    Hello!!! everybody, Fashion,low price,the good shopping place, click in.

    ===== http://www.gocn2trade.com =====

    Discount Air jordan(1-24)shoes $35,

    Air max shoes (TN LTD BW 90 180) $36,

    Nike/shox(R4,NZ,OZ,TL1,TL2,TL3) $35,

    handbags(Coach lv fendi DG) $36,

    Tshirts (Polo ,ed hardy,lacoste) $20,

    Jean(True Religion,ed hardy,coogi) $36,

    Sunglasses(Oakey,coach,Armaini) $16,

    Watches(Rolex BREITLING IWC),

    New era cap $12,

    Discount (NFL MLB NBA NHL) jerseys,

    free shipping,

    Accept credit card payment!

    ====== http://www.gocn2trade.com =====

  • Crusty the Ex-clown on December 27, 2012 11:18 AM:


    Read "The Signal and the Noise", wherein Silver offers a meta-view which is well in line with my own experience in scientific data analysis. I will hazard a guess that he's also correct on political polling inasmuch as the same rules of mathematics and psychology should apply there as well.

  • stormskies on December 27, 2012 11:18 AM:

    The reality was and is very simple about the corporate polls: they were purposefully structured to weigh in the favor of buffoon and pathological liar Romney. Purposefully.

    For over a year before the election Obama held wide leads among the Hispanics, the Afro-Americans, the Asians, the Native Americans, the vast majority of women, the young, and the educated Caucasian people.

    The only group that Romney consistently lead was the uneducated white people which of course speaks for itself. This cretins consistently vote against their own self interests because it is more important to them to sustain their delusional reality, ideology, that of course the pathological liar played right into.

    These are simple facts. And those facts demonstrated for over a year that Obama was well ahead of Romney. The Gallup poll was one of the worst offenders yet the corporate media relied upon them the most even though they were the worst prior to the election: getting it the most wrong even in the election between Obama and McCain.

    The fact that they corporate media relied upon them, given their track record, also speaks for itself in terms of whom the corporations really wanted to be president: not Obama.

    Thus, they purposefully skewed their methodologies om these polls in the hope that the voting public would 'believe' them. And, in so believing, want to vote for a 'winner'. That's the evil game that the corporations played.

    And despite that game Obama won as most of us knew on this blog by the margin that he did long before the actual election.

    Even now NBC has invented a new poll claiming that the country is now saying that both parties are equally responsible for the 'fiscal cliff' problem The facts are, as ever, very different than what NBC wants America to 'believe'.

  • ComradeAnon on December 27, 2012 11:28 AM:

    Polling has been around a while. If they wanted accurate polling, they go after it. If they wanted fact based news, they'd go after it. Polling doesn't change the outcome. It just delays the day of reckoning. And as soon as the truth comes out, it's time to start the blaming everybody but yourself all over again.

  • dweb on December 27, 2012 11:36 AM:

    One poll commonly queries maybe 400-1,000. An aggregator can take many polls and come up with a sample size of thousands and as any statistician will tell you, that tends to smooth out the outliers. Besides, good aggregators, over time, figure out tweaks to their algorithms which take into account the fact that Rasmussen is a tool and Gallup not much better.

  • New Orleans on December 27, 2012 12:11 PM:

    Ed Kilgore writes: "Now some people will ... say: 'Screw the polls.' That’s an understandable attitude, but not particularly helpful; candidates and elected officials are going to look at polling data even if you and I don’t, so we might as well keep up."

    You seem to suggest that the polls are needed for me to know how I should vote. I disagree. My goal when voting is to support the candidate who best represents my beliefs and interests.

    The problem is that the media share your apparent opinion, which is why they bombard us with horse-race dribble instead of offering substantive coverage.

    Yes, screw the polls.

  • dj spellchecka on December 27, 2012 12:23 PM:

    the national journal article leaves out the most important reason both gallup and ras were off this cycle...they massaged their numbers based on an expected voter sample that badly underestimated minority voters...

    kind of a major omission

  • low-tech cyclist on December 27, 2012 1:42 PM:

    'The one sort of Achilles’ heel of the regression trend line that we’ve done classically on our charts, there are two pollsters that contribute most of the data points," said Pollster’s Blumenthal. "Not only does that make the overall aggregate off, it can also create apparent turns in the trend line that are [because] we’ve had nothing but Gallup and Rasmussen polls for the last 10 days."

    "it’s not as though you can necessarily just pick a “good” pollster and stick with its results," says Ed, and that's true. But ISTM that the obvious solution to a particular pollster being way overrepresented by an aggregator is for the aggregator to adjust the tracking polls' weights downward so that they have about the same representation in the average as the most prolific non-tracking polls. That shouldn't be too hard.

  • Doug on December 27, 2012 3:19 PM:

    Why would ANY reputable news organization broadcast the results of a poll without knowing about the methodology and any possible "massaging" used by the polling organization? I'm not a pollster, nor have I studied statistics, but even I know that polling results for a Presidential election can't realistically be derived if the voter model being used is based on off-year elections; many who vote in the former, don't during the latter.
    Also, polls, so I understand, represent a "snapshot in time" of views being held by people. Never forget: snapshots CAN fade...

  • Mitt's Magic Underpants on December 27, 2012 6:15 PM:

    The MSM needed it to be an exciting, easy-to-cover horserace. They did everything they could to make it so.