Political Animal


October 03, 2012 12:58 PM No Such Thing As a “Statistical Tie”

By Ed Kilgore

Now that a couple of national polls have come out showing Romney moving a bit closer to Obama, we’re beginning to hear that ol’ chestnut: “The presidential race is in a statistical tie.” As it happens, the best brief demolition of that claim, and the best brief description of what “margin of error” means in polls, was done right here at PA by Kevin Drum in 2008. Here’s a snippet:

I originally wrote about this back in 2004, but here it is again. The idea of a “statistical tie” is based on the theory that (a) statistical results are credible only if they are at least 95% certain to be accurate, and (b) any lead less than the MOE is less than 95% certain.
There are two problems with this: first, 95% is not some kind of magic cutoff point, and second, the idea that the MOE represents 95% certainty is wrong anyway. A poll’s MOE does represent a 95% confidence interval for each individual’s percentage, but it doesn’t represent a 95% confidence for the difference between the two, and that’s what we’re really interested in.
In fact, what we’re really interested in is the probability that the difference is greater than zero — in other words, that one candidate is genuinely ahead of the other. But this probability isn’t a cutoff, it’s a continuum: the bigger the lead, the more likely that someone is ahead and that the result isn’t just a polling fluke. So instead of lazily reporting any result within the MOE as a “tie,” which is statistically wrong anyway, it would be more informative to just go ahead and tell us how probable it is that a candidate is really ahead.

By this standard, horse-race results within the MoE aren’t “ties,” but simply represent a reduced probability of the lead reported in the top-line results. While it’s possible Romney is actually at parity with Obama, it’s very unlikely, particularly since the Republican has been stuck at 47% or below throughout the entire cycle. It’s also worth noting that the MoE tends to be larger in state polls, since samples are typically smaller, but the same cautions must be expressed in talking about “statistical ties.”

It’s amusing in any event to see some pro-Romney gabbers seamlessly shifting from claims that all the polls are “skewed” to crowing over any relatively positive numbers that happen to come out, but I guess they are “arguing in the alternative.” Any evidence that the Great National Repudiation of Barack Obama is not just on the horizon is psychologically intolerable.

Ed Kilgore is a contributing writer to the Washington Monthly. He is managing editor for The Democratic Strategist and a senior fellow at the Progressive Policy Institute. Find him on Twitter: @ed_kilgore.


  • POed Lib on October 03, 2012 1:11 PM:

    Claimer: I am a biostatistician.

    While any poll in which the difference is less than the MOE is not conclusive AT THAT TIME, it is also important to remember that a MOE has 2 sides - the difference could be LARGER and will be so 50% of the time - there is a sampling distribution about the actual proportion, and the values can be above or below the point estimate. So when you say "they could be tied", you should add "or Obama could be walloping the tar out of him." Both are somewhat equivalently possible.

    However, the most important things are 1) the trend over several samples/polls and 2) the state-specific results. Here we see much more conclusive results in that many many polls agree that Obama IS walloping the tar out of Mittens. And the state results mirror the national.

    I think that Obama is ahead by 3-5 pts at this time.

  • martin on October 03, 2012 1:27 PM:

    My favorite statistical pet peeved and I haven't whined about it in awhile, so thanks for bringing it up. Now if only we could find a way to kill every reporter and pundit who says things like "the poll results are within the margin of error."

    They shouldn't be reporting on all of these polls anyway, but they think numbers make them look smart and informed. Just like report the daily Dow Jones number is supposed to say something about the economy.

    Lazy illiterates.

  • Matt McIrvin on October 03, 2012 1:32 PM:

    There's an even more fundamental error, which is that the sample-size MOE for an aggregate of many polls is much smaller than the MOE for each individual poll. So if you see Obama ahead of Romney by a margin smaller than the MOE, but he's still ahead in 9 out of 10 polls, that's a sign that something real is probably going on.

    Or that there's some systematic error biasing all of the polls (the great poll-truther hope). But systematic error isn't what MOE is about in the first place; it's about random sampling error due entirely to chance. (And anyone asserting systematic bias has to explain why such a huge bias didn't appear in 2004 or 2008.)

    So insisting on the MOE as the threshold for significance of a poll aggregate is, in a sense, saying both too little and too much.

  • BillFromPA on October 03, 2012 1:46 PM:

    This is another blatant effort on the part of the MSM to make this seem like a tight horserace, without which they cannot sell newspapers and ad space and time.

  • bdop4 on October 03, 2012 1:50 PM:

    Obama has been holding this lead for weeks now, with dozens of polls confirming a small but visible margin. Thousands are polled each week, yet there has been no meaningful movement in the dynamic.

    If the situation is more fluid as conservatives suggest, wouldn't there be more fluctuation from week to week, poll to poll?

  • Daddy Love on October 03, 2012 2:00 PM:

    A "statistical tie" is really the same as a "tie;" That is, the candidates are at the SAME percentage. The MOE means that the percentage could be wrong. But because you don't know that it is (but you have 95% odds that it is), a tie is a tie is a tie.

    ANd a trend is a trend, and we know where the trend is, and so do the GOP.

  • James E. Powell on October 03, 2012 2:40 PM:

    This is one of those constant things said by talking heads that will drive you crazy if you know what they are talking about, but you will likely accept if you do not know what they are talking about. It is not the only example.

    But this one is so common that I have to believe that several people have explained statistics to news directors, editors, and reporters several times. So it can't be that the press/media people don't know that what they are saying is inaccurate. Can it?

  • martin on October 03, 2012 2:55 PM:

    So it can't be that the press/media people don't know that what they are saying is inaccurate. Can it?

    Yes it can. Or, more, likely, the information just bounces around in their head awhile, gets lonely and leaves.

    It is like reporting that some bill "did not get the 60 votes needed to pass" when only 50 votes are needed to pass, it takes 60 to break a filibuster. I've pestered NPR plenty of times about this. They will do it right for a week or two and then revert back to their 60 vote version. And I'm sure other news organizations have been told the same thing. The filibuster is a lot less complicated than MOE, yet they can't retain such a simple rule.

  • Keith Roberts on October 03, 2012 3:28 PM:

    What the polls don't account for, of course, is voter exclusion. If the Republicans can, by hook or crook, keep a million or more Obama voters from reaching the polls or having their votes counted (which seems to be their goal, and may be within their reach), then I fear that anything that puts Romney within low single digit reach of Obama indicates that he will win.

  • Tom Dibble on October 03, 2012 4:54 PM:

    Yes, it is much better to speak in terms of "Poll X shows that Obama is y% likely to be preferred by Americans over Romney", where "y%" tends towards 95% or greater when the poll results are outside the "MoE" range.

    The problem is, this number is very hard to calculate, because we are comparing two related-but-dependant variables. We can tell what the likelihood is for one or the other number to be more than a certain point away from the measured value, but it is hard to determine how likely one of the numbers is to be more than a certain point away AND the other simultaneously to be more than another point away. To calculate that number you need to know how the variables interrelate; if the Romney number was undermeasured by 1%, does that mean the Obama number was overmeasured by 1%? If they are completely unrelated (an undermeasure of Romney by 1% is completely unrelated to the Obama number) it's straightforward - these are then just two independent poll results, and the likelihood of the actual number of one being above the other is easily calculated. If they are directly related the calculation is a little more complex but also possible. However, if they are partially related, as they tend to be, then it is hard to know how to calculate likelihoods.

    In any case, I strongly prefer news outlets wrongly claiming a "statistical tie" over them obsessing over every half-point shift from one poll to the next.