Thursday, February 11, 2016

Understanding "margin of error" on Opinion Polls

Over the weekend, I saw a somewhat frustrating comment on Facebook.  Irritation at Facebook happens all the time for me actually, but this time it had a statistical polling slant.  Here's what was said:
Bernie is only behind 3%, and the margin of error on the poll is 4%!  It's a statistical dead heat, he's totally going to win!
I'm generally a pretty calm person, but not when people use the term "statistical dead heat." (or the word impactful.  Or the word guesstimate.  Or the word mathemagician. Or the term "correlation is not causation...")  But anyways..

There are a couple of issues with the Facebook poster's logic, but underlying it all is a general misunderstanding of what that margin of error means.  This blog seems like an ideal place to explore what polling margin of error actually is, and how we should interpret it.

"MARGIN OF ERROR"

What people are actually talking about when they talk about "Margin of Error" is the statistical concept of "sampling error."  Sampling error is a bit difficult to explain, but it's effectively this:
The error that arises from trying to determine the attributes of a Population (all Americans) by talking to only a sample (1000 Americans).
That's pretty straight forward, but many people still misunderstand this, here are a few detailed points:
• Sampling error doesn't include sampling bias: The +/- 3% that you see on most opinion polls represents only the bias due to looking at a number smaller than the entire population.  That means it has an intrinsic assumption that the sample was randomly and appropriately selected.  There's an additional issue called sampling bias (in essence the group from which the random sub-sample was selected, systematically excluded certain groups).  An example of this is that if we sample randomly from the phone book, we would exclude a large number of millennials who only have cell phones only, and thus our sample would be biased.  Error arising from sampling bias occurs above and beyond the +/- 3% of sampling error.
• Sampling error doesn't include poor survey methodology: Another reason that polls can be incorrect is poor survey methodology, which is once again not included in the +/-3%. A few ways that poor survey methodology can contribute to additional errors:
• Poor Screening: Most opinion polls involve reducing population to "likely" voters by asking screening questions.  If these screening questions work poorly, or incentives are provided, the poll results will not accurately reflect the population of likely voters.
• Poor Question Methodology: Asking questions in ways that make it more likely for a voter to answer one way or another can also create additional error.  This is especially true in non-candidate questions where it may be shameful or embarrassing to hold one opinion or another, making the words used in the question very important.  Another poor question example: questions that contain unclear language (e.g. double negatives) or are long and winding may confuse voters.
This may all seem like overly detailed statistics information, but in reality, those other forms of bias are alive and well in the primary polling system.  If those other errors were not occuring: 1. most polls would essentially agree with each other (they don't) and 2. polls would be extremely predictive of actual outcomes (not really true either).  A statistic from the Iowa Caucuses:

The last seven polls leading into the Iowa Caucuses gave Trump an average of 4.7% lead with a margin of error on each poll being 4%.   Trump lost the Iowa Caucuses by 4%, a swing of 8.7%.

STATISTICS

Let's pretend that we ran the perfect poll with a perfect sample and perfect questions, then how do we calculate an accurate margin of error?  The statistics side of opinion polling is actually a bit boring.  Calculating a margin of error on opinion poll is generally done using what's called a binomial confidence interval.  That calculation is relatively (in stats terms) simple, and only uses the sample size, the proportion of votes a candidate is receiving, and a measure of confidence (e.g. we want to be 95% sure the value will fall within +/-4%).   Here's the normal calculator:

That calculator is great, but if you play around with it a little, or if you tend to do derivatives in your head of any equation you see (ahem),  you realize something:  That +/- 3% that you see on opinion polls is completely bogus.  That's because the margin of error varies significantly by what percentage a candidate is receiving, and generally that 3% is only valid for a candidate currently standing at 50%.  The margins of error compress as a candidates share of the vote approaches 0% or 100%.  So for a candidate like Rick Santorum, habitually at 1-2%, we aren't at +/-3%, we're actually at +/- 1%.  Here's a graph showing how that compression at the margins works:

A quick note on this statistical calculation: Our Facebook poster from earlier said that the race was a "statistical dead heat" due to the margin of error.  In a perfect poll, that's not true, especially with a 3% lead in a 4% margin of error poll.  The 4% margin is calculated at 95% confidence, but at 3% we're 85% certain that Clinton is leading.  85% certainty of a Clinton lead is not exactly a "dead heat."

And just to show what horrible people statisticians are, I want to point out one last thing.  You know how I told you how easy it is to calculate the margin of error?  That's still true, but know that arguing statisticians have created eleven total ways to calculate that statistic, all of which create nearly identical results.  They also regularly argue about the appropriateness of these methods.  No joke.

Here's a demonstration of the similarity of the methods, at 50% and 1.5% on a 1000 person sample.

CONCLUSION

A few takeaway points from our look at margin of error:
• The +/-3% on most opinion polls don't account for all the types of error a poll could have.  In fact, it seems likely that other forms of error are pushing polling error upwards in modern American political polling.
• The margin of error stated on opinion polls is valid for a candidate receiving 50% of the vote.  It compresses at very high and very low vote shares.
• Statisticians are nerds who use 11 distinct ways to get to essentially the same results.