There were about ten minutes Tuesday night when I was seriously excited.
Like every other excited election junkie, I was ready to track a potentially close Wisconsin recall election as indicated by the exit polls. But my hopes were soon dashed when the actual votes were counted.
The exit polls results were quickly mocked for their inaccuracy. How could the exits be so wrong, people wondered?
There were about 100 tweets in my Twitter feeds from very smart people wondering why the networks, specifically CNN, had been touting such bad information. The fact that Wolf Blitzer's reporting on the exits gave way to Piers Morgan's not-quite-hilariously awful jubilee coverage only added to the misery.
But the question I have is: where have all of these people who think that exit polls are accurate measures been the last six months, let alone 20 years?
On any number of occasions, exit polls underestimated Rick Santorum's vote in this 2012 Republican primary season. They predicted him to finish third in Iowa and Mississippi, for example – states where he actually won.
Exit polls projected John Kerry to win the 2004 election. You may have heard he lost. The bias in his direction was about 2.5 percentage points. Thus, a 51%-48% Kerry victory became a 50.7%-48.3% Bush triumph. Anyone remember 2000 and Al Gore in Florida? The same 2.5-point Democratic bias occurred in 1992, but Bill Clinton easily took the election, so no one actually cared. The list goes on and on.
People should know better than to believe that early exit polls are more accurate than pre-election polls.
You might wonder why exits get the results so wrong. That's a much better question.
Considering most of you don't want to get too much into the weeds, here's a briefer.
1. Exit polls are not designed to predict a winner. Why would you waste your time trying to figure out who will win at 9pm, when the actual results can usually tell you who won within an hour or two? This is in marked contrast to a country like Great Britain, where polls close at 10pm and results are counted by hand. If it weren't for exit polls, Brits wouldn't know who had won until well into the morning hours. It's also quite different from a country like Mexico, where exit polls are used to confirm election results in a country where government opponents long suspected officials of election fraud.
2. Exit polls are meant to tell us why voters voted the way they did. That means longer questionnaires are necessary than in the UK. Questionnaire length tends to be negatively correlated with response rates. That in itself is not a problem, but it can be if it leads to differing response rates between certain segments of the population. Say you are somebody really perturbed by Scott Walker, you might be more willing to take a longer exit poll in order to display that anger. You can try to control for this problem by noting the basic demographic facts of non-respondents, but you can't possibly know someone's education or income level simply by looking at them.
3. A perfect exit poll sample may not be "perfect". My friends at the NBC Political Unit make a simple point: you may not have non-response, but you still don't know how a sample precinct represents the state (see point 7). Exit pollsters aim for a precinct to fit a model, but it may not. Errors could come from improperly calculating how the votes of 37-year-old white males in precinct A correlate with the votes of 37-year-old white males in precinct B, or it could be underestimating the percentage of the state-wide vote coming from precinct A.
4. The initial Wisconsin exit poll error was compounded because absentee voters were not included. Absentee voters made up 15% of last night's electorate. That's far higher than the 10% they did in 2010. Absentee voters are historically more Republican, and they clearly were last night. That, in combination with a greater percentage of the vote being cast via absentees than usual, caused a tilt in the exit poll result towards Democrat Tom Barrett.
5. Exit polls don't make assumptions about the electorate. Exit polls may have historically missed male voters (who were very pro-Republican in the Wisconsin recall) and old voters, but how do you know this will occur in the future? Perhaps both of these groups will turn out differently than before. The 2004 electorate was more female and white than the 2008 electorate. Had someone weighted the 2008 electorate by 2004's, they would have produced a weighted sample that was weighted with the wrong assumptions. We know that weighting to prior absentee percentages in Wisconsin would have been wrong.
6. Only after we start getting the results can we properly weight the raw exit poll data. Even so …
7. The error of the whole is greater than the parts. US exit polls are assembled using a mathematical technique known as cluster sampling. The basic idea is that you take a sample of random precincts (eg, 35 in Wisconsin last night) supposedly representative of the state and add up the precincts to form a state estimate. Cluster sampling helps to compensate for the inability to sample every precinct in the state, but the errors of each precinct add together to form a larger state error. This error can be up to two or three times what you might expect from a pre-election telephone poll. That's why the final (not preliminary) exit polls in Wisconsin had a theoretical margin of error of ±4 points, while a telephone poll of the same sample size would have a margin of error of only about ±2 points. The preliminary exit polls in Wisconsin that were not weighted to the results had additional error because of the complications listed above.
The simple truth is exit polls are not perfect. No one who knows anything should claim otherwise. They have margins of error like any other poll. They aren't designed for prediction. They can, however, be great tools for understanding the electorate's behavior when utilized correctly.