Like many election watchers, I was shocked by the results of the 2016 U.S. presidential election. After reading article after article trumpeting Hilary Clinton’s lead in the polls, and reviewing reputable Election Day forecasts that proclaimed a 99 percent probability of a Clinton victory, who can fault such a reaction?
In the aftermath of the election, much blame has been cast on pollsters’ near-universal projections of a Clinton victory. Pundits have been quick to toll the polling industry’s death knell. Wading through the post-election polling discussion, I found myself wondering why so much attention was being paid to the polling methodology and so little to how polling results were communicated to the average voter. Pollsters — and the media that report on poll results — are misleading the public as to the certainty of their forecasts, resulting in a complacent and unprepared electorate.
A poll is intended to reflect the voting preferences of a representative sample of likely voters within a larger population. In a typical telephone poll, a pollster will canvass responses from 1,000 to 2,000 randomly selected households. The pollster then decides whether to adjust the weight of each response so that the sample is representative of likely voter demographics. These demographics — race, age, education, etc. — received substantial attention in the U.S. election coverage.
Can an electoral poll of 1,000 be used to predict the votes of over 100 million? Yes, but only with caution. To the pollsters’ and media’s credit, most published polls indicate which candidate is in the lead, by how much, and by what margin of error. For example, a poll might put Clinton in the lead by 4 points, with a margin of error of +/- 3 percent, 19 times out of 20.
The mystical “margin of error” is based on the confidence interval of a binomial probability distribution. Put simply, a margin of error of +/- 3 percent, 19 times out of 20, means that the pollsters are 95 percent confident that the candidate’s actual lead is within 3 percent of the lead reported in the poll.
Most national polls of the U.S. election reported margins of error of 2 to 4 percent. When polls are combined into a “poll of polls,” the margin of error decreases further, assuming the polls are independent. For instance, aggregating 20 national polls of 1,000 respondents each yields a best case margin of error of 0.7 percent for the U.S. electorate. With Clinton’s lead in the national polls averaging 3 to 4 percent on Election Day, the media were quick to anoint Clinton the next president, even factoring in the vagaries of the Electoral College.
However, the margins of error calculated by pollsters represent only a portion of the true error associated with polling. Pollsters exclusively quantify sampling error: the error from choosing a sample size that is smaller than the entire population.
Pollsters calculate margin of error as if they are flipping a fair coin 1,000 times and using the observed proportion of “Heads” (say, 53 percent) to predict the actual probability of a head (50 percent). But the U.S. population is not comprised of hundreds of millions of identical voters — the equivalent of fair coins. There is a great risk in projecting that the 1,000 respondents sampled are representative of the electorate, regardless of the demographic weightings used. Determining who is likely to vote, and who to stay home, is fraught with uncertainty. Adding undecided voters — the coins that land on their side when flipped — and the potential for respondent dishonesty into the mix strengthens the cocktail of unpredictability.
To use actuarial terms, the margin of error quantifies the “sampling error” portion of parameter risk: the risk that model parameters are not suited to the phenomenon being modeled, as described in Venter and Sahasrabuddhe’s “A Note on Parameter Risk” (2012). The margin of error does not consider data bias — the risk that experience period data cannot be used for predictions without first being adjusted — nor model risk — the risk that the selected model poorly represents the modeled phenomenon.
These additional sources of error are never included along with poll results. The net result is that the true margin of error of the polls is systematically under-reported.
How large is the additional, unreported error? Popular forecasting site FiveThirtyEight.com estimates that the average poll would still be off by 2 percent if no sampling error was present. In a less scientific assessment of the model error, the New York Times gave five well-regarded pollsters identical data from a poll of 867 likely voters in September 2016. After applying judgmental adjustments to the data, the pollsters’ forecasts ranged from a 4 percent Clinton lead to a 1 percent Trump lead.
Compounding this problem is “herding” — the practice of adjusting anomalous poll results to be more in line with those reported by other pollsters, or not publishing such polls altogether. Herding falsely increases the perceived certainty of a candidate’s lead and threatens polls’ independence. Evidence of herding has been found in several recent election polls.
As an actuary who makes a living from interpreting statistics, I understand the difficulty for even a trained professional to draw the correct conclusions from the data. We cannot expect the voting public to understand the real level of uncertainty underlying the polls without the help of pollsters and the media.
We need to convince pollsters to report the true margins of error of their polls, including error due to data bias and model risk. We need to encourage news media to spend less time talking about one candidate’s lead in the polls, and more about the uncertainty of the forecasts. We need to decrease focus on individual polls and to increase focus on estimates of a candidate’s victory probability accounting for all sources of polling error and poll dependence, such as that calculated by FiveThirtyEight.
Inflation of the certainty of poll forecasts contributes to voter complacency and leaves the general public unprepared for an upset on Election Day. Let’s guard against these consequences by equipping the average voter with an understanding of the uncertainty of election results.
Jeffrey Baer, FCAS, is manager, advanced analytics for Economical Insurance Group in Waterloo, Ontario, Canada. He is also a member of the AR Committee.
