The polls were wrong for the last two presidential elections. In 2012, poll aggregation (i.e., Nate Silver, AKA 538) got it right. Seeing a profit opportunity, a bunch of poll aggregators entered the market but this time they all got it wrong:
It’s a question that will be asked over and over in the coming days, weeks, months: How could so many pollsters and analytics experts get the election so wrong?
Sam Wang, the Princeton University neuroscientist hailed by Wired this week as the “new king of the presidential-election data mountain,” has been using probabilistic analysis to forecast the elections since 2004. Four years ago he was one of a group of quants who, as The Chronicle’s Tom Bartlett put it, “won the election.” This year his Princeton Election Consortium had given Mr. Trump’s Democratic opponent, Hillary Clinton, a 99-percent chance of winning in its final forecast, pegging her likely electoral votes at 323
The point isn’t to pick on Mr. Wang. An overwhelming majority of data-crunchers missed Mr. Trump’s ascension — including, it seems, those conducting internal polls for the Democratic and Republican parties. And college polling institutes like the one at Monmouth University — which earned a coveted A-plus rating from the data-reporting site FiveThirtyEight — will certainly be conducting lengthy post-mortems.
One scholar did expect a Trump victory: Allan J. Lichtman, a professor of history at American University, who, according to The Washington Post, has predicted three decades of presidential elections using a set of 13 true-or-false questions.
Does that mean that historical analysis outstrips quantitative work? Not necessarily. But this election campaign had already prompted political scientists and other scholars to rethink basic assumptions about political norms, strategies, and analytics. Now electoral forecasting will move to the top of that list, and scholars will have to play a major role in assessing the damage.
I can think of one technical reason the polls were wrong. The low response rate polls were subject to sample selection bias. Let's say that only 13% of the population is responds to the survey (13% is the response rate in the Elon University Poll). If the 80% that doesn't respond is similar except for observed characteristics (e.g., gender, age, race, political party) then you can weight the data to better reflect the population. But, if the 87% that doesn't respond is different on some unobservable characteristic (e.g., "lock her up") then weighting won't fix the problem. The researcher would need other information about nonrespondents to correct it (Whitehead, Groothuis and Blomquist, 1993). If you don't have the other information then the problem won't be understand until actual behavior is revealed.
Also, consider the headline from the Elon University Poll:
The race for the White House is essentially tied in North Carolina, with Clinton holding a lead of less than 1 percentage point. Among likely voters, Clinton has 42 percent of the vote while Trump has 41.2 percent, with 8.7 percent saying they are still undecided in the race.
Here are the results from the methods report (the brackets indicate that the order of the candidates was randomly assigned):
Incorporating the follow-up question the results would have been:
- Clinton - 309
- Trump - 310
I'm ignorant about the methods the poll aggregators use but, ignoring that this is a statistical tie, using the follow-up question would have made for a more accurate Elon University Poll headline for North Carolina. It could have read like this:
The race for the White House is essentially tied in North Carolina, with Clinton holding a lead of less than 1 percentage point. Among likely voters, Trump has 43.7 percent of the vote while Clinton has 43.5 percent, with 3.5 percent saying they are still undecided in the race.
Using either way of summarizing the votes, Elon got the North Carolina governor and senate races correct.
Note to self: Me and my co-authors have been getting a lot of don't know referendum voting responses ... why haven't we asked the "if you had to choose" follow-up question. Has anyone done that?
