In polls we trust

The big secret about polling companies, the thing you can’t know for sure until you’ve actually worked at one, is that they really, really want to get it right. Over the years and the course of some significant ballot upsets, they’ve lost our trust; during our conspiracy-prone times, some even accuse them of actively peddling misinformation.

The truth, though, is rather more mundane. Good quality opinion polls are the best — indeed, the only — real datapoint for quantitatively estimating the outcome of an election before it has happened. They’re obviously not perfect, but they’re the best information we have. Whatever you may hear to the contrary, the political betting markets are mainly a measure of the consensus view of politically engaged middle-aged blokes, who in turn calibrate their hunch against… opinion polls.

There are clear incentives for pollsters to get it right — but they aren’t stated often enough. For most of these companies, the political work is a loss-leader, a kind of shop window for how accurate they are so they can make their real money selling brand trackers to McDonalds or whatever. They join in at election time to prove that their method works, that the numbers they give you about your business demonstrably correspond to what the population really thinks. As far as evil masterplans go, pushing an agenda by putting out dodgy election poll numbers is a terrible one — because unlike the pundits, who can couch their analysis in noncommittal theoreticals, you will be found out on election day.

I’ve been closely involved in pollster world for most of the past ten years and have developed a sort of broad brush heuristic for thinking about the polling evidence. First, only pay attention to the good quality pollsters — in most elections, there’s usually a surprising “random pollster that called it right last time” but more often than not they don’t pull off the same trick twice. Second, don’t bet the house on differences of one or two percentage points — presume that even the best pollsters will be a little bit wrong. Third, pay particular attention to the in-house pollster “MRP” models, of which the most robust is from YouGov (where I used to work).

Once you’ve got the big picture of what the data is saying, compare it with the media narrative. This, in betting terms, is where the “value” lies. Given that the polls won’t be exactly right, does the consensus seem overly invested in small poll differences, or insufficiently convinced by large ones? In the turbulent past five years of politics, there have been upsets in both of these directions.

Further reading
What's your vote really worth?

By Tom Chivers

In the UK, the General Election of 2015 was a gruesome event for most pollsters – most had it near-even between Ed Miliband’s Labour and David Cameron’s Tories, but Cameron went on to win an overall majority, thanks to the Labour collapse in Scotland and the Lib Dem collapse in England. It was, at the time, the most-polled election in history, and the media were furious they had given the numbers so much air-time and had missed the real story.

So when the Brexit referendum came around in May 2016, they were determined not to make the same mistake twice. Despite many polls showing a very close race, there was a deeply-held superstition, sustained by the betting markets, financial markets and media, that people would ‘see sense’ at the last moment and vote to Remain (as they had in the Scottish referendum in 2014). In the run up to the Brexit vote I wrote with frustration in The Times, the Guardian and Newsweek about this mysterious overconfidence, and how it wasn’t borne out by the data.

From the same author
Is Donald Trump toast?

By Freddie Sayers

The Trump victory over Hillary Clinton later that year was a strange combination of both effects. The polls were slightly wrong, and some of the state polls very wrong, but overall they showed a close race. Had we purely gone on the data, with the appropriate humility that with such a novel candidate attracting a novel electorate the polls would likely be additionally off, the possibility of a narrow Trump victory across the battleground states should not have seemed remote at all.

Donald Trump is not a novel candidate any more, he’s the President; and there’s certainly nothing novel about Joe Biden. It was a volatile campaign in 2016, with a high number of undecideds and high third party support; 2020 is none of these things. The main technical error in 2016 — higher turnout among demographic groups that don’t normally vote in such number — has now been corrected by most pollsters. The much-quoted ‘shy Trump’ phenomenon remains unproven, and when we spoke to Robert Cahaly, its main advocate, he couldn’t explain convincingly how it could be measured. For it to be a serious effect, you would have expected Trump in 2016 to have overperformed in Democratic areas where it was most shameful to support him, but that is not what happened. Plus, the midterm polls in 2018 were already demonstrably better.

Since the pollsters are collectively terrified of repeating the errors of last presidential cycle, I wouldn’t be surprised if they had over-compensated in their adjustments and we end up finding they have been over-estimating the Trump vote. Nobody wants to be the pollster predicting a massive Biden landslide right now: if there’s any “adjustment of the weights” going on it is more likely to be softening the Biden lead into safer single digits.

Further reading
Of course Trump can win

By Matthew Goodwin

One clear indication of this reverse effect would be if Texas goes blue. Joe Biden doesn’t need to win it, but the polls are tantalisingly close, though moving slightly in Trump’s favour in the final week. In Democrat circles the possibility of winning Texas is mentioned with a whisper, as if to speak it out loud will jinx the possibility. It feels unlikely, like it somehow can’t happen, but the data says it is possible. This would put Biden in historic landslide territory.

More broadly, Biden’s advantage in the polls on election day – a 7-9 point national lead, and apparently ahead in almost all of the battleground states — equates to a much more comfortable margin than Hillary Clinton had. Biden has roughly double the lead in the top battlegrounds, double the national lead and double the favourability advantage of Clinton in 2016. If the same degree of polling error that afflicted 2016 was applied here, Biden would still win comfortably.

All of this is why I believe it is reasonable to expect that Joe Biden will win the election by a comfortable margin, possibly a very big one. Put another way: if Donald Trump does pull off another surprise and wins re-election, fair and square, it would be an astonishing event. It would mean that the mechanism we have relied on for assessing public opinion for decades — a key part of the way governments decide on public policy, most recently on Covid-19 — is fundamentally broken. It would mean we have to rethink the basic epistemic foundation of our public conversation. It would be scintillating — so much so that I wouldn’t give a moment’s thought to those bets I placed on a big Biden victory. There’d be far more important things to worry about.
view 36 comments