Do the polls know who will make America great again? Credit: Brendan Smialowski / AFP / Getty
The big secret about polling companies, the thing you can’t know for sure until you’ve actually worked at one, is that they really, really want to get it right. Over the years and the course of some significant ballot upsets, they’ve lost our trust; during our conspiracy-prone times, some even accuse them of actively peddling misinformation.
The truth, though, is rather more mundane. Good quality opinion polls are the best — indeed, the only — real datapoint for quantitatively estimating the outcome of an election before it has happened. They’re obviously not perfect, but they’re the best information we have. Whatever you may hear to the contrary, the political betting markets are mainly a measure of the consensus view of politically engaged middle-aged blokes, who in turn calibrate their hunch against… opinion polls.
There are clear incentives for pollsters to get it right — but they aren’t stated often enough. For most of these companies, the political work is a loss-leader, a kind of shop window for how accurate they are so they can make their real money selling brand trackers to McDonalds or whatever. They join in at election time to prove that their method works, that the numbers they give you about your business demonstrably correspond to what the population really thinks. As far as evil masterplans go, pushing an agenda by putting out dodgy election poll numbers is a terrible one — because unlike the pundits, who can couch their analysis in noncommittal theoreticals, you will be found out on election day.
I’ve been closely involved in pollster world for most of the past ten years and have developed a sort of broad brush heuristic for thinking about the polling evidence. First, only pay attention to the good quality pollsters — in most elections, there’s usually a surprising “random pollster that called it right last time” but more often than not they don’t pull off the same trick twice. Second, don’t bet the house on differences of one or two percentage points — presume that even the best pollsters will be a little bit wrong. Third, pay particular attention to the in-house pollster “MRP” models, of which the most robust is from YouGov (where I used to work).
Once you’ve got the big picture of what the data is saying, compare it with the media narrative. This, in betting terms, is where the “value” lies. Given that the polls won’t be exactly right, does the consensus seem overly invested in small poll differences, or insufficiently convinced by large ones? In the turbulent past five years of politics, there have been upsets in both of these directions.
In the UK, the General Election of 2015 was a gruesome event for most pollsters – most had it near-even between Ed Miliband’s Labour and David Cameron’s Tories, but Cameron went on to win an overall majority, thanks to the Labour collapse in Scotland and the Lib Dem collapse in England. It was, at the time, the most-polled election in history, and the media were furious they had given the numbers so much air-time and had missed the real story.
So when the Brexit referendum came around in May 2016, they were determined not to make the same mistake twice. Despite many polls showing a very close race, there was a deeply-held superstition, sustained by the betting markets, financial markets and media, that people would ‘see sense’ at the last moment and vote to Remain (as they had in the Scottish referendum in 2014). In the run up to the Brexit vote I wrote with frustration in The Times, the Guardian and Newsweek about this mysterious overconfidence, and how it wasn’t borne out by the data.
The Trump victory over Hillary Clinton later that year was a strange combination of both effects. The polls were slightly wrong, and some of the state polls very wrong, but overall they showed a close race. Had we purely gone on the data, with the appropriate humility that with such a novel candidate attracting a novel electorate the polls would likely be additionally off, the possibility of a narrow Trump victory across the battleground states should not have seemed remote at all.
Donald Trump is not a novel candidate any more, he’s the President; and there’s certainly nothing novel about Joe Biden. It was a volatile campaign in 2016, with a high number of undecideds and high third party support; 2020 is none of these things. The main technical error in 2016 — higher turnout among demographic groups that don’t normally vote in such number — has now been corrected by most pollsters. The much-quoted ‘shy Trump’ phenomenon remains unproven, and when we spoke to Robert Cahaly, its main advocate, he couldn’t explain convincingly how it could be measured. For it to be a serious effect, you would have expected Trump in 2016 to have overperformed in Democratic areas where it was most shameful to support him, but that is not what happened. Plus, the midterm polls in 2018 were already demonstrably better.
Since the pollsters are collectively terrified of repeating the errors of last presidential cycle, I wouldn’t be surprised if they had over-compensated in their adjustments and we end up finding they have been over-estimating the Trump vote. Nobody wants to be the pollster predicting a massive Biden landslide right now: if there’s any “adjustment of the weights” going on it is more likely to be softening the Biden lead into safer single digits.
One clear indication of this reverse effect would be if Texas goes blue. Joe Biden doesn’t need to win it, but the polls are tantalisingly close, though moving slightly in Trump’s favour in the final week. In Democrat circles the possibility of winning Texas is mentioned with a whisper, as if to speak it out loud will jinx the possibility. It feels unlikely, like it somehow can’t happen, but the data says it is possible. This would put Biden in historic landslide territory.
More broadly, Biden’s advantage in the polls on election day – a 7-9 point national lead, and apparently ahead in almost all of the battleground states — equates to a much more comfortable margin than Hillary Clinton had. Biden has roughly double the lead in the top battlegrounds, double the national lead and double the favourability advantage of Clinton in 2016. If the same degree of polling error that afflicted 2016 was applied here, Biden would still win comfortably.
All of this is why I believe it is reasonable to expect that Joe Biden will win the election by a comfortable margin, possibly a very big one. Put another way: if Donald Trump does pull off another surprise and wins re-election, fair and square, it would be an astonishing event. It would mean that the mechanism we have relied on for assessing public opinion for decades — a key part of the way governments decide on public policy, most recently on Covid-19 — is fundamentally broken. It would mean we have to rethink the basic epistemic foundation of our public conversation. It would be scintillating — so much so that I wouldn’t give a moment’s thought to those bets I placed on a big Biden victory. There’d be far more important things to worry about.
is the Editor-in-Chief & CEO of UnHerd. He was previously Editor-in-Chief of YouGov, and founder of PoliticsHome.
freddiesayers 
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
Subscribe“It would mean that the mechanism we have relied on for assessing public opinion for decades ” a key part of the way governments decide on public policy, most recently on Covid-19 ” is fundamentally broken.”
I fear that this is the case.
“It would mean we have to rethink the basic epistemic foundation of our public conversation.”
I believe that we must.
In an era with an ever-shrinking aperture of acceptable thought/speech/opinion, public conversation and honest debate taken in good faith is becoming increasingly difficult, almost impossible. I feel that this is a large part of the reason why we are where we are now, and why the polls keep getting it so staggeringly wrong.
Totally agree – when the MSM is committed to “repressive tolerance” even further left than Herbert Marcuse himself the polls are already a foregone conclusion. Its confirmaton bias gone mad. UK 2010, 2015, Brexit, 2019 and USA 2016 all called very wrong because of this. As Dinesh D’Souza points out the pollsters are trying the weigh-in trash talk common to high profile boxers. But its not 30 minutes in a boxing ring, its 4-5 years of damage to yours and your family’s economic and physical security that is at stake, so ppl will think carefully before they vote. The more the polls exaggerate leftwingers chances the more energised the regular working people are to come and vote for their own economic interests, victories for Trump, Bojo, Orban, Bolsinaro etc etc are the proof.
I agree.
I haven’t worked in polling but I have spent a lifetime in the Media and I know the relationship between the media and whatever is suppoosed to be the narrative at any given moment.
Polls have been getting it so wrong so often in recent times but the only time they’re *proven* wrong is because there happens to be an election that proves them wrong. For 99.9% of the time there isn’t any such actual event but the supposed views they produce affects the media, and thus amplified, social media and finally politicians….who after all are supposed to both get re-elected and ‘do what the people want’.
I feel polling creates a feedback loop that is near the centre of the divisiness we see on social media feeding into every aspect of our social and political life.
Polling that says Britain won’t back Brexit authenticates the idea that actually the vote is wrong, and the pro-UK vote in Scotland was *wrong*, and the landslide given to the Conservatives only last december was wrong as well.
So let’s have a legal challenge, or try and frustrate the will of the people, because they only voted that way because they were lied to.
I think the only reason Macdonalds or whoever use polls so much is because there are never any elections to make them wonder whether they are actually finding out the truth.
(I of course have the benefit of seeing a whole raft of states finally inching towards Biden, so making him President but by whiskers, instead of the avalanche of votes that were supposedly bearing him to inevitable victory upon and about which we have heard for the last year or so.)
Sad to even read this article. The demise of America as we know it is on the line. It is disheartening that so many are so ignorant of the opportunity they have that they would rather be “cared for” like a zoo animal than free. However, with the huge crowds and spontaneous parades that have popped up across the country, I pray these are indicative that the true American spirit is alive and well. Many conversations with real people in support of Trump and disgusted with Biden-people of varying skin color and background. I think this is not being accurately portrayed publicly or in the polls.
President Trump has definitely put it all out there in his campaign and that is meaningful to me. I guess who would want to be the president to go down in history as the last president of the free nation of the United States of America? Trump can certainly say he gave it all he had. Now we just hope for an honest election result.
The Democrats are readying to try and nullify the result if they don’t win. The media cartel has already said they will operate a news blackout if Trump has more than 270 EC votes by dawn on Wednesday. Thankfully the internet is going to make that hard to achieve and if we are mired in the same chaos as 2000 then Trump can sit steady in the WH until the courts do their work. If after all that he comes up short he can congratulate Biden and pack up and leave. As with the UK election last year it comes to something when the likes of Trump and Johnson are the least worse options….I do not know why the social democratic movements can’t pick electable leaders like Blair , Obama, even Bill Clinton?
Blair , Obama, even Bill Clinton?
So look at the reaction to them. All rejected.
Why? they screwed the electorate over.
You’re screwed because of the off the book debts run up by Social Security.
Not sure how that is relevant to whether the U.S. is a democratic republic or a dictatorship. A democratic republic gives the opportunity to recover economically much better than a dictatorship.
‘The big secret about polling companies, the thing you can’t know for sure until you’ve actually worked at one, is that they really, really want to get it right.’
You know, I am not entirely convinced. We know that magicians works by getting you to take decisions which they require for the trick.
Polls are no different. How the question is posed will illicit a specific answer. Deliberate bias on the part of the questioner, or unconscious bias?
this.
and it’s not as if the average reader holds them to account when they’re wrong. If that were the case, we’d all be ignoring this noise, simply for the 2016 performance.
and to your point, a clever writer of the question has already determined the outcome.
as a species, we’re more curious than discriminating, and they know it.
mf
Can you remind me which pollsters called Brexit and Trump (2016) right? Guess it’ll be be 3rd time lucky this time.
The biggest problem with online polls is that the weighting typically fails to correct for the almost inevitable skewing of selection toward people who are (1) able to/ choose to spend more time online, and thus are likely to be more strongly politicised (2) to be persuaded to participate by the kinds of banal incentives provided by polling companies.
Thus the latest UK Yougov snap poll which found a preponderance of support for the new lockdown appears to have been weighted by (1) vote in 2019 GE (by 3 largest parties; no category for the one third who did not vote) (2) 2016 referendum vote (again excluding a weighting for the 28 percent who did not vote) (3) Gender (4) Age (5) Region (London, rest of South, Midlands, North) (6) “Social Grade” (only 2 categories, ABC1 and C2DE.)
Polling companies who do wish to “get it right” must find ways of correcting for this inevitable skewing. This could be done by asking questions, for example, about time spent online. At the same time they will need to conduct regular face-to-face polling to obtain reliable measures of how this (time spent online) actually distributes in the population, to obtain the weightings.
Alternatively, given that it is much easier online to increase one’s sample, they could use much larger samples than is traditional, which will permit a much larger set of weighting factors and categories.
Right. This is actually far more important, because as Freddie says, polls are used by governments to define policy … like lockdowns. I am very suspicious that if you recruit people who have nothing better to do than fill out long online surveys, sometimes the same survey multiple times, in return for some tiny amount of money, then what you get are people who aren’t at all representative. There was a comment on UnHerd a while ago by someone who worked in polling saying that it’s an open secret you get a huge skew on any question that can be read as “pro social”.
Trump will win. Biden stands for nothing other than not being Trump. Any political analyst worth his salt will tell you that you need a positive message to appeal to voters. Just look at the turn out at the Trump rallies, he is inspiring people.
‘ he is inspiring people.’
That is very worrying.
Yes, but in a democracy “not being” is a very powerful force.
Boris wasn’t Corbyn for instance.
Boris also promised a weary electorate that he would “get Brexit done”
Two things: first, polls have gone from reflecting public perception to a sense that they now aim to shape public perception. Second, they have become make-news, lazy journalism at its worst.
Done well, they remain a viable research tool. Done well. We are only told of the result, never the question(s) or how it/they get asked.
Yet like the media, public trust in polling has been eroded. We saw that in 2016 and again this time, with poll after poll screaming about Biden’s big lead. Biden. A man who can barely construct a sentence and who only won after the party bosses forced a few others out of the campaign, a man whose running mate was summarily rejected by Dem voters. People may dislike Trump but the only affirmative votes for the other side are coming from people named Biden and Harris.
this gets me thinking, because in a way I agree and in a way I don’t:
– if betting on a result, I think you are correct about the lack of faith in polls – I believe very few people would wager cash in a bet, based on current polling.
but
– if simply scanning articles, I think the penetration effect of published polls are still far more effective than the admitted trust in them, which is to say ‘don’t think of pink elephants’ – they have to be digested to be ignored, and the digestion process is woefully flawed. kind of insidious, actually.
mf
Hi, I put in an application to write for UnHerd as there is a definite lack amongst the writers of understanding of how algorithms and data interact plus how to interpret a distribution of potential outcomes. This article has not changed my hypothesis. Polls fail as they are unable to define an unbiased representative dataset. Everything else is over-the-top adjustments to calibrate to recent historic results. This article is classic asymmetric in its argument. If they are right, you won’t hear the end of it. if they are wrong (again), then they will lick their wounds and come back next time.
Polling is a business, a profit and power-driven business, no more, no less and its focus is pleasing the people paying the bills and not public opinion.
I think polls do nothing but raise anxiety levels in the population. I also think rolling 24hr news does the same. They should both be stopped along with most of anonymous social media. There should be severe penalties for publishing anything that is provable to be untrue and the billionaire techies should have to pay reparations.
I know none of this will happen, but when you think about how much nicer life was when we had one or two doses of news in the evening, elections happened on voting day and then were over, and the only people telling us stuff was those that we wanted to hear from, and we could ignore the rest or get them to move on.
I think that in the future everyone will want to revert to those days.
Political polls aside, what about the ones that hold a mirror to society on contentious issues: look your a (small) “racist underbelly”. Immigration in NZ it is highly contentious and poll results on it potential political hot potatoes. I don’t think the UNophiles or corporates would be happy to have a wrong outcome.
Lately they have been pushing the Maori language and replacing New Zealand with Aotearoa. “And it’s fine over most of Aotearoa tomorrow…” A recent poll showed only 10% favor a name change. Trust me they have a postmodern agenda “a critical privileging of Maori culture”; “New Zealanders will come to know themselves in Maori terms”.
After the Christchurch Mosque shooting Research NZ ran a poll on how we feel about diversity. The polster concluded we are split down the middle.
So I’m wondering what happens when the person paying the piper doesn’t want to get is right?
‘They’re not perfect, but they’re the best indication of public opinion we’ve got.’
Makes one wonder quite how anyone ever managed to form an opinion on anything until they were ‘reliably’ informed by someone else who apparently knew irrefutably what everyone else apparently thought?
Maybe best not to predict the outcome, it’s little more than a space filler before the main event
I have a feeling that Trump will win, and he will win big.
Always room for an upset though.
I am not sure that the most important issue for pollsters is to get it right.
Their work is before the election. Polls are a form of news, very little more.
They keep people tuning into the news. Polls provide a constant fuel for the news cycle. Absent other news there are always “new poll figures.” Furthermore poll companies have their own business interests and political polls are, as Sayers suggests, the most visible aspect of their business. One set of the target audience comprises actual potential clients. But the larger portion of the target audience is the public, who are bombarded with political poll results on a daily basis.
Like PR generally, it doesn’t all have to be positive to have a cumulatively positive effect for the one putting out the PR. Polling companies have multiple agendas, like every other element of the media and and think-tank-istan.
Good quality opinion polls are the best ” indeed, the only ” real datapoint for quantitatively estimating the outcome of an election before it has happened.
==============
Or you follow the money and look at the betting market.
Or you look at what happened last time and general trends
Or you look at last time and other factors, such as the economy.
When people say the only way, you know they have a thinking problem or a bias problem
If polls were the ‘best indication’ then they would not be so wrong, so often. In addition, polls do not take into account human nature, where, in certain situations of possible ‘loss’ or ‘threat’, literal, emotional, psychological, mythical -people will not say what they really think but what they believe the other person wants to hear and what may be in the interests of their own safety.
This is why providing aid to the Third World is so hard because people are saying what they think the questioner wants to hear, not what they really think. Human nature is universal and never more so than on contentious topics and politics is always contentious.
And unless one knows who asked the question, who was asked the question, what the question was, it is impossible to assess the value of the so-called ‘results.’ You can create a question which delivers the answer you want very easily and that only reflects the agenda of those doing the polling and not the public.
There’s BS, then there’s ‘grade A’ BS, and I’m definitely smelling something wish-fulling here. How about Freedie puts his money where his mouth is – and offers to resign when he’s wrong…?
And here’s why he’s wrong…
https://amgreatness.com/202…
It’s unclear what use polls are beyond business profits or public curiosity/entertainment. Can they provide any wise guidance to the voting public?
Writing an article about polling without any mention of statistical methods (or lack of them) or the psychology of the sample is like reviewing a hotel without mentioning its price, star rating or location. It seems to me the polls are like weather forecasts – “human interest stories”. They are opinion pieces rather than stats. If you look at 5:38 in USA or “Electoral Calculus” in UK the genuine weighted stats show a huge range of outcomes, are dense and hard to read, and not usuable for a 30 second slot on the MSM. The established media need stated “facts” such as “Obloont leads Dequoont by 2.5 percentage points” or “50,00 will die of SARS-CoV2 in the next month”. The idea being the viewers will have forgotten the “facts” within a week at most and be receptive to the new “facts” which show no link to the old ones. The psychology of interviewees is even more unfathomable, perhaps Electoral Calculus’ “3d” predictions and “7 tribes” analysis are the best efforts but even they have huge margins of error. You wouldn’t use such a system to navigate a plane or fix the brakes on your car!
You were saying? Yep. fundamentally broken. Kaput.
“Like most people I have never even seen a polster”. So says Thomas Sowell, at minute 3:15 of https://tinyurl.com/y2hvnfks – a quotation which has even more resonance today, given the latest egregious failure of polsters.
Freddie, why do we care what the public thinks? They are largely uniformed, filled with bias, and with rare exception, unable t think critically or otherwise.