December 16, 2021 - 4:00pm
The vacating of an MP’s seat mid-term is always cause for excitement among political observers, especially when Government is facing serious political pressure. So we shouldn’t be surprised that today’s North Shropshire by-election is, according to commentators, pundits and even Tory MPs, a ‘test’ of Boris Johnson’s leadership.
The truism is that by-elections do not reflect general election trends, so anything can happen. But by-elections are still bound by patterns that go beyond the day-to-day news cycle even though the tendency is for the noise of the latter to drown out the signal of the former.
Taking a superforecaster’s approach to this by-election, we can assess just how likely it is that the Tories will hold onto their seat. By looking at the percentage of by-elections that led to a seat change between parties since 1945, we can establish a base rate to give us a probability of what will happen in North Shropshire.
Contrary to the claim made by Liberal Democrat leader Ed Davey that winning the constituency is a “coin toss” (bookies agree), the Liberal Democrats’ chance of winning the seat is actually much lower. According to my estimate, based on tallying up by-elections over the last 70 years, the Lib Dems’ chances stand closer to around 20%. This is to say that it’s possible, but it’s far from the likeliest outcome.
Coverage of by-elections tends not to focus on such facts. Before the Batley and Spen election, senior Labour figures put the party’s chances of holding the seat at 5%, even though Labour had held onto the seat for over 20 years. Betting markets meanwhile put Labour’s chance of winning at around 20%. When Labour won, the New Statesman‘s follow-up analysis argued that Labour’s subsequent victory was “against the odds” while the likes of the Mirror, the i and the Daily Record presented the result as a surprise, rather than the most common outcome in such by-elections.
Bookies similarly put the Brexit Party’s Mike Greene as the favourite to take Peterborough in 2019, a highly marginal seat that became available after a recall petition succeeded against Fiona Onasanya. While the race was tight, punters should have been more mindful of incumbents’ statistical advantage during by-elections. In the end, Labour won by nearly 600 votes.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeThis hasn’t aged well.
Beat me to it.
Oh I dunno, I’m still happy that I said there was a one in five chance of this happening.
What would the result have to have been for you to say ‘ok, I was wrong’?
I’d put it this way. If you were rolling a dice and asked me to predict what the odds were of it not landing on 1, I’d say 83%. If it then lands on 1 my prediction was still correct: those are the odds, irrespective of the result.
Put another way: if the Tories had retained their 23,000 majority, that doesn’t mean my 80% prediction of them holding the seat was right. I’m giving a probability for an event happening, but whether it does happen doesn’t tell you a lot on its own. Unlikely events happen all the time over the sum of human experience, just as likely events frequently fail to happen.
You need to assess a predictive model over time, of which the one I described here is very crude. If you use it again over many by-elections, the test would be whether it holds true (ie, by-elections give hold results about 80% of the time).
If you wanted to tell whether I was generally good or bad at forecasts you’d have to assess my predictions over time. If things I say have a 20% chance of happening tend to happen about 20% of the time, I’m doing a good job. (See section 4 here on Brier scores for more info.)
So essentially your predictions can never be wrong? As long as you don’t say there’s a 100% chance of something happening then every prediction is correct?
Wasn’t that super was it
Superforecasting sounds dead easy. You just have a look over the last [large number here] of events and say, well them’s the odds. Bookies, pundits and academics are simply wasting their time weighing up the convictions of cash in the game punters, utilising their experience and expertise, or conducting surveys and interviews with actual people (a la Tanya Gold of this parish).
That said, I’m always surprised when the Lib Dems win anything more than a scratch card these days.
Now the hilarious truth has arrived, this article reads like Professor Ferguson’s take.
Same way that fact checkers have become synonymous with “narrative”, forecasters have become synonymous with “wrong”.
All of them wrong, all the sodding time. You can see why people are so bored of it all.
As a genuine question, what do you think I got wrong? Do you think a Lib Dem win was likelier than I said? Less likely? What about the Tories’ prospects? Bear in mind the message above is broadly that a Lib Dem win was possible, but not the likeliest outcome.
If the LibDems had a 1 in 5 chance of winning, wouldn’t the margin of victory have been much much smaller by rights? The size of the victor’s majority implies a 1 in 250 type event does it not. Either that or your model is inaccurate.
I don’t think that follows, as saying so and so have this chance of winning doesn’t imply anything about margins of victory. That chance captures everything from a narrow victory to a big one.
So the swing was 34%. If the swing size is irrelevant then would you have forecast at 1 in 5 chance even if the by-election was in a constituency with a wafer thin majority?
That makes almost no sense to me statistically.
One of the limitations of this article is I didn’t do any adjustment for local conditions. If you wanted to build a more accurate model for predicting by-elections you’d start with a base rate (not necessarily my one) and then start to consider how other factors affect probability. For example: size of majority at last election, time since last general election, approval rating of sitting government.
To assess properly you’d need to look back on previous by-elections, but intuitively I’d expect that seats are more likely to be held if the last majority was large. Perhaps that is untrue though, in which case it would have been sensible to revise that 80% figure down. But from what I’ve read most of these forecasts don’t tend to move much away from the base rate, even after adjusting for specifics.
Thanks for the explanation.
There were a whole bunch of factors at play in the result; some specific to this time and others specific to this type of constituency. Extrapolating from historical by-election results was never likely to work.
Prashant summed this up in another comment, but your forecast that the Lib Dems’ chance of victory stood “at around 20%” is at odds with the magnificence of their win. If you’d said their chances were 40%, or even 60%, that would still have looked off target.
I say that because your 1 in 5 chance was presumably them squeaking in, rather than romping straight past the finish line, collecting the bunting on the way through and then heading for the pub.
Hence the pithy remark, which I now regret knowing that you’ve been polite enough to engage BTL.
No worries, I can certainly see why people are critical. I don’t think the base rate says much about the size of a win. The figure that incumbents win 80% of the time captures everything from huge majorities to narrow victories. As a prediction starting point it’s not supposed to imply anything about the size of a potential win or loss.
Ok, lib-dem it is, then.
ouch.
Oops. But bold to print this yesterday.
This aged well.
Well, full marks for being brave enough to take a punt. How wise we all are after the event. You wrote “One could also argue that … has given the Lib Dems a real chance.” If only you’d stopped there.
We can think of this particular piece as being for light relief only – like a scratch card. However, we are plagued nowadays with so-called experts who tell us what is to be and how to behave – and who turn out to be just as ill-informed, biased and wrong as the rest of us. And yet on they go, and on and on.
If we can’t reliably predict the weather, the physics of which is well understood, how on earth can we think we know enough to predict the evolution of hideously complex and ill-understood systems such as we find in politics or virology? Hubris is what the ancient Greeks feared most of all – we would do well to fear it too.