by Freddie Sayers
Thursday, 1
July 2021

Govt modeller: What our Covid forecasts got wrong

SPI-M member Dr Mike Tildesley considers whether June 21st could have gone ahead
by Freddie Sayers

SPI-M  (the “Scientific Pandemic Influenza Group on Modelling) is the government committee in charge of producing forecasts for the future direction of the pandemic in different circumstances. It was their report in early June, combining mathematical models from Imperial, Warwick and LSHTM, than persuaded Boris Johnson to delay the planned re-opening of society on 21st June to its current scheduled date of 19th July.

In the weeks since that report, two things have become clear: the raw case numbers have been rising very rapidly, but the hospital admissions have been much lower than expected when the PM made his decision. As of today, 1st July, just over 250 people per day are being admitted into hospital with Covid, compared to over 600 as forecast by SPI-M.

Freddie Sayers spoke to Dr Mike Tildesley, an infectious disease modeller from the University of Warwick who sits on the committee and works the models himself, about how his forecasts have performed against reality, and whether the PM made the right decision.

Why were the SPI-M models overly pessimistic?

  1. Underestimated vaccine efficacy
I think the vaccine efficacies throughout have been slightly underestimated, shall we say, by the modelling groups, we are actually find that the vaccines are much more effective than previously we thought they would be. Now when these models are parameterised,  the vaccine efficacy data came through from Public Health England, so we’re not making up these values, we are using the best estimates of values that are coming through from those on the ground that have their estimates of them.
- Dr Mike Tildesley, UnHerd
  1. Overestimated behavioural change
I suspect this is something else that perhaps some of these models have slightly overestimated as to what we might expect that we’ll do in terms of the R numbers. This is partly because of people’s behaviour. So just because controls have relaxed, it looks like looking at the data that actually people haven’t gone back to  ‘normal’ in terms of what we might have expected prior to the pandemic. So people are still being a little bit more cautious. Maybe they’re not going to the pub in the way that they were, say, back in January 2020. And that, obviously has some implications upon these forecasts that when these models were done.
- Dr Mike Tildesley, UnHerd

So, had we known then what we know now, should the PM have gone ahead with June 21st?

In hindsight, possibly — we’re in a position that the vaccine efficacy is much more effective, but the delay has also enabled us to vaccinate a lot more people with slightly higher level of restrictions in place. This is where it’s always a bit of a challenge because in a way if you wait, you’re always going to have a smaller wave. But of course, if you wait, then that’s much more damaging for the economy and for people’s well being and mental health and so forth. So there’s always a little bit of a trade off. The epidemiologist in me would always say it’s slightly better to err on the side of caution, but I was always very adamant when I said this delay was in place that it is really important that we get back to normal. I think it’s really important that we do get back to normal on the 19th of July. And the delay was probably necessary to allow us to resolve that uncertainty.
- Dr Mike Tildesley, UnHerd

Why 19th July should go ahead as planned

Looking at the data, looking at possible admissions and deaths, there’s nothing at the moment that really worries me. And I think if we are going to get back to normal, we’ve really got to do it over the summer, when the virus is less likely to transmit anyway. Otherwise I think we’re going to be in a situation where it’s going to be really hard. So I’m hopeful 19th of July does go ahead as planned.
- Dr Mike Tildesley, UnHerd

Is the Government placing too much emphasis on models?

The model should only form part of the decision making process. You also need health experts, economists, people from all these social sciences with a huge range of different expertise to advise the government, because it’s not just what the models are predicting. There isn’t certainty in those models, I think our responsibility is to make sure that we communicate what we expect from the models, but also communicate the uncertainty in those models. That’s really, really crucial. I don’t ever think it’s helpful to go in the media and say, if we’d locked down three weeks earlier, we would have saved 100,000 lives or whatever that’s not productive. Because any one of those decisions, there’s uncertainty around what we might expect to see. So I do worry a little bit that too much of the responsibility has been put on modellers. And I also worry a little bit about some of the rhetoric and in among some members of the government, who have always used the mantra “we’re following the science”, which almost seems like it’s a little bit of a get out of jail free card.  
- Dr Mike Tildesley, UnHerd

On the forthcoming Delta wave in other countries

We’re doing pretty well — getting back to domestic freedom, as it were, is looking really possible over the next few weeks. Internationally it’s a much more bleak picture…. In certain countries in Europe this is a real worry, because of course, if they start to relax, and vaccine uptake is really low, then they’re going to see a wave. And actually, their wave is likely to be not just a wave in cases that we’re seeing, but potentially a wave of hospital admissions and deaths as well, which is a big worry.
- Dr Mike Tildesley, UnHerd


Should we stop publishing daily death numbers?

I think, ultimately, we’ve got to do that. I think particularly with the deaths as I said, I’ve talked in the media before about how we don’t say how many deaths there are every day from, say, cancer, or from road accidents and all these other things. Actually, we’re at the stage at the moment where the number of deaths are sort of in the 10s, significantly lower than deaths from many other causes. And I think we need to put it into perspective. As I said, we don’t want hundreds or 1000s of deaths, but having a situation where for the foreseeable future, we report the number of deaths from COVID, even when it’s really, really low numbers, I don’t think is helpful in terms of enabling us to get back to normality.
- Dr Mike Tildesley, UnHerd

On why we need to start treating Covid like flu

In the longer term, we have to get more into a kind of a flu type relationship. We don’t want 1000 deaths per day, clearly, that’s catastrophic. But if we get into the winter, and we have a rise in cases and a rise in hospital admissions, similar to what we’ve seen in previous years for flu, do we consider that to be acceptable? Anyway, everyone will have their own opinion as to what the answer to that question is. But clearly, in the longer term, we need to develop that more of that kind of flu relationship with COVID.
- Dr Mike Tildesley, UnHerd

Join the discussion

  • Interesting set of bullet points though I suspect that the first four are not relevant to the modelling end of the epidemiological community – more biological than statistical. Presumably their effects can be parameterised and incorporated in the model runs that go into the appendix with the sensitivity analyses. Of course they couldn’t be if you had your way and defunded the area of research. Is funding “extravagant”? There aren’t that many groups, no labs so those sort of costs will be minimal and I guess they’re all on salary scales that are common across their organisations.
    Is the fourth bullet point right? Variants tend to get associated with geographical locations suggesting a pattern of radiation rather than simultaneously popping up everywhere.
    I know from my own field, hydrology, that modellers tend to fall in love with their models and they are as subject as much as any of the others in the chain from the basic medial science via epidemiological modelling through to policy of minding their backs. So everybody nudge their conclusions to the end where they are least likely to suffer opprobrium from the biens pensants and each nudge en route accumulates. It’s certainly the case with climate change whose chain has many more links than epidemics and a lot more nudging.

  • I’m guessing that Freddy knew that the best way to allow this madmodeller to clearly demonstrate his own ridiculousness was to dig a very gentle hole in which the modeller could bury himself.
    Additionally – the vaccines haven’t been nearly so amazing as the modeller says.
    If you compare the mortality data now with that of the winter just gone (ie if you compare apples with oranges) – the vaccine (or some other factor?) does indeed look impressive.
    However if you compare now with last July (apples with apples) the difference is very small (and non-existent in anyone under 65), and will be at least in some part due to better treatment.
    The University of Cambridge Biostatistics Unit has this data available in graphical form for easy comparison.

  • The variants are being re-assigned arbitrary names because the geographical names are supposedly not meaningful, being more related to the vagaries of where the sequencing is being done than true geographical origins.
    The lack of biology in epidemiological modeling is a rather glaring problem. Claiming “biological” issues like seasonality are not relevant to predicting disease is exactly the kind of alarming statement that should make us hold grave doubts about all academic modeling papers! Seasonality is obviously critical if you’re trying to plot future cases of a respiratory illness in the UK, given that such diseases are usually seasonal. That’s not even biological knowledge really, it’s the sort of common sense thing your grandmother could tell you, yet SAGE modeling only started trying to take it into account very recently and at least one model still isn’t doing so (see the recent analyses by Glen Bishop).
    Meanwhile these models are still incapable of explaining when and why outbreaks start or end. It’s not even that they try but fail – they don’t even try. The models simply believe an outbreak of any infectious disease cannot stop until more or less the entire population is infected, a facile assumption that doesn’t hold up in the real world with any actual epidemic in the past. If you can’t explain why outbreaks of COVID suddenly go into reverse even in the absence of any government policy changes (and they do), then you have no business modeling COVID, yet this obvious fact hasn’t stopped anyone in academia from doing so.
    As for costs, costs have to be seen relative to benefits. The job of a scientist is to produce accurate and generalizable theory. Modern model-based epidemiology appears to have produced no theories of disease whatsoever, not even non-viable ones. Instead every single outbreak of every single disease gets its own ad-hoc model, one that encodes no real insight into disease but instead is a mere exercise in curve fitting (at best). That is not science!

  • To get involved in the discussion and stay up to date, become a registered user.

    It's simple, quick and free.

    Sign me up