So, a country’s CFR will vary depending on how many tests it’s done, because that changes the denominator. But at least the numerator — the number being divided — is probably pretty straightforward, right? A death is a death.
Sadly, that’s not the case either. Dr Hannah Ritchie, one of the data scientists at OWID, points out that the death statistics are complex too, for two reasons. One is prosaic: a lot of people who have the disease and will die of it have not yet died. “The period from onset to death is about a month,” she says. So your simple “divide the numerator by the denominator” rule — the number of deaths by the number of patients — doesn’t work, because your number of deaths is a product of how many people had the disease a month ago, not how many people have it now.
That’s bad enough. But there’s a more profound problem, which is that deaths themselves can be recorded very differently in different places. Professor Sir David Spiegelhalter, a statistician at the University of Cambridge, says that the UK simply counts people who have tested positive and then died. But in some other countries, people are recorded as having died of Covid-19 if they had the symptoms, even if they weren’t tested (“suspected” as opposed to “confirmed”); in others, people outside hospitals are not tested and so are not recorded.
“Even the number of deaths is not a perfect statistic at all,” Spiegelhalter says. El Pais did an interesting look at some of the international differences here; Britain’s Office for National Statistics explains why its numbers look different from the official Government ones here.
To some extent it doesn’t matter, says Spiegelhalter: as long as each individual country maintains the same regime, ”the number of deaths is still a good monitor for the shape of the epidemic”.
But it’s not clear that they are fixed; countries may have good reasons to change the way they collect data as circumstances change, but it apparently happens often enough that the World Health Organisation feels that they have to ask countries to notify them when they do it. Famously, China did so earlier in the epidemic, but others do too: in complying with the WHO’s request, Australia has noted that it has changed its definition of a Covid-19 “case” (and therefore a Covid-19 “death”) at least 12 times since 23 January.
So, in essence, to work out the IFR — the number we want — we need to know two other factors: the number of people infected with Covid-19, and the number of people who died of it; the denominator, and the numerator. And, sadly, both numbers are uncertain.
How much does any of this matter? Well: now, I’m going to try and build my very own, rather stupid model. I’m not going to try to predict the future; I’m just going to try to use some very simple numbers to “predict” the number of cases there are in the UK right now.
First, the number of reported Covid-19 deaths in the UK is 1,408. Second, the lag between infection and death is about three or four weeks; let’s say three. Third, the number of (confirmed) cases in the UK has been doubling about every three to five days; let’s say five (and assume it’s staying constant; forget social isolation for now).
With those numbers, we can plug in our guess at the IFR (the infection fatality rate, remember: the real number we want to know, the “if I get it, how likely am I to die” number) and use it to work out roughly how many cases we’d expect now.
So what number should we use as our IFR? I’m going to use three: one from Imperial College London’s MRC team, the one behind the famous model; and two from CEBM. Imperial’s latest work assumes an IFR of 1%; CEBM estimates it to be between 0.1% and 0.26%.
If we take the Imperial 1%, then that means that we can multiply the 1,408 deaths now by 100 to get the number of people who’d had the disease three weeks ago, because we think about one in 100 of them died. So about 140,000 people.
Then we can take our doubling time — we said five days — to get how many cases we’d see now. In three weeks, 21 days, you’d see four doublings; two to the power four is 16. So 16 times 140,000, which is 2,240,000. We could imagine that we’ve probably got about two million cases now.
But let’s use the CEBM numbers. First the highest one, 0.26%. If we take that, we can multiply the 1,408 deaths by 400, instead of 100, to give us the number of infections three weeks ago. That is 560,000. Then we can multiply that by 16 to give us an estimate of how many there are now: about 9,000,000.
And how about if we use their lowest estimate, 0.1%? Same routine: 1,408 multiplied by 1,000, multiplied by 16: more than 20,000,000.
So by changing a single number, the IFR, to one of three plausible values, in a very simple model, we get outputs that range from “3% of people have already had it” to “30% of people have already had it”. And that’s before we start messing around with the other assumptions; is doubling time three days, not five? Is infection to death four weeks, not three? Is “1,408 deaths” even correct?
I want to reiterate: this is a very simple, stupid model, put together by a journalist, not an epidemiologist. The actual models will be far more complex, and will take into account other things — the number of cases in hospital and so on — to try to ground them in objective fact. Don’t mistake this for some plausible estimate of infection numbers. And there are loads of other things to worry about: people suffering long-term health consequences, even if they live; people dying of other things because the healthcare system is overwhelmed.
But the problem that I’m illustrating is real. Small, plausible adjustments to your inputs make your model spit out very different things. The assumptions you make are vital. We haven’t even started to think about other crucial things — for instance, government interventions, and how effective they are.
“The different scenarios — isolation, closing schools, quarantines — they come with massive assumptions about how adherent people are,” says Ritchie. “You get massively varying outputs, depending on what you put in.” Plus, of course, it’s all circular: your model influences how you respond; your response changes what numbers get put back into the model.
What you need is better numbers, to plug into the models. And that’s what people are trying to get. The CEBM paper uses, among other things, numbers from Iceland, which — being tiny — managed to test a huge proportion of its population, nearly 3%. It found 963 cases and just two deaths; an CFR of 0.2%, from which the CEBM extrapolates an IFR of 0.05%. But that’s likely an underestimate, because quarantining has protected the elderly, the most at-risk group. Others have done something similar with passengers on the cruise ship Diamond Princess, finding a CFR of around 1% — likely an overestimate, since the passengers tended to be older.
Another way has been to screen small groups who are in quarantine, such as the Diamond Princess passengers or passengers on aircraft, to see how many people 1) test positive and 2) show symptoms. If you know how many people have the disease but are asymptomatic, then you can extrapolate to the wider population — if you’re testing all the people who are symptomatic, and you know that 50% of infected people don’t show symptoms, and you find 20,000 cases, you can estimate that the real number is more like 40,000.
But there’s a problem here, too, which is that “asymptomatic” is not a simple thing, according to George Davey Smith, an epidemiologist at the University of Bristol. If you’re sitting in quarantine and you cough, you might be recorded as symptomatic; but out in the real world, you’re not going to take yourself to hospital for a gentle cough, so you still won’t get tested.
Instead of there being a neat “symptomatic/asymptomatic” division, you have a third group: people with some symptoms but who think it’s just a cold in the chest. (As I write, I’m coughing a little; but I don’t think that’s Covid-19, I think it’s just because I went for a run this morning and the cold irritated my lungs. But I’d probably be recorded as “symptomatic” if I were in quarantine and being screened.)
And yet another way is to look at how many people die every year, and how many people are dying now, and seeing whether more people are dying than usual. That’s how we attribute deaths to flu each year, says Spiegelhalter; the European Monitoring of Excess Mortality group (EuroMOMO) uses this data to say that in an average year in the UK, 17,000 deaths are “associated” with influenza. But so far there is no excess death at all, except in Italy: the EuroMOMO charts are all around the seasonal average. That will likely change, but we forget that in a population of millions, you’d expect thousands of deaths every day anyway: even the Covid-19 pandemic is still being lost in the noise.
In the end, we need testing. And not just the sort of testing we have now — PCR testing, which shows who has the virus right now; we need serological testing, which shows who has had it in the past. That will come along relatively soon, and hopefully can be quite quickly used to test randomly selected people, like an opinion poll sampling a population; then we can see how many people have had it, and from there work out the IFR. But for the moment we don’t have that.
I wanted to write this to give an impression of how appallingly difficult the modeller’s job is. I write, sometimes, pieces about statistics — I suggested that claims about the loneliness epidemic, teen suicides, and the media’s influence on Brexit were overstated, for instance. They involved very basic maths, done for very low stakes: if I messed up, if I failed to carry a 2 or whatever, I would look very stupid and would be very embarrassed, but no one would die.
Whereas, if the Imperial College modellers get it wrong, with their far more complex maths and their far more uncertain inputs, they could sway government policy enormously. Whether we lock down society or carry on as normal depends heavily on the outputs of models like these. And it’s not that there’s an easy “better safe than sorry” option; if we crash the economy, it will (eventually) cause real health problems.
A February 2020 review found that 10 years of austerity may have caused the growth in life expectancy to stall, especially among the poorest; I’m sceptical of the “130,000 deaths caused by austerity” stat, but it’s pretty clear that it had a real negative impact. The post-Covid-19 world will almost certainly involve huge austerity to pay for the vast costs incurred during the virus.
Get it wrong one way, and thousands of people die unnecessarily from the virus; get it wrong the other, and you crash our public services and kill people that way. (I’ve only seen one attempt to model the health outcomes of that crash, and I have no way of judging it; for what it’s worth, though, it does say they will be extremely terrible; on the other hand, recessions don’t seem to shorten life expectancy, so who knows.)
So I’m very sympathetic to the modellers. But there are things which would help, and which they can do, but haven’t, so far at least. Ferguson’s team has not released the code his model is based on; he says he is working with software developers to do so, but proponents of open science, like Davey Smith and his colleague Marcus Munafò, say this isn’t happening fast enough.
“These models are so sensitive to their assumptions,” says Munafò. “And they’re black boxes.” The code is 13 years old; it’s vital that other scientists are allowed to look at it, check it for mistakes and stress-test its assumptions. There are other, open-source models available, but the Imperial one is still kept under wraps, and it shouldn’t be.
Because a lot rides on the outputs. If millions have already been infected and the disease is less deadly than we think, then our response should be very different to if millions are still to be infected and tens or hundreds of thousands more will die.
The latest from Ferguson’s team suggests that between 1% and 5% of the UK’s population has already been infected. Oke of CEBM thinks that that could be an underestimate — he thinks that the disease was circulating in China for a month or so before it was announced. “There were early reports of doctors saying they were seeing unusual respiratory symptoms, which were suppressed,” he says. That could have brought it all here much earlier.
When I mentioned the Oxford “study” — in fact a model showing what plausible inputs could produce what we’ve seen, one of which was a very low IFR and huge number of people already infected — he didn’t endorse it, but said “I think a lot of people have underestimated how far this has already spread, and how early.” Davey Smith also thinks that those Imperial figures could well be an underestimate. I have no idea if they’re right or wrong, but whether they are or not matters a great deal.
For the record, to take us back to the beginning, Peter Hitchens had flatly misunderstood what was going on. The 500,000 number was a worst-case scenario if we did nothing; the 20,000 was Ferguson’s team’s estimate of what we’d see now that social-distancing measures and so on are in place.
The 5,000 wasn’t from Ferguson’s team at all but from an electrical engineering group also at Imperial who just eyeballed the death curve on the China graph, fitted the UK numbers so far onto that, and extrapolated from there. “They explicitly say they’re not doing any epidemiological modelling at all,” says Spiegelhalter, “and they retracted it two days later on Twitter,” after it became obvious that it was wrong.
But we shouldn’t be complacent and assume that, just because people are misunderstanding what the modellers are doing, the models must be correct. Everything that comes out of them is the product of what goes in, and all of that is going to be wrong, to some degree. “The key point is that the numbers we have now are not correct,” says Ritchie.
If you look back to previous outbreaks, such as the 2009 swine flu epidemic, the numbers people were using while it was still going on were wildly different from the ones scientists settled on afterward: early estimates in 2009 were between 0.1% and 5.1%; the eventual WHO estimate was just 0.02%, similar to seasonal flu. In a fast-moving situation, it is easy to make large mistakes. (In either direction! I am not suggesting that the Covid-19 situation will necessarily be similarly overstated.)
These models are the best information we have at the moment. But they are hugely uncertain, and likely to be wrong. All we can do is try to get better information for them, and make the best decisions we can under conditions of appalling uncertainty — and be forgiving of the modellers who are desperately trying to make life-changing, history-changing decisions at high speed and with bad data.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeExcellent article. We’re stuffing our economy for something that might be no worse than the 1968 ‘Hong Kong flu’
Mortality in ’68 was circa 80,000 and we shut nothing down.
Upon first death in Italy the whole village was mass tested on Feb 20 and found 50% were infected with 45% of those asymptomatic – Prof Crisanti Padua Univ. This supports the “Oxford Univ finding suggesting 20 mill may already be infected in UK with associated very low IFR NB that there is currently no excess mortality. This suggests that lockdown of young population should be released asap with continued protection by isolation of vulnerable population. This will prevent economic disaster.
The veracity of the contagion appears to be the key, so if approximately 50% of the population of U.K. contracted Covid 19 which seems quite plausible, you can then play with the death rate percentage.
In the UK and setting aside princes, prime ministers and celebs, we have until recently only tested people in hospital with covid symptoms – so people who are ill and who are reasonably thought to be infected. So why has there been such a low positive rate?
That is a very good question!
UK has 1 positive in every 3.3 tests – about the highest positive ratio in the world, because we have done so few tests targeted on those who most probably have it. The bad news in there is that many of the people who think they might have had it but were never tested probably didn’t
Tom’s model may be simple but it is also intelligible. Models with many equations and many parameters and intricate mathematics are not intelligible. I mean no-one can fully grasp them or see in any detail how changing the parameters changes the model.
This is a standard criticism of complex multi-equation models in economics. For example one by the late Wynne Godley in 1999 purportedly showing that EU peripheral; countries would always be at a disadvantage compared to Germany had over 70 equations! How do you criticise that?
That leads to the second issue. because they cannot be properly understood they cannot be criticised either and so we are at a loss to see when they are wrong. In the worst case of this – the climate models – we will have to wait till 2100 before we can assess whether they were right or not.
So Tom in the end defers to complex models by clever people run on big computers (well actually you don’t need a big computer). I prefer simpler models that promise less but whose actual performance can be monitored.
After all the only useful message that has come from the Imperial model is that allowing the epidemic to run would overwhelm the NHS – and this message was only issued when Italian experience showed that the rate of hospitalisation was far greater that had previously been fed into the model.
From the article: “found 963 cases and just two deaths; an CFR of 0.05%”. Since 2/963 is approximately 0.002, isn’t this supposed to be a CFR of 0.2% (and use ‘a’ instead of ‘an’ while at it)?
b****r! Yes, you’re right – the CEBM used the 0.2% to extrapolate an IFR, not CFR, of 0.05%. My mistake – hopefully should be fixed in a little bit.
Thanks for pointing out! Whoops.
However Tom the IFR is the fatality rate for infections NOT disease. Not all infections result in disease. That’s an important distinction.
I got lost in your maths I think.
If for 1% you multiply by 100 then if you use 0.1% shouldn’t you multiply by 10?
A very good article. I fell ill on the 15th January with what was the worst illness of my life . A virus entered through my left eye I believe. I took to my bed for close on a fortnight and apart from my wife saw nobody. She too fell ill and took longer to recover. The course of the illness was exactly like Covid 19 except that it did not proceed down into my lung other than leaving me a bit short of breath and in a bit of pain when I breathed. The exhaustion was the most I have ever experienced. Recovered by the 4th week but took it easy as I had the feeling that this would recur if I pushed myself.
It did not feel like flu at all but something else. I have since found that others had a like illness around the same time.
I have this feeling that the virus was here from December and thus earlier in China. It must be extremely infectious and when it reaches a critical mass in any population almost seems to explode. This is why i suppose we are trying to delay the spread.
It would seem that random testing of the whole population for antibodies would tell us much.
Good article Mr. Chivers, thanks.
Another angle I’ve picked up on, the difference between death with CV19 and death of CV19 i.e. was it the primary or exclusive cause of death and the consequent difference in various counties as to how the death is recorded. Another factor being how vulnerable different people are to it… not just based on age or pre-existing conditions. Seems it does not effect us all in the same way.
Conclusion being, we just don’t know enough about this thing!
Quite how our Government responds therefore is hard to judge, I guess they have to do what they’re doing, assume the worst case for deaths and try and manage the economic impact as best they can whilst protecting the NHS.
Thank you for a very well written article. I will add that I like your writing style and its humility whilst also asking the right questions. I appreciate your requesting all of us to go easy on the modellers, they are only human too.
This isn’t quite right. The IFR is the fatality rate for the number of people who have the INFECTION. You can be infected ie a carrier without having the disease.
Best balanced article I have read on this – but I’m only saying that because I had thought most of this myself without doing the detailed research and I agree with it. But why, given that we have COBRA scenario planning for situations like this weren’t we more prepared for the process of testing and quarantine and the overall response, in the light of COBRA being there to plan it, has the response been so shambolic?
Yes, and why did government allow Spanish football supporters en mass into Liverpool when such was not the case in most of Europe?
We don’t need to model this to know that is the reason that Liverpool is having greater rate of increase than Manchester. Or was it modelled before it was allowed as an experiment to judge what proportion of Spanish were infectious?
Surely among the unknown variables discussed the total number of Covid 19 deaths in a county, which will checked for verification in several months time, compared with its total population, and the rate of increase or decrease in the deaths/day are figures that need to be explained & reflect on the state of its health system & the ‘scientists’ that the government was prepared to take advice from. In UK & USA the desire to let it rip through the population & so gain ‘herd immunity’ has been disastrous for the economy they wished to protect. Of course the death toll will be mainly of older people & /or those with underlying health conditions, which it seems was thought to be a price well worth paying: they would die soon anyway as a respected UK statistician on BBC ‘s ‘ More or less ‘ said unchallenged in the programme before last. Cummings has been reported as saying pensioners will die & of course they will. Just how many die in each country will be judged globally in months & years to come
My preferred metric: Deaths / 1 million population, but you only know the final number when the epidemic/pandemic is over (5.7 globally on April 01, 2020, 14:40 GMT, and counting)
Which source are you using
Am I right in thinking Covid19 has only recently been officially adopted as a principle cause of death . Before that happened people could have died with it but not because of it ( because it wasn’t listed as a notifiable cause)
Why has no-one suggested large scale random surveys using the same methodology as opinion polls. By asking if each respondent has no symptoms, has symptoms or has recovered from symptoms we get results that can be repeated say weekly to give trends and provide breakdowns by region etc.
This is just one aspect covered in an outstanding article. I have been trying to get this considered for weeks with no result.
As unherd has an ex YouGov exec maybe you can help!
Mr Chivers , don’ t know if you will read my comment which is 3 days after your article , but here goes…..
The German news “100 Sekunden” says that a thousand strong sample of the population in Munich will soon be tested for antibodies in respect of the corona virus. Perhaps that will give an idea of the true prevalence of the spread of the virus. The newscast did not say which test was to be used , nor when the results will be available . According to a different item in the same newscast , the Robert Koch Institute says the disease is slowing due to Social Distancing and due to the restriction on going out .
Don’t know if this helps . …..
Isn’t is the case that the models are dynamic rather then static given the inputs put in it.
The early information from china was that the disease was mainly effecting the older population, over 70s, with a mortality rate of around 11%. This must have been the original input in the UK team strategy.
The inputs from Italy and now here in UK is that the age range of the severe COVID-19 patients is younger then anticipated in a model and if this input is added to the model then the model will say something else then the strategy the team started with.
Imperial are using code that is 13 years old and no one else has seen it?
That sounds a lot like the climate alarmists approach to models. Come to think about it, this article is interesting for pointing up all the problems of definition, data reliability and modelling. The climate alarmists seemed to have no such difficulty when modelling the eath’s atmosphere and oceans and using data from multiple measuring devices and types from several millenia ago to the present day. They claim to have been able to reliably tell us what the climate will be in a hundred years time down to tenths of a degree.
“All models are wrong, some models are useful in certain specific circumstances” is the starting phrase of any half decent modelling course. Our gov’t seems to think its approach to modelling is world leading. I suspect they are wrong about that, But given the daily repeated claim that they are following the science (not just the model, but mainly the model) to do the right thing at the right time, undermining that assertion would do far more harm to decision making (what little we have actually seen, other than deciding it is too early to decide) and more importantly public confidence in the decisions that are made. Maintaining public confidence in and therefore broad compliance with what is eventually decided is going to be even more important as we try to find our way out of this maze..
Another thing to factor in is how accurate are the tests? Many different tests looking for different aspects of the virus are being used around the world. I understand WHO is still evaluating a large number which have been submitted for evaluation – proper evaluation takes time. I would suggest that a lot of the early testing was done using tests of dubious accuracy. With the push to achieve more testing in UK, has the standard for accuracy been relaxed? The much hoped for and publicised antibody test does not seem to have made the grade and it has all gone deathly quiet on that front. Though I understand some limited random surveillance testing has occurred.
It need to be remembered that there are lots of Coronaviruses in circulation all the time. The term describes the shape – it looks like a crown in 2D. Are many of the virus tests detecting specifically CV 19, or are many of them also detecting some or all of the many other Coronaviruses as well? Will there be a similar problem with antibodies?