August 27, 2020 - 3:00pm
Here’s a nice, if somewhat terrifying, demonstration of how easily numbers can go wrong if you’re not careful. Even if you’re the commissioner of the US Food and Drug Administration who — you’d think – would know the difference between absolute risk and relative risk, and percentage increases and percentage point increases, and so on.
The FDA has issued an emergency approval for “convalescent plasma” as a treatment for Covid-19. Convalescent plasma is basically the blood of people who’ve got better from the disease, with the actual blood cells taken out. The plasma contains antibodies for coronavirus and, it is hoped, will help fight the disease in the patient’s body.
It’s all a bit speculative, as I understand it, but it might be of some good. Where it went a bit wrong was when Stephen Hahn, the FDA commissioner announced that 35% of patients “would have been saved” because of the administration of plasma. President Trump said this was a “tremendous” number.
Kevin McConway, a professor emeritus of statistics at the Open University, pointed out to me that this was a “classic absolute versus relative error”. The study injected some patients with blood plasma that had lots of antibodies in it; as a control, they gave other patients plasma with much fewer antibodies in. After seven days, about 11% of the control patients had died, but only 7% of the patients given the high-antibody plasma had.
So the relative risk for the patients given the high-antibody plasma was about 35% lower than those given the low-antibody plasma. But the absolute risk was only 4% lower. If you were in the study, and were given Good Plasma instead of Bad Plasma, there was a 4% chance — not a 35% chance — that it would save your life. That is — if confirmed — a very respectable and important number, but a very different one.
There are other issues: notably, it’s a small study looking at a subset of people — hospitalised patients under 80 who weren’t ventilated; also, after 30 days, the death rates were 21.6% and 26.7%, which changes both the absolute and relative risk calculations, but more importantly show that whether you survived the first seven days or not is very much only part of the story.
I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified. What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.
— Dr. Stephen M. Hahn (@SteveFDA) August 25, 2020
But fundamentally, it’s a misunderstanding that a ‘35% reduction in risk’ is a very different thing from ‘saves 35% of patients’. In almost all situations, it’s really important to give the absolute risk — or, even better, talk in terms of how many people it would affect (in this case, four out of every 100).
Hahn, to his credit, admitted the error (see above). But it’s a very clear demonstration of how even clever, numerate people can make these mistakes, and mislead everyone without meaning to. Someone really ought to write a book about it.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeWhen I worked in local radio, management used the same trick to manipulate listener figures. If listening share fell from from 20% to 15% it was announced as a 5% drop. If figures rose 20% to 25% it was a 25% rise
This misinterpretation is often made and can apply both ways.
For example, in the EU referendum the 52/48 result has generally been reported to mean that 4% (52-48) more people voted to leave than to remain. However 52% voted to leave which is 8% (52/48×100), not 4%, more than the 48% who voted to remain.
It’s always necessary to be clear about what something is a percentage of.
One has to be a bit careful here. I was unable to click through and see the exact context of the “would have saved 35% of patients” comments. Given a 35% reduction in mortality, it would be accurate to say that “35% of deaths would have been avoided”. Thus, if you were in the study and were given Good Plasma, the chance that the treatment would save your life is only 4%, but your chance of dying has decreased 35% (down to 7% from 11%). If lives saved is our main metric, that is still a very significant decrease. (Of course, there are many many many caveats with a study like this, so I would be very doubtful if the true effect ended up being that large.)
I agree, Hahn’s initial statement sounds correct to me. Presumably he meant 35% of people who died would have been saved had they been treated with plasma (it would make no sense to say that 35% of people who didn’t die would have been saved). So putting aside other potential issues with the experiment, if say 110,000 people have died without plasma and we equate these with 11% of those infected, then with plasma only 7% of those infected, or 70,000 people, would have died. That is, 40,000 people saved, about 35% of the 110,000 who died.
So Hahn’s mistake was in admitting to a mistake he hadn’t made. On the other hand, if even someone like Tom Chivers, who writes about this stuff, can make such a mistake, then Tom has thus proved his own point even more categorically than he intended.
Both the above comments are another lesson in abuse of statistics.
I would, quite honestly, like to know what part of what I wrote constitutes a misuse of statistics. I completely agree with the other comments posted here, especially that of Mike Ferro. When one uses “%”, we must be clear what something is a percentage of. For example:
Let’s say the population of a country A is 10 000 000. Of those, 100 000 become sick and 11 000 die (the 11% figure from the study), as country A has “Bad Plasma”.
Country B, which has “Good Plasma” also has a population of 10 000 000, with 100 000 who become sick, and 7 000 who die.
Country B experiences 4000 fewer deaths, due to the difference in treatment. Is that better explained by:
a) Claiming an improvement in treatment of 35% relative to country A? (35% deaths/deaths)
b) Claiming an improvement in treatment of 4% relative to country A? (4% deaths/sicknesses)
c) Claiming an improvement of 0.04% relative to country A? (0.04% deaths/population)
None of these is incorrect (unless I have made a math error). Rather, the `misuses of statistics’ comes from using one of these numbers to imply something it doesn’t mean. It is reasonable to argue which statistic is most relevant in a given situation. It is thus a question of statistical literacy, which was Mr. Chivers original point.
(Sorry for the long post – I hope this clarifies what I was saying above.)
How is my comment an abuse of statistics? Or indeed Philip Rempel’s?
Whether you can say with any certainty from the experiment how many people would be saved in real life, and after how many days, is another matter. But to say that 70k people dying instead of 110k can be described as 35% of people being saved looks OK to me. It seems obvious he means 35% of those who died.
Of course ideally everyone would always say as a percentage of what whole one us talking about, especially when it’s not obvious.
Where Tom Chivers is right is that as most of these patients were going to survive anyway, the chances of survival increased only from 89% to 93%, which is where his 4% number comes from. With the same relative risk, if the disease had been much more deadly, the benefit would be more than 4%. Imagine that you have a 53% chance of dying with the control, and this is reduced to 35% on the high-titre plasma (ie by 34%). Then the chances of survival would be increased in the ratio (100-35)/(100-47) for an improvement of 38%.
Possibly of interest, there was a refinement to the study, quoted on p11 of the FDA Emergency Use document, that showed a 45% reduction in 7 day mortality for high-titre plasma relative to the low-titre control, for patients who were treated wtihin 72 hours and not intubated.
You may have overlooked after 30 days, the death rates were 21.6% and 26.7% as opposed to 7% and 11% after 7 days. So the chance of dying after 7 days had decreased by 35%, not the undated chance of dying (of COVID) rather than of recovering.
So no, according to the article it would not have been accurate to say that “35% of deaths would have been avoided”.
Thank you for the correction. However, I believe the point I was making stands regardless of whether the statistic is a decrease of 35% or 20% (from 26.7% down to 21.6%) or some other value (as my last sentence indicates, I would be doubtful about drawing broad conclusions from the study for many reasons). I was merely pointing out that the question was of relative vs. absolute risk, and that either one can be misleading and that absolute risk is not necessarily a better statistic to use.