by Tom Chivers
Thursday, 27
August 2020

A lesson in statistics… for the FDA

by Tom Chivers
President Donald Trump listens to FDA Commissioner Stephen Hahn, who made a statistical error earlier this week

Here’s a nice, if somewhat terrifying, demonstration of how easily numbers can go wrong if you’re not careful. Even if you’re the commissioner of the US Food and Drug Administration who — you’d think – would know the difference between absolute risk and relative risk, and percentage increases and percentage point increases, and so on.

The FDA has issued an emergency approval for “convalescent plasma” as a treatment for Covid-19. Convalescent plasma is basically the blood of people who’ve got better from the disease, with the actual blood cells taken out. The plasma contains antibodies for coronavirus and, it is hoped, will help fight the disease in the patient’s body.

It’s all a bit speculative, as I understand it, but it might be of some good. Where it went a bit wrong was when Stephen Hahn, the FDA commissioner announced that 35% of patients “would have been saved” because of the administration of plasma. President Trump said this was a “tremendous” number.

Kevin McConway, a professor emeritus of statistics at the Open University, pointed out to me that this was a “classic absolute versus relative error”. The study injected some patients with blood plasma that had lots of antibodies in it; as a control, they gave other patients plasma with much fewer antibodies in. After seven days, about 11% of the control patients had died, but only 7% of the patients given the high-antibody plasma had.

So the relative risk for the patients given the high-antibody plasma was about 35% lower than those given the low-antibody plasma. But the absolute risk was only 4% lower. If you were in the study, and were given Good Plasma instead of Bad Plasma, there was a 4% chance — not a 35% chance — that it would save your life. That is — if confirmed — a very respectable and important number, but a very different one.

There are other issues: notably, it’s a small study looking at a subset of people — hospitalised patients under 80 who weren’t ventilated; also, after 30 days, the death rates were 21.6% and 26.7%, which changes both the absolute and relative risk calculations, but more importantly show that whether you survived the first seven days or not is very much only part of the story. 

But fundamentally, it’s a misunderstanding that a ‘35% reduction in risk’ is a very different thing from ‘saves 35% of patients’. In almost all situations, it’s really important to give the absolute risk — or, even better, talk in terms of how many people it would affect (in this case, four out of every 100).

Hahn, to his credit, admitted the error (see above). But it’s a very clear demonstration of how even clever, numerate people can make these mistakes, and mislead everyone without meaning to. Someone really ought to write a book about it.

Join the discussion

  • August 28, 2020
    How is my comment an abuse of statistics? Or indeed Philip Rempel's? Whether you can say with any certainty from the experiment how many people would be saved in real life, and after how many days, is another matter. But to say that 70k people dying instead of 110k can be described as 35% of people... Read more

  • August 28, 2020
    Thank you for the correction. However, I believe the point I was making stands regardless of whether the statistic is a decrease of 35% or 20% (from 26.7% down to 21.6%) or some other value (as my last sentence indicates, I would be doubtful about drawing broad conclusions from the study for many... Read more

  • August 28, 2020
    I would, quite honestly, like to know what part of what I wrote constitutes a misuse of statistics. I completely agree with the other comments posted here, especially that of Mike Ferro. When one uses ``%'', we must be clear what something is a percentage of. For example: Let's say the... Read more

To get involved in the discussion and stay up to date, join UnHerd.

It's simple, quick and free.

Sign me up