Even numerate people make mistakes — and mislead without meaning to
Here’s a nice, if somewhat terrifying, demonstration of how easily numbers can go wrong if you’re not careful. Even if you’re the commissioner of the US Food and Drug Administration who — you’d think – would know the difference between absolute risk and relative risk, and percentage increases and percentage point increases, and so on.
The FDA has issued an emergency approval for “convalescent plasma” as a treatment for Covid-19. Convalescent plasma is basically the blood of people who’ve got better from the disease, with the actual blood cells taken out. The plasma contains antibodies for coronavirus and, it is hoped, will help fight the disease in the patient’s body.
It’s all a bit speculative, as I understand it, but it might be of some good. Where it went a bit wrong was when Stephen Hahn, the FDA commissioner announced that 35% of patients “would have been saved” because of the administration of plasma. President Trump said this was a “tremendous” number.
Kevin McConway, a professor emeritus of statistics at the Open University, pointed out to me that this was a “classic absolute versus relative error”. The study injected some patients with blood plasma that had lots of antibodies in it; as a control, they gave other patients plasma with much fewer antibodies in. After seven days, about 11% of the control patients had died, but only 7% of the patients given the high-antibody plasma had.
So the relative risk for the patients given the high-antibody plasma was about 35% lower than those given the low-antibody plasma. But the absolute risk was only 4% lower. If you were in the study, and were given Good Plasma instead of Bad Plasma, there was a 4% chance — not a 35% chance — that it would save your life. That is — if confirmed — a very respectable and important number, but a very different one.
There are other issues: notably, it’s a small study looking at a subset of people — hospitalised patients under 80 who weren’t ventilated; also, after 30 days, the death rates were 21.6% and 26.7%, which changes both the absolute and relative risk calculations, but more importantly show that whether you survived the first seven days or not is very much only part of the story.
I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified. What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.
— Dr. Stephen M. Hahn (@SteveFDA) August 25, 2020
But fundamentally, it’s a misunderstanding that a ‘35% reduction in risk’ is a very different thing from ‘saves 35% of patients’. In almost all situations, it’s really important to give the absolute risk — or, even better, talk in terms of how many people it would affect (in this case, four out of every 100).
Hahn, to his credit, admitted the error (see above). But it’s a very clear demonstration of how even clever, numerate people can make these mistakes, and mislead everyone without meaning to. Someone really ought to write a book about it.