November 26, 2020 - 3:32pm

I am extremely confused, and I want you to be confused too. Normally I like to make things clearer for readers, but today I don’t think I reasonably can.

I wrote on Tuesday that we ought to be excited about the Oxford/AstraZeneca vaccine; I still think that’s true, but I have a lot of questions that I would like answered.

In my piece I wrote that some participants were given a lower dose of the vaccine, then a standard dose booster, unlike most who got a standard/standard regimen. And people who were on the LD/SD were much better protected than those on SD/SD: 90% efficacy to 62%.

When you chop the data up into “subgroups” like this, you have to be careful that you’re not “noise-mining”. Say your drug doesn’t work overall, so you look at smaller and smaller subgroups until you find that a tiny set who happen to have done better, such as elderly hispanic women — but it was just a fluke result that you only got by torturing the data. For an intuitive explanation, see here.

This problem is much less likely if you specify in advance what you are looking at. I was told the trial did specify that in its protocol, but in that protocol the only mention of LD/SD comes on page 186: “Only participants who received two doses will be included (LD/SD or SD/SD)”. None of the tables seem to mention it that I can see.

I was assured that it was prespecified, but I really want to see that written down somewhere; I have looked at various Ox/AZ protocols (1, 2, 3, 4) and cannot find a simple timeline of what was specified when, and the regimen to have stemmed from an initial accident in which some people were given the lower dose. The excellent Jen Rogers, a clinical trials statistician at the consultancy Phastar, says that it may be an ad-hoc analysis done after the data was in — that is, at risk of noise-mining.

(Weirdly, in the Clinical Trials registry in the US, there is a mention in the earlier Phase 2/3 trial protocol of people having a full dose then a smaller booster, but not the other way around: see Group D here.)

Also, it seems that the group on the LD/SD regimen were all under the age of 55, which — obviously — is not the group most at risk. Further, because the LD/SD group was smaller, there was more uncertainty about the results, which is fine.

Partly this is the problem of science by press release, and as I understand it that’s a hard problem to avoid, because pharma companies are required to inform the market of upcoming results to prevent insider trading. That’s why we only saw headline results from Pfizer and Moderna as well. Oxford/AstraZeneca say full results will be available soon, perhaps early next week; also, the data only went up to the 3rd November, and more has come in since which may make things clearer.

I think we should still be excited. The trial was monitored by two independent statisticians at Oxford, AstraZeneca’s own statisticians, and then a separate Data Safety and Monitoring Board. (Incidentally, someone mentioned to me a few weeks ago that there was something weird with the DSMB on the Ox/AZ trial: unusually there appear to be two. I don’t know if that’s significant.) They have also been submitting data to the regulators as they go along. So it should be hard to sneak any really weird stuff through.

But I really would like to see the full paper, because I am utterly confused. And I hope you are too, because I think that’s the appropriate response.

Tom Chivers is a science writer. His second book, How to Read Numbers, is out now.