by Tom Chivers
Thursday, 26
November 2020

I am utterly confused about the Oxford vaccine

by Tom Chivers
A scientist at Oxford University’s Jenner Institute and Oxford Vaccine Group. Photo: John Cairns

I am extremely confused, and I want you to be confused too. Normally I like to make things clearer for readers, but today I don’t think I reasonably can.

I wrote on Tuesday that we ought to be excited about the Oxford/AstraZeneca vaccine; I still think that’s true, but I have a lot of questions that I would like answered.

In my piece I wrote that some participants were given a lower dose of the vaccine, then a standard dose booster, unlike most who got a standard/standard regimen. And people who were on the LD/SD were much better protected than those on SD/SD: 90% efficacy to 62%.

When you chop the data up into “subgroups” like this, you have to be careful that you’re not “noise-mining”. Say your drug doesn’t work overall, so you look at smaller and smaller subgroups until you find that a tiny set who happen to have done better, such as elderly hispanic women — but it was just a fluke result that you only got by torturing the data. For an intuitive explanation, see here.

This problem is much less likely if you specify in advance what you are looking at. I was told the trial did specify that in its protocol, but in that protocol the only mention of LD/SD comes on page 186: “Only participants who received two doses will be included (LD/SD or SD/SD)”. None of the tables seem to mention it that I can see.

I was assured that it was prespecified, but I really want to see that written down somewhere; I have looked at various Ox/AZ protocols (1, 2, 3, 4) and cannot find a simple timeline of what was specified when, and the regimen to have stemmed from an initial accident in which some people were given the lower dose. The excellent Jen Rogers, a clinical trials statistician at the consultancy Phastar, says that it may be an ad-hoc analysis done after the data was in — that is, at risk of noise-mining.

(Weirdly, in the Clinical Trials registry in the US, there is a mention in the earlier Phase 2/3 trial protocol of people having a full dose then a smaller booster, but not the other way around: see Group D here.)

Also, it seems that the group on the LD/SD regimen were all under the age of 55, which — obviously — is not the group most at risk. Further, because the LD/SD group was smaller, there was more uncertainty about the results, which is fine.

Partly this is the problem of science by press release, and as I understand it that’s a hard problem to avoid, because pharma companies are required to inform the market of upcoming results to prevent insider trading. That’s why we only saw headline results from Pfizer and Moderna as well. Oxford/AstraZeneca say full results will be available soon, perhaps early next week; also, the data only went up to the 3rd November, and more has come in since which may make things clearer.

I think we should still be excited. The trial was monitored by two independent statisticians at Oxford, AstraZeneca’s own statisticians, and then a separate Data Safety and Monitoring Board. (Incidentally, someone mentioned to me a few weeks ago that there was something weird with the DSMB on the Ox/AZ trial: unusually there appear to be two. I don’t know if that’s significant.) They have also been submitting data to the regulators as they go along. So it should be hard to sneak any really weird stuff through.

But I really would like to see the full paper, because I am utterly confused. And I hope you are too, because I think that’s the appropriate response.

Join the discussion

  • The guy on Have I Got News for You, who’s Irish, just said it’s the most British thing he’s ever seen. They create an effective vaccine and everyone’s dying to pile in and see it fail.

    OK there are questions for sure, but in fairness Chivers should point out that the AZ results counted asymptomatics as well as the properly sick, which the Pfizer and Moderna did not. Oxford looked a lot harder for the cases.

    That alone could account for a difference and is at least as relevant. Chivers must know this. Why not mention?

  • Richard, it is not yet illegal to ask if some covid orthodoxy is inefficacious, although saying it is can get you prosecuted, and will get you banned from the usual social media platforms, and from The Guardian.

  • Indeed, but the issue is with a – dare I say it – misleading presentation which pushes up efficacy by averaging across two discrete groups. Even I with my meagre o level maths could work that out.

  • To get involved in the discussion and stay up to date, become a registered user.

    It's simple, quick and free.

    Sign me up