by Tom Chivers
Thursday, 26
November 2020

I am utterly confused about the Oxford vaccine

by Tom Chivers
A scientist at Oxford University’s Jenner Institute and Oxford Vaccine Group. Photo: John Cairns

I am extremely confused, and I want you to be confused too. Normally I like to make things clearer for readers, but today I don’t think I reasonably can.

I wrote on Tuesday that we ought to be excited about the Oxford/AstraZeneca vaccine; I still think that’s true, but I have a lot of questions that I would like answered.

In my piece I wrote that some participants were given a lower dose of the vaccine, then a standard dose booster, unlike most who got a standard/standard regimen. And people who were on the LD/SD were much better protected than those on SD/SD: 90% efficacy to 62%.

When you chop the data up into “subgroups” like this, you have to be careful that you’re not “noise-mining”. Say your drug doesn’t work overall, so you look at smaller and smaller subgroups until you find that a tiny set who happen to have done better, such as elderly hispanic women — but it was just a fluke result that you only got by torturing the data. For an intuitive explanation, see here.

This problem is much less likely if you specify in advance what you are looking at. I was told the trial did specify that in its protocol, but in that protocol the only mention of LD/SD comes on page 186: “Only participants who received two doses will be included (LD/SD or SD/SD)”. None of the tables seem to mention it that I can see.

I was assured that it was prespecified, but I really want to see that written down somewhere; I have looked at various Ox/AZ protocols (1, 2, 3, 4) and cannot find a simple timeline of what was specified when, and the regimen to have stemmed from an initial accident in which some people were given the lower dose. The excellent Jen Rogers, a clinical trials statistician at the consultancy Phastar, says that it may be an ad-hoc analysis done after the data was in — that is, at risk of noise-mining.

(Weirdly, in the Clinical Trials registry in the US, there is a mention in the earlier Phase 2/3 trial protocol of people having a full dose then a smaller booster, but not the other way around: see Group D here.)

Also, it seems that the group on the LD/SD regimen were all under the age of 55, which — obviously — is not the group most at risk. Further, because the LD/SD group was smaller, there was more uncertainty about the results, which is fine.

Partly this is the problem of science by press release, and as I understand it that’s a hard problem to avoid, because pharma companies are required to inform the market of upcoming results to prevent insider trading. That’s why we only saw headline results from Pfizer and Moderna as well. Oxford/AstraZeneca say full results will be available soon, perhaps early next week; also, the data only went up to the 3rd November, and more has come in since which may make things clearer.

I think we should still be excited. The trial was monitored by two independent statisticians at Oxford, AstraZeneca’s own statisticians, and then a separate Data Safety and Monitoring Board. (Incidentally, someone mentioned to me a few weeks ago that there was something weird with the DSMB on the Ox/AZ trial: unusually there appear to be two. I don’t know if that’s significant.) They have also been submitting data to the regulators as they go along. So it should be hard to sneak any really weird stuff through.

But I really would like to see the full paper, because I am utterly confused. And I hope you are too, because I think that’s the appropriate response.

Join the discussion

  • December 31, 2020
    You would have to be able to follow the maths, but you would do a calculation something like "What are the chances of observing a cases in the placebo arm and b cases in the vaccine arm if the vaccine has no effect". Then stop the trial when a+b is sufficiently large to give a 99% probability. a+b... Read more

  • December 4, 2020
    I'm following David Speigelhalter who points out that the same labs do the ONS tests as do the other tests, so the rate of all positives on the ONS tests is an upper bound on the false positive rate. Fullfact's This post telling healthy people to stop getting tested is full of inaccuracies gives... Read more

  • December 3, 2020
    I find your opinions depressing. Rightly or wrongly, people in westernised society now anticipate living into their 80s. That is largely in part due to the suppression of diseases that killed healthy children and adults in the centuries before this one. Take all vaccines out of the equation and... Read more

To get involved in the discussion and stay up to date, become a registered user.

It's simple, quick and free.

Sign me up