Amid Covid crisis we can’t wait for perfect data With so many people dying each day, we must make fast decisions as best we can

Perhaps my biggest mistake of 2020 was when I said I didn’t think the evidence for shutting borders was very good at the beginning of the pandemic. To make matters worse, I recently did it again, in an aside in a post about the new variant: I said that other countries shutting their borders to the UK now was probably “shutting a stable door on a long-bolted horse”.

Probably it is. But maybe it isn’t, and one thing I should have learnt by now is that, in this pandemic, the “maybe” is important. Maybe we can slow the spread of a new form of the variant; maybe that maybe is worth it. Last  March, we thought masks probably wouldn’t help – but we should have thought maybe they would. We need to think not in terms of certainties, but in terms of costs and benefits, gambles and payoffs. It’s an approach that’s all the more important as the vaccine starts to roll out. 

There was a piece in the New York Times about the British regime which illustrated this rather well. “Britain Opts for Mix-and-Match Vaccinations, Confounding Experts”, said the headline (later changed to “Britain Opens Door To…” etc). The British government, said the piece, was going to allow doctors to give patients a dose of one vaccine (for instance the Pfizer/BioNTech vaccine), and then a booster shot of a different one (say the Oxford/AstraZeneca one). This was, according to experts quoted, not a method supported by science: they “seem to have abandoned science completely now and are just trying to guess their way out of a mess”, said one.

British people, especially those with some understanding of how the British vaccination regime actually works, got understandably annoyed. “This is a complete misrepresentation,” said one fairly typical response; “fake news … false information spread by reputable media organisations”, was another.

That’s probably fair. As the NYT story did say, but didn’t make clear in its headline, the “mix and match” approach is a legitimate one of last resort: the guidelines tell vaccine administrators that “For individuals who started the schedule and who attend for vaccination at a site where the same vaccine is not available, or if the first product received is unknown, it is reasonable to offer one dose of the locally available product to complete the schedule.” 

It is not saying “Go to it lads, throw the vaccines around like you’re mixing drinks at a stag do”; it’s saying “If you can’t get hold of the vaccine you originally gave, it’s probably better to give a different one as a booster than none at all.” The BMJ has written to the NYT asking for a correction; the drug safety researcher Anthony Cox has written an excellent piece on why the story is unhelpful and dangerous.

So, bad NYT, do better, etc. But there’s a more interesting question: will the mix-and-match vaccinations actually work?

There’s a stock answer to this if you take a certain view of science and knowledge: they won’t. Or, at least, we don’t have any evidence to suggest that they will. It’s not “data driven”, says one expert quoted in the NYT piece. “There are no data on this idea whatsoever,” says another.

This is all true, of course. There have been no randomised controlled trials into whether a Pfizer prime and Ox/AZ boost works, or vice versa. And, traditionally, in medical science, if you haven’t got an RCT showing that your drug regimen works, then you haven’t got any evidence that it works.

But I want to argue that this sort of thinking doesn’t work in emergency situations like the one we’re in. More than that: it’s been a repeated problem throughout the pandemic that we have relied on this “we have the evidence/we don’t have the evidence” binary, meaning that we have moved slowly, waiting for rock-solid confirmation; but moving faster, making decisions on imperfect information, would have saved a lot of lives.

So what do we know? First, it’s worth noting that although the Pfizer and Oxford/AZ vaccines — and the Novavax, and the Moderna, and whatever else — all work through different mechanisms, at quite a fundamental level they all do the same thing. They all present the immune system with an antigen – a little thing that looks like a part of the virus, so that your body produces antibodies which will fight the real virus when it arrives. And in all cases, the antigen is the same thing: the spike protein, the little protrusions that come out from the surface of the virus.

In the case of the mRNA vaccines, such as Moderna’s, and Pfizer’s, the spike protein isn’t in the vaccine itself; the vaccine contains little lengths of RNA which get into your cells and tell them to produce it. The Oxford vaccine does something similar, but uses a genetically modified chimpanzee cold virus to get into your cells. Others, such as Novavax’s, use the spike protein itself. (You can read more about different kinds of vaccine here, should you wish.) But at base, they all work by making sure that your immune system will recognise the spike protein when it sees it for real.

So there are good theoretical reasons to think that a shot from one vaccine will work in combination with another. But there’s another reason as well: analogy from other vaccines. Stephen Evans, a professor of pharmacoepidemiology at the London School of Hygiene & Tropical Medicine, told me that “heterologous prime-boost vaccinations” — that is, mixing and matching the two doses — have quite a long history: they were first proposed in 1992.

“The idea has mainly been studied in diseases for which we do not have an effective vaccine,” he said. “In HIV or malaria, or in dengue, where there is a single vaccine but it’s not very good.” In many cases, heterologous vaccinations like this can be more efficacious than homologous ones, he says (and there are papers to back him up). Most of the time, you don’t really need to get your vaccine efficacy up, because most vaccines are amazingly effective, hence their use in situations where the vaccines don’t work very well.

Of course, all those other heterologous vaccinations are backed up with RCTs. “We clearly do not have randomised trial evidence confirming efficacy of this,” says Evans. 

But those other vaccinations aren’t being carried out against a backdrop of about 500 people dying every day in the UK alone from a disease for which we have a working vaccine. “This is not an ideal, to do it until we’ve done the trials,” says Evans. “But in a situation of crisis where we’re not able to get people vaccinated with the same dose of the vaccine, it seems a reasonable thing to do to go beyond where we have trial evidence.”

Dr Zania Stamataki, a viral immunologist at Birmingham University, told me much the same thing. “We wouldn’t do this if we weren’t in a dire situation,” she said. “If we didn’t need to vaccinate as many people as possible as quickly as possible to save lives, we’d stick to the on-label advice. But at the moment we don’t have the luxury.”

It’s worth emphasising that this will only happen in extremis: if, say, the Pfizer vaccine has run out locally, or someone’s records are lost. “We’re anticipating that we stick to the regimen, and only in situations where we’ve run out supplement it with other things,” says Stamataki. But both Stamataki and Evans say that it’s a reasonable thing to do in those extreme cases. “Because the [the spike protein antigen] is the same,” says Stamataki, “we anticipate seeing the same result.”

Of course, it may not work. Babak Javid, an immunology lab director at the University of California San Francisco, says that the Moderna vaccine and possibly the Pfizer one introduced a small mutation to the spike protein to “stabilise” it, and that as far as he knows the Oxford vaccine didn’t; so the antigen may not be exactly the same. Will that affect it? Perhaps. It’s vital that we carry on monitoring; in an ideal world we might even do some randomisation, saying that certain GP practices should give the mix-and-match dose to patients where the original vaccine is unavailable, while other practices leave them on single doses.

But the point is that even though we don’t have 20 RCTs and a meta-analysis showing that heterologous vaccinations work for Covid-19, that’s not the same as saying that we don’t have any knowledge or evidence at all. There are reasons, from theory and from analogy from other vaccines, that it should work fine. As Robert Wiblin, the director of research at the evidence-based charity 80,000 Hours, says in this wise Twitter thread, there is a widespread attitude among certain decision-makers that “if a study hasn’t been done on a particular question we have ‘no data’, and therefore no basis on which to form beliefs or act”. But that is flat wrong, and dangerous.

It’s that form of thinking which led people — not just ordinary people, but the UK Chief Medical Advisor and the US Surgeon General, among many others; top, highly credentialled experts — to say early in the pandemic not just that we don’t know if masks work, but that masks don’t work, and to advise against them. They were right that there weren’t any good RCTs. (There still aren’t, really.) But there were theoretical reasons to think that masks might work, and reasons from analogy: we all accept that surgeons wear them to prevent infection during surgery, for instance. And as social interventions go, masks are relatively low-cost, compared with nationwide lockdowns or closed schools. If you multiply the good it might do by the chance that it works, and subtract the costs, the expected value looks pretty good.

You can understand the “no RCTs, no evidence” attitude. Over the last century or so, medicine has established a norm where the only evidence that really counts is lots of RCTs. There is an excellent reason for that: doctors’ intuition as to what worked and what didn’t was often extremely flawed, because doctors are people. So it is very important to show that your new drug or your surgical intervention does more good than harm before you introduce it.

But we simply can’t afford to wait for that level of certainty. Evidence, a wise man once said, is like money: “Obviously it’s useful to have as much evidence as possible, in the same way it’s useful to have as much money as possible. But equally obviously it’s useful to be able to use a limited amount of evidence wisely, in the same way it’s useful to be able to use a limited amount of money wisely.”

That means looking at the evidence you have, and making your best guess at the costs and benefits of one course of action versus another course of action. The costs of, say, closing schools are considerable. Meanwhile, the benefits are very hard to be sure about – how much do they influence transmission? How much are teachers at risk? But we can’t wait for more evidence to come in; we have to make decisions now, as best we can.

Similarly, there’s been a bit of a hoo-hah about another British plan to delay the booster jab for up to 12 weeks, so that we can get as many people the first dose (and some level of protection) as possible. Again, there are no specific RCTs looking into that – Pfizer only tested its booster three weeks after the prime, although the Oxford study did look at various different regimens including up to 12-week gaps. But again, we don’t have the luxury of demanding certainty and perfect evidence. The question is: do we think that a longer gap will work reasonably well, and if so, do we think the benefits of giving many more people their first dose outweigh the possible costs in efficacy? The MHRA obviously thinks so, and my own suspicion is that it’s pretty clear-cut. “We haven’t got the data” is partially true, but, again, it doesn’t mean we know nothing.

And in the case of mix-and-matched vaccines, once again, it’s true that we don’t have an RCT that says they’ll work. But we have good reasons to think they’ll work; perhaps not as well as the on-label regimen, but well enough to save lives and prevent severe disease and hospitalisations. 

And we know that the likely costs are low — there’s no evidence of any safety issues with heterologous vaccinations, and if the alternative is simply not giving the patient a booster at all because you’ve run out of the original vaccine, then this seems a reasonable course. “I don’t know of any instance where the heterologous prime boost has led to particular harms or to lesser efficacy,” says Evans, “although it hasn’t always led to higher efficacy.” This seems an obvious bet, in the rare cases when the original vaccine is unavailable.

With a bit of luck, we’re near the end of this. I’m sceptical that the Government will meet its target of getting the first dose to the four most vulnerable groups by mid-February, but March should be doable (and Nilay Shah, a professor of chemical engineering at Imperial, told the Science Media Centre that it the February target is “ambitious” and “needs everything to click every day”, but that it is “achievable”). 

So we’re no longer talking about delaying Covid deaths; we are talking about actually preventing them, if we can keep people from getting the virus for a few more weeks. And that means making decisions quickly, making decisions that are good enough rather than perfect, making decisions with limited information rather than waiting for the perfect RCT that shows that giving the vaccine in a red room works just as well as giving it in a blue one.

The NYT’s concern about the vaccine dosing is understandable: it is simply applying the same standard that medical science applies all the time, under usual circumstances. But we aren’t under usual circumstances. In this case, it is inadvertently propping up a decision-making system which has meant that we didn’t introduce masks until too late and which meant that we didn’t lock down or close borders until the virus was already spreading.

I got it wrong by saying that border closures “probably” wouldn’t stop the new variant; I should have talked about how likely I thought it was, and what the costs and benefits would be if it did. The MHRA, by allowing mix-and-match vaccines and a 12-week gap, is thinking in those terms, and doing the right thing. You can’t “follow the data” when there is none: you have to use the information that you have.

---

Tom Chivers is a science writer. His second book, How to Read Numbers, is out now.

TomChivers