What Covid tests can we trust? A ferocious row has broken out among scientists over the reliability of the lateral flow method

The tests

Here is the issue. Testing people for Covid using the standard method, a lab-based polymerase chain reaction (PCR) test, has some great strengths and some profound weaknesses. Its strength is its accuracy; or, more precisely, its sensitivity. There are two kinds of accuracy: sensitivity and specificity. Sensitivity is a measure of how likely it is to detect positive cases, rather than giving false negatives; specificity is how likely it is to correctly identify disease-free people, rather than giving false positives.

PCRs are extremely good at both. For all the mad talk of a false-positive epidemic, the false-positive rate of PCRs must be very low, because for a long time in the summer, the total number of positives – false and true – was around the 0.05% mark. The false positive rate must be lower than that.

The PCR is ferociously good at spotting the virus when it is there. It works by detecting the SARS-Cov2 genome; to oversimplify enormously, it finds small segments of the viral RNA and forces them to replicate until there are enough to detect by more conventional methods.

It works in cycles. Each cycle roughly doubles the number of RNA molecules. If there was lots of viral RNA in the original sample, then it won’t take all that many cycles to “amplify” it to detectable levels. If there is a tiny amount, it will take more cycles. But, in theory, it can detect even a single RNA molecule. You just need to do more cycles. 

Those are its strengths. But it has weaknesses, too. One of them is prosaic, which is that the PCR tests we use usually take two days or more to get to a lab, be processed, and return a result. That’s two days in which someone is either self-isolating, perhaps unnecessarily, or not self-isolating, perhaps dangerously.

The other, though, is an inverse of its strength. PCR tests are amazing at picking up tiny amounts of viral RNA. But if we want to tell who is infectious and needs to be isolated, we need to detect active virus.

“In peak infection period, you will have literally trillions of virus particles in your nose,” says Michael Mina, a professor of epidemiology at Harvard University, and one of the protagonists in the row that I mentioned above. A PCR of a swab taken then will detect the viral RNA easily, with only a few cycles.

But after that, your immune system gets to work, tearing the viruses to pieces, a job it completes in a few days. As it kills each one, it leaves between one and 20 RNA molecules floating around in your body. Those molecules are not capable of replicating in your body or infecting anyone else. But they can be detected by PCR. It takes “weeks or months” for the debris to clear, says Mina. “The average duration people are PCR positive is about 25 to 35 days,” he says. “The average time people are infectious is between four and eight days.” Most of the people walking around with PCR-detectable viral RNA in their nasal passages are post-infectious.

This doesn’t matter a huge amount when you’re testing symptomatic people whom you expect to be positive. But it is a real problem if you’re trying to find asymptomatic cases and get them to isolate.

This is where lateral flow tests (LFTs) come in. Unlike PCR, they don’t test for viral RNA but particular proteins made by the virus. Instead of taking two days to get a result, they take half an hour or so. They’re also cheap and easy to mass-produce; some brands are simply strips of paper, like Litmus tests. The idea is that you make them by the million, and test people at mass testing stations or before they enter workplaces. 

But what’s crucial — either the crucial design flaw or the crucial design feature, depending on whom you ask — is that LFTs only return a positive result for about four to eight days in the infection cycle. If their proponents are right, that means they only pick up infectious cases, and it could mean that you – or your child – could have a test in the morning, wait 30 minutes for the result, and then go to work, or school, or university classes, or whatever. “The idea is to move on to event screening,” says Tim Peto, a professor of medicine at Oxford University who has been involved in the testing of LFTs in the UK. “You have a test 24 hours before and if it works you can go to the football match.”

The impacts

If Mina and Peto are right, it could be transformative. Mina envisages a large proportion of the population testing itself regularly — in a Time article he suggests 50% of the population every four days. That would break enough chains of transmission to reduce the R value below one, so that the pandemic would die down to manageable levels and we could get our lives back to normal.

Others, though, disagree. They say that the tests are gigantically unreliable; that they do indeed miss non-infectious cases, but that they also miss a huge percentage of dangerous, infectious cases. And that will mean that people will be given false reassurance, and will unknowingly carry the virus with them into the places they now have the permission — or confidence — to visit.

Both sides agree that LFTs are much less likely to detect dead virus floating about in the system. But what is contested is whether they reliably detect people in whom there is real virus at infectious levels.

Mina and Peto say that above a reasonable level of viral load, it will detect the large majority of cases. But Jon Deeks, a biostatistician at the University of Birmingham, disagrees, and has done so loudly. 

Deeks has repeatedly argued that LFTs will miss a large — perhaps overwhelming — percentage of positive cases. He and other senior academics, such as Prof Sheila Bird and Prof Sylvia Richardson, both of the Cambridge Biostatistics Unit, say that it missed more than a third of cases that have high viral loads in a mass testing trial in Liverpool. Other scientists tend to agree that the false negative issue is real. 

Deeks also ran a study of his own, testing 7,000 students in Birmingham with LFTs: they found just two positive cases, a startlingly low positive rate in December when the virus was circulating widely. They then took a random sample of 10% of the negative cases, tested them again with PCR, and found six cases. Deeks extrapolated from that to say that there were probably 60-ish cases in the full sample, so, he says, the LFTs had just a 3% sensitivity rate.

Mina, on the other hand, says this is a straightforward misunderstanding. First, he says, the Liverpool study doesn’t show what Deeks thinks it does. The high viral loads Deeks mentions in the Liverpool case are determined by how many PCR cycles are required, but, says Mina, you can’t directly compare those values between different PCR tests, and the Liverpool numbers are very unusual. He expounds on that in this Twitter thread.

And in the Birmingham study, says Mina, there’s another misunderstanding. None of the six cases that LFT missed but PCR caught had high viral loads; they were all low and probably non-infectious. This was, he says, the LFT doing exactly what it is intended to do

Both Mina and Peto are insistent that the LFTs should not be compared to PCR, because they’re simply doing different things. “We don’t want the lateral flow to be 100% effective,” says Peto. They only want it to detect people with a high viral load, because the rest probably won’t be infectious.

Deeks doubts that – “these tests are not designed to tell if you’re infectious or not,” he says – and points out that there is no straightforward cutoff between infectious and non-infectious; it’s a continuum of risk. He also says that Peto et al, in their studies with Porton Down using the LFTs in schools, never checked any of their negative results against PCR to get a clear picture of the false-negative rate. 

Test and release

The other question is what the tests should be used for. The manufacturers say that they should be carried out by experts, and that they should be used for determining the prevalence in the population and for detecting symptomatic cases. But they are being used to screen pupils going to school, and students going to university.  The MHRA has given emergency use exemption to allow healthcare workers to do their test at home. Comments from the Prime Minister suggest that they will be used to allow people to attend weddings, theatres and sporting venues.

This isn’t just Boris Johnson’s usual overpromising. Peto and Mina both think that this is exactly the sort of thing they should be used for. “The idea is to have a more focused approach where you only lock down people who’ve tested positive,” says Peto. 

But others think this “test and release” is dangerous. “They’re certainly good enough for a lot of purposes, like prevalence studies,” says Dr Rob Wootton, a scientist with many years experience developing point of care diagnostics. “But they are no damned good for test and release. That is not what they are for.” Deeks points out that a low viral load could be because you had the virus a week ago and are now recovered – or it could be because you’ve just got the virus and you will be infectious tomorrow. Peto and Mina say that testing every two days would minimise that problem, and besides a PCR test would take two days to get a result to you anyway.

And Deeks, Wootton and others worry about false-negative results leading to false assurance. “Watching the news after the Liverpool pilot study,” says Deeks, “there were people coming out of the test saying they’ll visit the care homes to see their mother.” 

Costs and benefits

This is a complex, technical question. Credentialled experts on each side disagree not just on what will happen in the future, but on the facts of the matter now.

The question at issue is: should we roll out mass home testing using LFTs, and let people with negative results out into society? It’s a hard question, but here is my best effort to unpick it, from my non-expert position. What we need to do is look at this not as a rigorous scientific question but as a cost-benefit analysis, decision-making under uncertainty. 

In the debit column is the risk that a false negative will increase the likelihood of someone going out and spreading the disease. 

In the credit column there are two entries. First, the likelihood that the LFTs will spot an otherwise undetected case and keep them inside, stopping further transmissions. And second, the rapid turnaround of the tests allowing people back into economic life and restarting society faster.

I’m less concerned about risk compensation behaviour than are Deeks and others. It was floated early in the pandemic as a reason not to wear masks, but a literature review in the BMJ found that the evidence in other contexts was weak. I had a look myself for research on false negative results providing false reassurance, for instance in cancer screening, and again found no real support for it (1, 2, 3). I’m not hugely confident, but that’s my best guess.

On the other hand, individual risk compensation behaviour isn’t the whole issue. If institutions are explicitly saying that a negative LFT result gives permission to engage in more Covid-risky behaviour, such as attending schools or universities, whether or not it makes people individually more likely to lick lampposts. And since that is what is happening, it seems very likely that false negatives will lead to some non-trivial increase in risk spread.

The trouble is that I can’t assess how common those false negatives will be. Deeks says common, even in high viral loads; Peto and Mina say rare. Both agree that they’ll be lower in lab settings than when used by trained professionals, and lower when used by professionals than when self-administered. But Deeks thinks that’s crucial, while Peto is largely unfazed. “It’s like boiling an egg,” he says. “You won’t get it right the first time. But after a week the error rate goes down.” Mina agrees: “It doesn’t have to be perfect. It just has to get R below 1.”

The next question is whether the number of asymptomatic cases caught will outweigh the cost of the false negatives. Peto thinks it’s a clear yes: “We’ve only had two million used,” he says, “and found at least 30,000 asymptomatics, so it’s already reduced the number.” Mina says that in some cases, people stay infectious for longer than usual, and PCR can’t tell – you’d still be PCR-positive after 10 days’ isolation, so there’s no point testing again. But in his studies in Harvard, LFTs find those rare people and get them to stay isolated. Mass testing in Slovakia has been credited with possibly reducing infection rates there.

Deeks, though, says that the issue is that comparison shouldn’t be between getting tested or staying out in the community. “The comparator is what you used to do,” he says. In the case of schools, if you sit next to someone who gets a positive PCR test, you are required to self-isolate. “Now they stay in the classroom, and if they get a false negative they’ll infect the person next to them and you get an outbreak in the school,” he says. He thinks this sort of mechanism was behind an outbreak at the Jaguar-Land Rover plant in Halewood, Liverpool.

And finally, the question is how useful shortening the quarantine will be. Peto and Mina think it’s vital; Deeks, obviously, less so. It strikes me that with the vaccine on the way, the calculations change somewhat, and it may be that the pressure to get the economy started in any way possible is less dramatic than it used to be.

Imperfect solutions

Dr Alexander Edwards, a biomedical engineer at Reading University, agreed with Deeks that the false negative rate probably is quite high, especially when self-administered. But, he said: what’s the alternative?

“You can take two approaches,” he said. “One, push on and do what you can, or two, you can wait for a better solution. But it’s difficult to see how a better solution will appear magically.” Lateral flow tests are “extraordinarily good” for what they are: scalable, cheap, easily mass-produced. They will miss cases, but we’re not going to be able to build anything better quickly. In situations like HIV, he says, LFTs proved useful as part of a wider testing toolkit. He added that Liverpool’s public health authorities have said that they won’t use them to allow people to visit care home residents: that seems the sort of sensible measure that can prevent the worst harms even if the tests do lead to some further spreading.

Deeks’s concerns have got attention. Mina says that he is constantly fielding calls from government officials worrying that the 1.5 billion LFT tests that they plan to buy are a bad idea. “I advise the government in the US,” he says. “And because of Jon I’m getting call after call from people saying that they’re seeing tests of 3% sensitivity. It’s confusing people the world over.”

There isn’t time, in a pandemic, to do RCTs on everything. Sometimes decisions have to be made quickly; an imperfect decision taken quickly is better than a perfect one two weeks later. Deeks, though, thinks that it would be easy to do good tests, quickly: “In a pandemic, it’s easy to do studies, because you have enough patients.” He wants more testing, with PCR checks for false negatives, before full rollout.

Deeks is absolutely right that testing should carry on, but delaying the rollout until then seems too cautious. I think the Slovakian experience suggests that mass testing can be effective; the possible benefits of catching asymptomatics seem significant, and there are real advantages over PCR testing alone. But it would be mad to use LFTs as a passport to go and visit care homes; avoiding obvious pitfalls like that should reduce harms.

Edwards compares the situation to a hand of poker. The reason that it’s tense is because the stakes are high – getting it wrong could kill thousands – and because both players have good hands; there are excellent reasons to push forward, and excellent reasons to be cautious. And, as in poker, we have to operate with imperfect information. When the cards are turned over, we will learn who was right.

His bet, and mine, is on rolling out the lateral flow tests, with caution and pragmatism and constant evaluation. “We’re in an urgent situation, with huge health and economic costs,” he says. “I genuinely don’t think there’s a better alternative.”

---