March 10, 2021 - 11:36am
I’ve been assessing medical tests for thirty years, and Covid testing is no different — it’s always a game of probabilities. The current mass testing regime for children in schools simply doesn’t pass by the numbers.
The fitness for purpose is a combination of the sensitivity of the test (what percentage of infected cases it correctly identifies) and the specificity (what percentage of uninfected people it correctly says are negative). But it also depends on how, where and when a test is used.
Different studies of the UK Innova lateral flow test (the test being used in schools) have reported variously that its sensitivity is 78%, 58%, 40% and even 3%. The higher 78% and 58% figures come from using the test among people with symptoms, the lower 40% and 3% figures come from using it for mass testing among people without symptoms, as is being done in schools. (And none of these studies have assessed how well the test detects infection in children). So although the test can pick up people who have the infection, it will miss quite a few — so there is a risk that disinhibition after a negative test could actually exacerbate case numbers if children incorrectly think they are safe and the rules no longer apply.
But the more concerning aspect are the false positives, related to the specificity. The original Government studies found only around 3 in 1000 people were getting false positives, and this dropped to 1 in 1000 in the Liverpool study. Doesn’t sound like a lot, right?
But consider the problem from the perspective of a pupil who has just got a positive test result. The reasonable question for them (and their parents) to ask is “what are the chances that this is a false positive?” Given that a positive test result means the pupil, their family and their school bubble will have to isolate for 10 days, a high false positive probability is a real problem.
The answer to that pupil’s question depends on the prevalence of the disease and the accuracy of the test — let’s consider three scenarios.
PROBABILITY OF FALSE POSITIVES IN THREE SCENARIOS
Where 1 in 100 pupils have the infection (Scenario A), by testing a million we would find 5,000 cases but get 990 false positives. This ratio of true to false positives is quite favourable — 5 out of every 6 with positive results actually would have Covid-19 infection — so the probability that the pupil genuinely has the infection is over 80%.
However, the picture becomes less favourable as the infection becomes rarer: if only 1 in 1,000 pupils were infected (Scenario B) we would detect 500 cases but get 999 false positives. The ratio of true to false positives is now unfavourable – one true result for every two false results.
If only 1 in 10,000 had the infection there would be one true result for every 20 false results (Scenario C). Why would anybody consent to a test where the chances that a positive result is wrong are so much higher than it is right? This isn’t the fault of the test — it’s the application in a low prevalence setting. Using any test — even one with an incredibly high specificity — will lead to more false than true positive results when the disease becomes rare.
Test-and-trace have been publishing weekly figures for lateral flow tests done in schools. The prevalence of positive results has dropped — at the beginning of the term it was 0.3%, but last week’s data (up to the 24th Feb) was down to 0.07% (1 in 1500) based on observing only 189 positive results from 288,958 tests. This incredibly low positivity rate is a concern. Clearly it indicates that cases must be rare, but it also raises concerns that the test may be missing more cases in children than it has in adults. It seems very likely we are in the zone where false positives may considerably outnumber true positives.
The solution is simple: children and teachers who get positive lateral flow test results should all get confirmatory PCR tests that determine whether or not they need to continue to isolate — including if the test was done at school.
If the specificity of the PCR test is the same as an LFT test (a conservative assumption) only one in a million would now get a false positive result —whether prevalence is 1 in 100 or 1 in 10,000. So it all but removes the false positive problem.
But would this measure risk more genuinely infectious pupils falling through the net? Recent data suggests that PCR may miss 10% of the more infectious cases. Based on this figure, and assuming a prevalence of infection of 1 in 1000, confirming with PCR would avoid 20 families and bubbles being wrongly isolated for 10 days for every additional Covid infection missed. (If confirmation was done using 2 PCR tests rather than one, the missed cases would reduce to 1 for every 200 false positives avoided).
This is a policy judgement, but without proper confirmatory testing the Government is asking us to consent to children being tested in a way where (a) negative tests have probably only about halve the chances of having Covid-19 so do not effectively make schools safe and (b) positive test results give us somewhere between a 70% and 95% chance of families and bubbles being unnecessarily put into isolation for 10 days. It is not an attractive proposal.
Jon Deeks is Professor of Biostatistics in the Institute of Applied Health Research at the University of Birmingham.
Jon leads the international Cochrane COVID-19 Diagnostic Test Accuracy Reviews team summarising the evidence of the accuracy of tests for Covid-19; he is a member of the Royal Statistical Society (RSS) Covid-19 taskforce steering group; co-chair of the RSS Diagnostic Test Advisory Group; a consultant adviser to the WHO Essential Diagnostic List; and Chief Statistical Advisor to the BMJ.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeAs the infection rate drops to zero which is probably the case in some places, then all ‘positives’ will be false. This illustrates the madness of this scheme.
It is alarming that most politicians seem unaware of this Bayesian method of calculation but even more concerning that their advisors have either also not understood it or not explained it sufficiently throughout the pandemic. Mass testing was always going to produce inaccurate results.
It is refreshing to see a clear explanation of the problem.
The raw results will be inaccurate but the trends revealed by those results over time will be valid.
Long way to go before infection rate zero – still ten times the +ve test results we had last summer and clearly real infection rate wasn’t zero even then. By the time we get to that point we’ll easily be in the next phase of the school testing in which confirmatory PCR tests are to be used (if I’ve read my letters-to-parents right), which is what the article says is “the solution”. So by the time low prevalence might become a problem, it’ll already be solved.
Have read this three times and I think I understand it – statistics was one of my courses at university. Do the government ministers also understand it? This is why a Classics background is not good for a Prime Minister in today’s complex world.
He might try to work through his advisors but which advisors?
I don’t suppose there have been been more than two or three government ministers in all of British history who would have had the intelligence (and time) to understand those statistics. From recent decades, I guess Lawson and Thatcher might had have had a chance.
It isn’t necessarily about intelligence – it is about training. So Boris, along with ever other head of state has to rely on those that are trained – the ‘Expert’
The problem any PM/President has at the moment is – WHICH expert you should be listening to. Not exactly an easy task when you think that there are approximately 450 members of SAGE. Do you think that they all agree? Because it has been obvious that they don’t.
Maybe the last bit is harder to follow – but the basic premise seems pretty simple to me . In fact this article has ‘solved’ to me a New Scientist article on stats I read and couldn’t fully understand years ago – funnily enough it was on testing positive for a relatively rare disease, and the false positives.
I think one issue is the use of terms/concepts: False positive rate – so this is say 0.1% of tests, but it could be understood as 90% of positive results (example), it’s quite confusing if misued. I wonder if people are using differing meanings when they discuss PCR false positives? I dismiss them as cranks if they suggest that there’s a 5% false positive rate, but it’s plausible that 5% of positives are false. If the testing remains the same then the overall false positive rate is fixed, but it’s relation % vs real positive can litterally alter by orders of magnitude as infection rates alter.
If I was being generous I’d say Boris and Co haven’t time to read everything and believe what advisors tell them. Or they only care about certain statistics: opinion polls and voting.
Reality matters far less than public perception in most cases. Remember last year when by some polls a significant proportion ~20% of the population believed that 10+% of the population had died of Covid (there were flaws to the polls mean figure, but not the people choosing 10%+). A lot of people are still hysterical about Covid, including some parents; so ‘punishing’ 100,000s of people with a bad testing regime might even be seen as worth it.
Valuble expertise certainly. But the solution is even simpler. Stop testing children. Stop blaming children. Stop jeopardising their education and well-being in the vain, hubrisic pursuit of eliminating an endemic virus.
The school testing regime, a consequence of the wholly immoral teaching unions, is symbolic of a society that has become totally unmoored from all reason or integrity.
We all know that the PCR tests deliver a lot of false positives. Thus children and families will be locked away, and schools shut down, on the basis of fake results. But a lot of people are getting very rich from the whole farrago, and that’s the main thing.
Also the concept of asymptomatic transmission of the virus a major justification for lockdown is potentially based on inaccurate testing data.
From my reading the justification for this is not as solid as has been portrayed. ‘Act as if you have It’ may hopefully be consigned to the well intended but wrong bin.
Bring back ‘Keep calm and carry on’.
Asymptomatic transmission is part of the fiction underpinning the Great Lie and is the basis for the liberties of the asymptomatic majority being withheld.
A very clear summary, thank-you.
The Parliamentary Accounts Committee report on test and trace that was published today – but which has now been removed from the govt web-site – explains that govt has purchased or is the process of purchasing 800 million lateral flow tests. These tests may be valuable in some situations but mass testing in schools is not one of them. The manufacturer Innova makes it clear in their instructions that the tests should be used if a medical specialist suspects covid ideally within the first 5 days of infection. Mass testing asymptomatic people with these tests does not fit these criteria and makes little sense when virus prevalence is low.
Hancock is certainly aware of Bayesian maths. See Hansard 17 Sep 2020 – answer to Q from Desmond Swayne. He understands and yet he persists with testing asymptomatics in a low prevelance environment. He is looking for positives to promote excessive caution and fear as part of the Great Lie.
“If the specificity of the PCR test is the same as an LFT test (a conservative assumption) only one in a million would now get a false positive result”
Can I please ask: Could this not necessarily the case, as it depends if the reason for less than 100% specificity is the same for both tests?
If they’re truly independent then, like rolling two true dice, the probabilities multiply. If, for example, the reason for the false positive in both tests is linked to the same cause e.g. residual virus particles, then the probability of a false positive becomes less than the multiple of the probability for each test.
Have I got that right?
In my opinion you are correct, in the general case. However, when talking about lateral flow and PCR it is important to understand that they test for two very different things (RNA vs. antigens) so it is difficult to envisage a common cause failure – for a false positive. If considering false negatives then you’d definitely be right because there are significant commonalities including swabbing process (and yes false negatives happen, even for PCR, I’ve had a fully symptomatic child test repeated -ve on PCR and yet come back positive a few weeks later on antibody blood test).
Deeks does not provide a balanced view. Michael Mina (an expert in this field as well) has called out Deeks to have debate to settle the LFT argument. I appreciate we all listen to the ‘scientists’ however, Deeks cannot claim transparency. He refuses to acknowledge Mina’s invitation and has actually blocked Mina on Twitter. Talk about professionalism. I would recommend having a look on Mina’s Twitter profile for a balanced view, alternatively read the Lancet article: https://twitter.com/michaelmina_lab – https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)00425-6/fulltext
This illustrates the difficulty of random testing in asymptomatic groups. However, although the ratio of true to false positives certainly looks poor, it remains the case that in the low prevalence situation, only 1,000 out of 1,000,000 children will be wrongly told to isolate. This is a very low number and, as these 1,000 false positives will occur randomly, will not disrupt the operation of any particular school.
I agree that a positive lateral flow test should be followed up with a PCR test. This is very easy for parents to organise using the current, highly effective, booking system.
Finally, I must pick up Jon Deeks for his use of the phrase “making schools safe”. This is lapsing into the language of the teaching unions – schools are very safe places for children and staff. Their possible roles as transmission hubs doesn’t make the schools themselves unsafe.
Does anyone have links to the studies mentioned in this informative and well written Artikel ?