Is this the second wave? Sceptics who talk up 'false positives' are engaging in wishful thinking

In ancient Greek mythology, Cassandra was cursed by the gods: she was given the gift of prophecy, but although her prophecies would always be true, no one would believe them. She foresaw that Helen’s abduction would lead to the destruction of Troy; she warned of the wooden horse, filled with Greeks, which was the instrument of that destruction. But her warnings went unheeded.

I don’t want to say I’m exactly like Cassandra. But I do want to say that everyone is ranting and raving about false positive rates at the moment, almost as if they didn’t listen when I was ranting and raving about them back in early April. It all seems to stem back to an article on Toby Young’s “Lockdown Sceptics” site a few days ago, in which the author correctly pointed out that if a test has a 1% false positive rate, that doesn’t mean that only 1% of positives are false.

From there it’s sort of spiralled out of control, with, among other people, Julia Hartley-Brewer saying that “the Government is planning to lockdown our country again when there is no evidence of a second wave” and John Redwood demanding that “government advisers today need to tell us how they are going to stop false test results distorting the figures”.

Here’s what’s going on. First, it’s absolutely true that the “false positive rate” does not on its own tell you the percentage of positive results which are false. Imagine this: you’ve got 10,000 people; you have a test that gives you the correct answer 99% of the time (i.e. 99% of people who do have the disease get a positive result, and 99% of people who don’t have the disease get a negative result); and you have a disease that only 0.1% of the population has (which is roughly what the ONS reckons is the real prevalence of Covid in the population).

So out of the 10,000 people, there are 10 people with the disease. Your test will probably tell you, correctly, that they all have the disease. But it will also give false positives to 1% of the 9,990 people who don’t have the disease. So it will probably tell about 100 entirely healthy people that they have Covid-19. So in this situation, given that a disease that only 0.1% of people have, a positive result on a test with a 1% false positive rate means you only have about a 9% chance of actually having the disease.

This is all very straightforward and exactly what Bayes’ theorem tells us. If you get a positive test, your chance of actually having Covid is dependent not just on how accurate (or technically how “specific”, the technical term for how good it is at avoiding false positives) it is, but also how common the disease is in the population.

What Hartley-Brewer, John Redwood and others are suggesting is that this means that, actually, there’s no real reason to believe that there is a second wave, because the positive test results hugely overstate the real number of cases. I think they’re wrong about that, and quite crucially wrong. Let me try to explain why.

(Once again, my thanks to the indefatigable Kevin McConway of the Open University for helping me understand this stuff, and once again, any errors are mine, not his.)

Let’s start with the least important thing first. The sceptics’ argument assumes that the tests are applied to the population entirely at random. Only 0.1% of the population has the disease, so only 0.1% of the people you’re testing have the disease. But that’s obviously false, because the people who are coming forward to get a test usually have some reason to think they might have, or might have been exposed to, the disease. So the figure will probably be higher than the 0.1%-ish prevalence in the population. Even if the true number is only 1%, that makes a huge difference: suddenly, plugging that into our equation above, you’d be looking at 50% of positives being false, not 91%*. The risk of false positives is hugely sensitive to quite small changes in the true incidence rate.

We don’t know the true incidence rate of people coming forward to get tested, but we can be confident it’s higher than the background rate in the population. That said, it doesn’t change the underlying point, that some number — perhaps some large number — of the positive results are false.

But here’s the second point, which is that it doesn’t really matter. The number of positives has increased, and assuming that your testing process hasn’t changed — that the tests haven’t literally become worse in some way — then your false positive rate shouldn’t have changed. If your number of positives has gone up, then that must be because the true positives have gone up.

Let’s go back to our 10,000 imaginary people and our 99% specific test. If absolutely nobody has the disease at all, then you’ll get about 100 false positives. If 1% of people have it, then you’ll get about 99. If 10% of people have it, you’ll get about 90. Whatever is going on with the disease, whatever the real-world prevalence, the number of false positives won’t ever make your data show an upward curve.

But if the number of real-world cases goes up, your true positives will go up very quickly. If absolutely no one has it, obviously you’ll get zero true positives. If 0.1% of people have it, as we saw, you’ll get about 10 true positives. If 1% do, you’ll get about 99 (assuming a false negative rate of 1%, which isn’t right, but for simplicity). If 10% do, you’ll get about 990. If you ever see your number of positives going up, it must be because the number of real cases is going up, unless your testing regime has changed. And we are seeing the number of cases go up, so, almost certainly, the number of actual cases is going up too.

Admittedly, if the government started doing many times more tests than it had been in previous weeks, you’d see the absolute number of false positives go up — but it seems unlikely that the problem with our testing regime, at the moment, is that it’s expanded too fast. It’s also worth noting that even though we don’t know the true false positive rate — there’s no perfect standard to judge it against — we know that fewer than 1% of all the tests the ONS performs come back positive, so the true positive rate cannot be higher than 1%.

And the important thing here is that we’re dealing with an epidemic, and exponential numbers. How fast the number of cases is doubling is much more important than the actual number itself. As Chris Whitty and Patrick Vallance said in their press conference on Monday 21 September, that number appears to be doubling every seven days. If the real number of cases is only half what the tests say, that only makes a week’s difference to how quickly we have to respond to it. We should really remember that from last time around.

That’s not the only thing, of course. These tests are one strand of evidence, but there are others. Most notably, the ONS survey on which Hartley-Brewer and the others base their 0.1% figure (and therefore their “91% false positives” claim) itself shows a significant rise in prevalence. The dataset is here; the estimated percentage of people who had the disease was 0.64% at the end of April, when the survey began (it was probably even higher in March), and dropped as low as 0.03% in late June and early July. But since the end of August, it’s risen back up to about 0.1%. So we’re nowhere near the early peaks, but the number has trebled in the last two months, with especially rapid growth since mid-August.

If you want more evidence you can look at the KCL-ZOE Symptom Study, which tracks the number of people reporting Covid-like symptoms, and which also says that there’s been a rapid growth since the end of August, taking us back to about where we were in early June.

Then there are hospitalisations, which, again, are absolutely nowhere near the horrors of early April, when about 3,500 people across the UK were admitted on a single day; but nor is it particularly near the low point of 22 August, when fewer than 70 people were admitted across the whole country. Now it’s around 250.

Hospitalisations lag behind infections, and deaths lag behind hospitalisations: deaths in the UK dropped to fewer than 10 a day in early September and reached 22 on 16 September, so we may be about to see a real increase there. (The data there gets a bit complicated, because deaths take some time to be reported — sometimes months — so those likely aren’t the final figures. But I think it’s reasonable to say, looking at the graph, that the beginnings of a rise are visible. Again, it’s nothing like the April-May peak, but I think it’s real.)

Is this “evidence of a second wave”, as Hartley-Brewer argues it isn’t? Well, it all depends on what “second wave” means. There’s absolutely no evidence that people are dying, or getting hospitalised, or even getting infected, at the rates they were six months ago. But there is evidence that they’re getting infected at the rates there were maybe seven months ago. Eight at a push? I don’t think we’re going to go waltzing merrily down the exact same path as we did last time, but we should have learnt by now that exponential curves can go up pretty quickly if you don’t take action.

None of this is to say that the false positive rate complaint is garbage — Hartley-Brewer, Redwood et al are right to say that the number of positive test results does not, on its own, tell us about the number of actual cases.

And there’s a bigger point, which McConway pointed out to me, which is that you see charts like this one going around, showing the number of test-confirmed cases, and which make it look as though the numbers we’re seeing now are in any way comparable to the numbers back in March and April. But back then, as McConway says, “the only way you could get a confirmed test in April was if you were in hospital, pretty much.”

The true peak was many, many times higher than the number of test-confirmed cases; we are now doing hundreds of times as many tests. It’s not enough, but it’s loads more than in those hectic early months. So the number of positive results — false or true — will be many, many times greater, per actual case in the community, than it was back then. Those sorts of comparisons are silly and spurious, and it’s right to say that we’re nowhere near those numbers.

But I’m very wary about being complacent. I was sceptical, a few months back, about claims that locking down a week earlier would have saved half of the lives lost; maybe I was right to be, but even if that was an overstatement, I think most scientists agree that faster action could have saved many thousands of lives. If the curve is going up, that’s what matters, not whether the exact number is 6,000 cases or 3,000.

The Government seems to be taking some sort of action this time, and hopefully it won’t repeat all the mistakes it made back in March and April. There were plenty of Cassandras back then, rightly warning of the doom to come, and being ignored; hopefully this time they’ll be listened to.

* There’d be 9,900 people without the disease in your 10,000 tests; your test would give you false positives on 99 of them. But there would be 100 people who actually had the disease, and your test would identify 99 of them correctly. So that’s 50% true and 50% false positives.

---

Tom Chivers is a science writer. His second book, How to Read Numbers, is out now.

TomChivers