Everything is awesome. Photo: Mike Hewitt/Getty Images

Back in the middle of March, I was pretty stressed out. Funnily enough it was the handwashing that got me; this was a few days before lockdown, and the real public health message had been wash your hands. So I washed them, dozens of times a day. My hands were raw and cracked, sometimes to the point of bleeding. They itched ā burned ā constantly. I was at every moment aware of this burning sensation, and it was an ever-present, unignorable reminder of how the world had changed.Ā
My six-year-old sonās hands were red and raw too, from his conscientious efforts at washing his hands at school. He would proudly tell me that his teacher had said he was the best hand-washer in his class, and it broke my heart even as I praised him for it. Iām not an anxious or depressive person, and Iām not usually stressed. But the coronavirus pandemic ā the uncertainty, the fear, and the god-damn handwashing ā got to me.
Itās easier now: you get into routines, the weird becomes normal; plus I started to backslide a bit on the handwashing, once they shifted the emphasis to social distancing. But, for a while, I could feel myself struggling.Ā
So when people say that Covid-19 is bringing with it a secondary epidemic of mental health problems, I can easily believe it. I can believe it because I could see how tough being out of school was on my kids, and because I heard other people saying the same; and because we were all well-off people of privilege, so I could only imagine that it was worse for others. I can believe it because I know so many people who went from being full-time workers to full-time workers and full-time parents and educators. Itās a bloody stressful time.
Mental health matters. If people are suffering significant mental health impacts because of coronavirus, or because of the lockdown, then we need to know, because that will affect the correct way to respond, personally and as a society. We need to honestly address the trade-offs, the costs and benefits, of the policies our country takes to minimise the damage of the virus.
That means that research into the mental health impact is important ā and thereās lots of research going on. The trouble is that, just as in so much of science at the moment, everything has to happen fast. Thatās understandable, but it can cause real problems.Ā
Pete Etchells, a professor of psychology at Bath Spa University, told me that āthereās a lot of scrabbling around to do research on the wider effects of Covid. Not just in psych. And it comes from a good place; people want to help and do something useful.ā But, he says, we need to be extraordinarily careful. āWhen thereās stuff going out there that either has the aim of, or has an impact on, policy decisions, it has a real potential to have a big impact on peopleās lives.āĀ
Careless science can lead to wrong conclusions amazingly easily. Psychology has learnt that lesson more harshly than most disciplines, having suffered huge and embarrassing setbacks during the āreplication crisisā. What it has discovered is that some precautions are vital. Some of these are slightly technical-sounding but important, such as preregistration of hypotheses, and committing to publishing regardless of the results, which avoid innocent and deliberate statistical malpractice. And, perhaps most importantly, itās vital to share your data and methods, so other researchers can check your work.
I want to flag two papers looking at Covid and mental health, each with very different approaches to how you do science, to show you what I mean. One of them came out in May, and was intended to establish the āvalidityā of a new measure that one of the authors had developed, a āfear of Covid-19ā scale, back in March. The authors have since used it in further research.
The āvalidityā of a scientific measure is how well it applies to the real world. For instance, most research into smartphones and mental health has been carried out using one of several questionnaires which asked people about their phone use. But a study last year discovered that how much people thought they used their phones was only tangentially related to how much they actually used it, as recorded by a special app. So the phone-use scales are not very valid: they arenāt very accurate at measuring the thing theyāre supposed to be measuring.Ā
Validity is crucial in psychology, because so much of it is based on questionnaires and surveys. If I make a questionnaire to give to patients, and which then gives them a score, and that score is called their āsuicidality scoreā, then it is really important that people who score highly on it are at higher risk of suicide than people with low scores.Ā
Itās important because, if that scale is picked up, other people will do research using it. So they might do a trial of some antidepressant, and measure subjectsā suicidality scores on this scale before and after. Then, if the subjectsā scores improve, the authors can say āthis drug improves scores on the Chivers Suicidality Scale by an average of 11 pointsā. And it might get picked up by NICE and given to patients by the NHS.Ā
So, if the suicidality score is actually not valid, if it doesnāt relate to real-world outcomes, then it is worse than useless: it is directing resources and future research away from the places where they can do good.
That means the paper establishing the validity of the new āfear of Covid-19ā scale is very important. The authors seemed to take it seriously, recruiting 8,500 Bangladeshi students to test it on. And, vitally, they said that ādata will be available on requestā if other researchers want to look at their findings.
But ā disturbingly ā they havenāt gone ahead with that. Another researcher, Brittany Davidson ā one of the authors of the smartphone self-report validation study I mentioned earlier āĀ asked them for the data several months ago. She had wanted to look at the data because, she says, she was wary of psychometric scales coming out with a ālack of rigourā, and she had questions over how the authors had managed to recruit such a huge sample size in a very short time, apparently without offering to pay the subjects anything; that is, she says, very unusual.
So far, none of the data has been forthcoming. This is really out of the ordinary, and worrying, especially since ā as far as we can tell ā the data is not sensitive. Researchers keeping data to themselves was a significant part of why the replication crisis got so bad. Itās especially bad when they say in the paper that theyāll share it.
You may feel this is all a bit technical and unimportant, but itās not. The original āfear of Covid-19 scaleā paper has more than 120 citations already. The validation paper was carried out in Bangladesh, but the authors have also created English, Turkish, Iranian, Spanish and other versions of the scale, in each of those languages. A small but non-negligible percentage of the research into the mental health impacts of Covid-19 is carried out using this scale.Ā
And its authors make serious claims about those impacts: they say in one paper that Covid-19 has led to people āsuffering from elevated anxiety, anger, confusion, and posttraumatic symptomsā, to āunusual sadness, fear, frustration, feelings of helplessness, loneliness, and nervousnessā, and that in āextreme cases, it may trigger suicidal thoughts and attempts and, in some cases, actually result in suicideā. This is important, non-trivial stuff.
More than that: the authors use the scale to propose policies for national governments, such as banning and removing websites that host āmisinformation (e.g., false COVID-19, treatment remedies, COVID-19 conspiracy theories)ā, or having government health agencies āmake online counseling sessions available to help ease the mental health concerns and worries of the general populationā. It is very possible that, somewhere in the world, a government white paper will be written citing the āfear of Covid-19ā scale as support for some policy or other. (Stranger things have happened.) And yet other researchers cannot look at it to see how it was validated.Ā
(There are, I should note, other concerns with some of the research of one of the authors, some raised here by psychologist Dorothy Bishop and others here by the psychologist Nick Brown.)
As we said earlier, it seems obvious that Covid-19 is dragging a mental health crisis in its wake. It certainly seems obvious to me. So you might think it doesnāt really matter if some papers cut corners a bit in order to tell us things that we already know.
Thatās why I wanted to bring up another piece of work, which seems to me an example of careful research in this fast-moving and scary topic. It looked at 800 adults living alone in the UK and USA, and measured their mental health on well-established, well-validated scales (the standard measures of depression, anxiety and loneliness).Ā
The paper measured their responses at three points, one some time before the crisis had really taken off, and two after lockdown had started. Vitally, also, it was a āregistered reportā: that is, the authors said in advance what they were looking for, and the journal agreed to publish it as long as it was carried out according to its methods, regardless of its results. That rules out lots of the bad statistical practice that can distort science.
Just as the existence of a Covid-19 mental health crisis seemed obvious to me, it seemed obvious to the authors of this study, Dr Netta Weinstein of the University of Reading and Dr Thuy-Vy Nguyen of the University of Durham. They just assumed that it was true; so much so that they were only measuring them to look at other things, such as whether introverted people were coping better.
But to both authorsā surprise, they simply didnāt find any increase at all on any of the three scales. The average level of loneliness, anxiety and depression did not go up. It was ā to me, and to the authors ā a startling result.
Itās not the last word. There could be all sorts of reasons why it didnāt find any impact. Weinstein wondered if the study, looking at people living alone, missed the stress placed on families. Or it could be that depression, anxiety and loneliness arenāt the main things that went up. āWe didnāt look at stress,ā she says: āI can remember that moment when I realised I was a full-time mum and a full-time academic, that sense of panic, but I donāt think I felt depressed.āĀ
But the point is: we would all have confidently predicted, like Nguyen and Weinstein, that lockdown would make people living alone more lonely, more anxious and more depressed. They didnāt find that. Psychological research is messy and hard; it demonstrates the importance of really good, careful science, especially in messy topics like psychology, and especially in fast-moving situations like this. āThis is a good opportunity to practise all the stuff that came out of the replication crisis to make sure that the answers we get are the right ones,ā says Etchells. Open data, preregistered hypotheses, well-validated scales: we know this stuff is important.Ā
My stress levels have returned to normal now; partly thatās routine, partly itās kids being back at school, partly itās my hands being back to their usual washing-up-liquid-advert softness. But the issues surrounding Covid-19 and mental health have not gone away; if nothing else, we can expect economic hardship and widespread unemployment in the coming years, and we know that is linked to mental health problems. Research into those issues will only get more important, to help mitigate those problems. But it needs to be good research. The lessons of the replication crisis need to be learned.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
Subscribe