The hypothesis was simple. Lead interferes with brain development, children with underdeveloped frontal lobes grow up to lack impulse control and the ability to succeed in modern society, and those children become violent, criminal adults.

I really liked this hypothesis. It fitted the data (as I understood it) and I liked that it suggested societal problems could have a realistic solution, rather than the “remake society so it’s better” solutions that most societal problems require.

Sadly, a new meta-analysis suggests that the link between lead and crime is overstated. It doesn’t say it disappears altogether, although it may have done; but it looks a lot smaller than it did. 

What is fascinating, though, is how they worked that out. The new meta-analysis looks at unpublished studies — or, rather, it tries to work out, in their absence, what those unpublished studies would have said.

Which sounds impossible, right? You can’t read the studies: they weren’t published. It’s like trying to read books that were never written.

But here’s (part of) how they did it. They used something called a “funnel plot”.

When you do a study into something — lead and crime, public opinion, particle physics, whatever — you’ll get an answer, a number. But that number won’t necessarily be the exact right answer, even if you performed the study well, because of noise: sometimes your sample will randomly be different from the total population, so your answer will be higher or lower than the truth.

But if you do lots of studies, and if they’re all well-performed, then they should cluster around the true answer: some will be higher, and some will be lower, but on average they should be about right.

And some studies are better than other studies. Big, well-performed studies should be closer to the truth than small, rubbish ones. If you plot them on a graph, you’ll see something like this — a rough triangle shape:

(These marvellous graphs are drawn by my sister, Sarah, for the book my cousin David and I wrote recently about statistics in the news.)

The dots are studies: some miss to the left, some miss to the right, but roughly speaking, they cluster around the real effect size, the vertical line. And better, bigger studies tend to be closer to it.

But in science, sometimes, if your study doesn’t find the thing you’re looking for, it doesn’t get published. This is a huge problem in science because it means that the literature gets skewed: it fills up with the studies that randomly happen to be on one side of the line.

A funnel plot, though, can let you see that. If you look at your funnel plot and you don’t see a neat triangle, but a lopsided thing with most of the dots on one side or the other, then it might be because lots of studies happened randomly to cluster on one side — or it might be that loads of studies were carried out but never published. It’s like a wall in a Roadrunner cartoon, all covered in bullets except where Wile E. Coyote was standing.

That, among other things, is what the researchers behind the meta-analysis did — and the dots did indeed cluster on one side:

And when you take that (and other things, such as simply asking researchers for any unpublished papers they have, but I’m trying to keep this simple) into account, the link between lead and crime becomes much less clear and possibly disappears.

Which is a huge shame, because it suggests that crime is complicated and multifaceted and bringing it down isn’t a simple job of banning a particular additive in petrol. But it’s a fascinating piece of scientific detective work, nonetheless.

---

Tom Chivers is a science writer. His second book, How to Read Numbers, is out now.

TomChivers

---