[Note: insofar as I understand what follows, it is thanks to the marvellous Kevin McConway, professor emeritus of statistics at the Open University. All errors are entirely my own.]
Regression to the mean goes like this, using IQ as an example. Imagine that everyone’s real IQ and their self-assessed IQ were precisely the same: someone who estimates their IQ is 99 would have a real IQ of 99; someone who estimates it at 100 would have a real IQ of 100. If you draw them on a graph, with the objective IQ on the X (across) axis and the self-assessed on the Y (up) axis, the line would be a perfect 45° angle: if it’s 99 on your X axis, it’ll be 99 on your Y axis; same for 115 or 87 or anything. They would be perfectly correlated.
We know, though, that the two aren’t perfectly correlated. Your line will not be exactly at 45°; in fact it will be shallower, because when your line goes one to the right, it will, on average, go less than one up. The two extremes will be closer — will regress — towards the mean. What that means is that people will guess themselves closer to the average than they actually are. People who have real IQs above 100 will, on average, underestimate their scores; people who have real IQs below 100 will, on average, overestimate them.
Now, we bring in the “better than average” effect. It is exactly what it sounds like: most people, when asked to assess their ability at some task, will estimate themselves to be better than the average person.
To use the same example: on average, people estimate their IQ at 115. However, since average IQ is literally defined as being 100 — if people start doing better on the test, the scores are recalibrated so that the average remains 100, which is exactly what happens in the Flynn effect — that can’t be right. On average, we overestimate our own intelligence.
But look what that means for the Dunning-Kruger effect. That line we drew on our imaginary graph when we were talking about regression to the mean — it’s still the same shallow slope, but now it’s higher up the graph. The average person’s IQ is still 100, but the average person estimates their IQ at 115.
As you move to the right — as actual IQ scores go up — so do estimated IQ scores, but more slowly, because regression to the mean keeps the slope shallow. So as you move to the right on the graph, people’s assessments of their own IQ get steadily less wrong. As you move left, the real scores drop off rapidly, but because of regression to the mean, the self-assessed scores drop more slowly. So as you move to the left on the graph, people’s assessments get wronger and wronger.
The paper shows that it can get Dunning-Kruger-compatible results from simulated data with nothing more than these two effects. A similar result was found way back in 2002. This doesn’t rule out the existence of the effect as postulated by the original paper, and perhaps it’s real in some contexts — “it is possible that the Dunning-Kruger effect may be identified for some cognitive abilities not measured in this investigation”, as the authors of the Intelligence study put it. But most of the time we don’t need it to explain anything.
To some degree, you might think this is all just an interesting but not greatly relevant piece of academic trivia. I think, though, it matters, for two reasons.
One is that Dunning-Kruger itself gets so much attention, at least by the standards of psychological discoveries. It won an Ig Nobel — the comedy mirror of the Nobel prizes — in 2000, and had an opera written about it. An academic pointed out to me that there’s a paternalistic aspect to it: it implies that “my skills license me to think for you”, as he put it. We all point at the stupid people who believe the wrong things, and we say: these people are idiots, it’s OK to ignore them, they’ve just got a nasty case of the Dunning-Krugers.
It’s important that we try to believe in things that are real; so if it’s not real, we shouldn’t believe in it. The DK effect is a high-profile thing which lots of people believe, and I think we ought to be careful to make sure that it’s real. The fact that it’s such an easy way of accusing your political enemies of being stupid and wrong is an extra reason to be wary of it — it feels good to say that someone is a Dunning-Kruger case, rather than engage with the real reasons they might believe the things or behave in the ways that they do.
It’s also yet another reminder that there are deep-seated problems in the social sciences — bad statistical and experimental practice — and, subsequently, a lot of the things that many of us believe are, in fact, false. It’s pervasive, almost endemic. A few days before I became aware of this paper, a thread was going around social science Twitter, looking at the problems the social sciences face. The key takeaway was probably that barely half of all papers — 54% — can be expected to “replicate”: that is, if someone were to run the same experiments again, only about half of them would find the same results.
These are findings which influence real-world policies. Millions of dollars were spent by the Obama administration on policies based on worthless research by the food scientist Brian Wansink, for instance. Growth mindset is on somewhat pretty dubious ground, replication-wise, but influences education around the world. Companies and governments spend millions on implicit bias training to reduce racism, with little to no evidence that it works or that the phenomenon is even relevant to actual, real-world racism.
Dunning-Kruger is part of this same milieu of half-understood and poorly evidenced claims about the human psyche. And people do base business decisions on it (“The best employees will invariably be the hardest on themselves in self-evaluations, while the lowest performers can be counted on to think they are doing excellent work”) and offer advice for how to minimise it in themselves and others (“Always be learning, and promote learning … seek mentors and experts”). These decisions and pieces of advice are based on a misunderstanding of a thing that may or may not actually be real.
As I said: the Dunning-Kruger effect is immensely useful as a rhetorical device; it points to something that we instinctively feel exists, some a-little-knowledge-is-a-dangerous-thing phenomenon in our heads. And it’s such a tempting insult to use. But it seems very likely that it is not as relevant as we would like to think. Rather beautifully, it seems that the people who know the least about Dunning-Kruger are the most likely to overestimate its value.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeI appear to be the first to comment, and i am not surprised. i don’t really know what the article is saying. The DK effect is explained, but the counterevidence just has me confused. In fact, it doesn’t appear to counter anything.
Do less knowledgeable people understand themselves and their capabilities far less than knowledgeable people? A little knowledge is a dangerous thing? I have no idea. But I have observed that those with some knowledge in one subject and who might consider themselves experts often think that that confers on them a wider omniscience and they can make pronouncements on other subjects with the same degree of authority. Academics come to mind.
I feel compelled to ask if the replicability crisis is unique to the social sciences. My suspicion is that it is not, and that the more traditional “hard” sciences are equally prone to replicability problems. I may be wrong, but if so I want data!
This view is despite having written a book which debunks a lot of psychology in my own field (https://www.amazon.co.uk/Ba…. One of the problems is that people (including scientists) are great at stringing facts together to make a coherent picture, even when they don’t merit it. It’s like one of those “join the dots” puzzles but without all the dots present.
Thank you Mr. Chivers for another insightful article. While avoiding Twitter entirely, I still encounter too often in media and in personal conversations this assumption that disagreement must be a result of stupidity of the other person. Walter Lippmann in his famous article on free speech, “The Indispensable Opposition” writes “the essence of freedom of opinion is not in mere tolerance as such, but in the debate which toleration provides; it is not in the venting of opinion but in the confrontation of opinion.” Thank you for your contribution to encouraging such debate to occur rather than a dismissal of any opposing ideas. One small note: was the labelling of the X Graph in the first table showing the Dunning-Kruger effect “No Nothing” intended to be funny or just poor grammar, a la Dunning-Kruger?
The last sentence is a nice way to round off the argument, as with a good chess problem the finish is both effective, and elegant.