by Tom Chivers
Thursday, 10
June 2021
Factcheck
08:02

Don’t take university rankings too seriously

They're a classic case of Goodhart's law — the metrics are easily gamed
by Tom Chivers
What makes a “best” university? Teaching? Student experience? Research output? Pretty quadrangles?

The QS World University Rankings were released this week; there was much excitement about the UK having four universities in the top 10. And no doubt all my fellow alumni of King’s College London were thrilled to find that their alma mater had been ranked the 35th best seat of higher education in the world. 

They may, however, have been somewhat surprised to realise that (according to the Guardian’s university rankings) they were also the 42nd best in the UK.

Sharp-eyed observers will have noticed that 42 is a bigger number than 35, and that it is logically impossible for there to be 41 better universities in the UK but only 34 better in the world, because the world contains the UK. So: what’s going on?

Rankings like this sound awfully important and objective. But (as my cousin David and I explain in our recent book How to Read Numbers) they conceal an awful lot of information. For instance: every so often, India overtakes the UK to become the fifth largest economy in the world, and people get very upset about it. According to the World Bank’s economic rankings, India is currently ahead of us. Meanwhile, China’s is the second biggest, while Japan is third. 

That ranking makes it sound like the difference in size between India and the UK is the same as that between China and Japan; but India’s economy is 2% larger than the UK’s, while China’s is 280% larger than Japan’s — almost four times the size. A footling change in either the UK or India’s economy, or indeed a measurement error, could make them switch places, and regularly does; but if Japan were to overtake China, that would be a huge, and shocking, story.

But at least the size of an economy is an objective fact that could, in theory, be measured. What makes a “best” university? Teaching? Student experience? Research output? Pretty quadrangles?

For instance: the QS World University Rankings give heavy weight (40% of a university’s total score) to “academic reputation”: a survey goes out to 100,000 academics worldwide and they are asked to give their opinion of the “teaching and research quality” at the world’s universities. Since the large majority of those academics will never have visited, for instance, the University of Durham, their input will be based heavily on guesswork. 

And there’s nothing set in stone about that 40%: they could have given more weight to student satisfaction, or student/faculty ratio, or research impact. And then the list would look entirely different. The Guardian’s list, indeed, is entirely different, because it uses different metrics: “spend per student”, how many students are in a career 15 months after leaving, and so on. Which is right? Neither; both. It’s an arbitrary choice.

You may reasonably say “who cares”: every year Oxford, Cambridge, and a few American big hitters (MIT, Harvard, Stanford, etc) make up the top dozen or so, along with Imperial and/or UCL. Does it really matter if the University of Liverpool is 158th or 189th?

But Goodhart’s Law — ”when a measure becomes a target, it ceases to be a good measure” — is a thing. In the UK, when schools are given targets of how many A*-C GCSE results they need, they start focusing on the children who are on the cusp between a D and a C, and neglecting the rest.

Similarly, if you rank universities, and then would-be students choose their universities on the basis of those rankings, then universities will try to game those rankings. In fact the formulation of Goodhart’s Law I gave above comes from a 1997 paper on how UK universities game the auditing system. It is a racing certainty that universities will focus on getting their rankings as high as possible, whether or not that actually translates to being better universities.

It’s not that these rankings are entirely meaningless. It is, for instance, probably true that UC Berkeley (32th on the QS list) is a “better” university than, in some fuzzy but real sense, the University of Nottingham (103rd). But nor are they very informative. And if KCL can be the 35th best in the world on one list and the 42nd best in the UK on another, it’s probably wise not to take any of them all that seriously.

Join the discussion


To join the discussion, get the free daily email and read more articles like this, sign up.

It's simple, quick and free.

Sign me up
Subscribe
Notify of
guest
13 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
Andrew Raiment
Andrew Raiment
1 year ago

Why bother taking universities too seriously…

Simon Coulthard
Simon Coulthard
1 year ago
Reply to  Andrew Raiment

Because they help form a large proportion of the next generation and have a great influence on future morals and policy?

Andrew Raiment
Andrew Raiment
1 year ago

You mean all those useless social science graduates that have been pushed out over the last 20 years, versed in idiotic ideologies such as Post-modernism and Critical Theory, and not good governance. Too stupid to recognise fallacy and paradox.

Simon Coulthard
Simon Coulthard
1 year ago
Reply to  Andrew Raiment

Well yes them. But also all our doctors, lawyers and scientists. My point being that universities are not insignificant to our society.

Andrew Raiment
Andrew Raiment
1 year ago

Indeed. My original comment was flippant but it appears there are serious problems within our universities that do need addressing, especially since those ideologies are now “infecting” the natural sciences.

Last edited 1 year ago by Andrew Raiment
Simon Coulthard
Simon Coulthard
1 year ago
Reply to  Andrew Raiment

Oh my so many problems you’re right. The stifling of real debate and cancellation of tenured professions who go against the currently accepted theory. It’s mad

Andrew Raiment
Andrew Raiment
1 year ago

You go to universities to learn, not learn garbage. Why are universities now teaching garbage?

Last edited 1 year ago by Andrew Raiment
Alka Hughes-Hallett
Alka Hughes-Hallett
1 year ago

Previously respectful professions i.e. Doctors, Lawyers, Scientists are all now corruptible and untrustworthy. When scientists can stifle debate, science from their papers and scientific journals becomes meaningless. When doctors can worry more about reputation & play politics or dish out cures to save being taken to court then it’s hard to believe them. I think I’ll go with lawyers . The most honest bunch.

Andrew McDonald
Andrew McDonald
1 year ago

Can’t remember who it was who said a few decades ago : ‘In California the research labs are using lawyers now instead of rats – there are more of them, and there’s less emotional attachment…’ Hope this has come true by now.

Alastair Herd
Alastair Herd
1 year ago

The University league tables are a joke. Institutions shoot up and down them by just gaming the requirements while not really doing anything to improve the quality of the education to the purchaser.

While I was studying at my Russel Group University I spoke to my Tutor about struggling with the STEM course after a significant period of illness (just so you don’t call me snowflake). His advice was to sign up for the Masters! Despite being the head of all Undergradute wellbeing he was more interested in things that look good on paper than genuine student outcomes.

As someone who voted Brexit I must give it to the Europeans that their universities seem a damn sight better at actual education than ours do…

Jon Redman
Jon Redman
1 year ago
Reply to  Alastair Herd

I strongly suspect that the dominance of universities in English-speaking countries in these tables is a consequence of how they are compiled. If it’s based on subjective views of their research, then German universities’ research that is published in German will not score many brownie points. There are probably fifteen times more people living in countries where English is used than in countries where German is used, which seems certain to affect how German language research is perceived (i.e. it mostly is not perceived at all).

Jon Redman
Jon Redman
1 year ago

Clearly a fair bit of what goes into these various rankings is subjective opinion, or is unrelated – maybe even inverse – to the actual intellectual rigour involved. Ratings that reflect how much the students are enjoying it could easily be a proxy for how cushy the place is.
More informative would be to know the average A-Level grades held by the entrants; the percentage of its graduates that is unemployed after a year; and the average graduate salary ten years on. Ideally you would have this sorted by degree subject and / or class.
Taken together, this tells you roughly how smart the cohort is to begin with but also what difference the university makes for them. Sure, you can argue that A-Levels don’t measure smarts, but that’s fine – if people with the best A-Levels aren’t very smart after all, the other two measurements will expose this.
One interesting thing that emerges from the above is that there are 5 times as many candidates with Oxbridge standard A-Levels as there are places there. There are 47,000 students at Oxbridge, so inferentially there are 190,000 elsewhere with much the same grades. As the average UK tertiary has 15,000 students enrolled, there are about 12 more whose students have Oxbridge grades. The first rank of UK universities therefore contains not two universities, but about 14.
The other thing that surprises is that notoriously “useless” courses like Media Studies have a very high employment rate. The money is not great, but neither is the money if you read English Literature at Oxford. Stats like this tell you whether the job and pay situation is poor because the course is worthless, or because its graduates are preening entitled delusional pseudo-intellectual left-wing gobs4ites who consider real work beneath them and are waiting for the right charity or quango post to come along.

Jane Morris-Jones
Jane Morris-Jones
1 year ago

When I was a student in the 70s it was reckoned that academics split about 60/40 left to right in political orientation; that is now estimated to be 90/10, to the grave damage of free speech and exploratory debate. I think this metric ought to be included in the measure of excellence to highlight where students might get the best value in terms of mind-stretching…