X Close

How Ofqual failed the algorithm test The embarrassing truth is that their mathematical model was a prejudice machine

"Closer to buying car insurance than taking an exam." Credit: Guy Smallman/Getty

"Closer to buying car insurance than taking an exam." Credit: Guy Smallman/Getty


August 18, 2020   7 mins

Of all the surprises 2020 has thrown at us, I certainly didn’t expect to see teenagers with placards taking to the streets to shout “Fuck The Algorithm!” For me, an extra twist of irony was that they appeared to be outside the building where I took some of my Statistics exams as an Open University student, sitting at a tiny desk for three hours with pencils and calculator, wishing I had done more work before it was too late.

Exams are a blunt instrument. They assess performance on the day, not ability. But when they were cancelled, they left a gaping hole in an education system that depends on the grades they spit out. If only we had an oracle that could see into the mind of each student and judge them: a statistical model, objective, fair, and well-fed on data from every student in the country. So that’s what Ofqual built.

Then, around 40% of the A level grades awarded by the algorithm fell below the teacher predictions for the student in that subject. Cue teenage demonstrations and widespread political recrimination. But, contrary to what students may have expected, those teacher predictions were never the starting point for the awarded grades. In many cases, they didn’t even form part of the calculation.

Instead, the system was designed to give an overall distribution of grades that looked similar to previous years, with similar numbers of A*, A, and all the other grades for each subject — though they did allow more A and A* grades than usual. Ofqual even went so far as to check that the proportion of grades handed out to different subpopulations (by gender, ethnicity and deprivation, for example) would look similar to recent years. If your definition of fairness is that boys, or claimants of Free School Meals, won’t do demonstrably worse than last year, you should be happy.

Ofqual’s Direct Centre Performance model is based on the record of each centre (school or college) in the subject being assessed. Whatever the range and distribution of grades achieved by previous students over the last three years, that is the range of grades allocated to the class who would have taken A-levels in 2020. There was some adjustment, if your class has shown better (or worse) performance than its predecessors in GCSEs or other previous assessments, or if other changes would leave the national distribution of grades looking too different from previous years.

Meanwhile, each teacher was asked to rank each class from highest to lowest in expected achievement. That ranking, not predicted grades, was used to slot each student into the predetermined range of grades from A* to U. The exception to this was small groups, less than 15 in most cases, where Ofqual did resort to teachers’ predicted grades. That is why less popular subjects and smaller schools have seen less marking-down from expected grades.

Any one individual’s achievements so far, or their potential in the view of teachers who know them, had less influence on their eventual results than the attainments of others who attended the same school in past years. It’s closer to buying car insurance than taking an exam for which you have worked for nearly two years. Just enter postcode, make and model and we will predict your likelihood of making a claim, and hence your premium.

This wasn’t inevitable. “Any statistical algorithm embeds a range of judgments and choices; it is not simply a technically obvious and neutral procedure,” wrote the Royal Statistical Society (RSS) in a scathing statement published on 6 August. “Calibrating this year’s estimated grades to previous years’ exam results is one such choice. How to take account of evidence of individual students’ prior attainment is another. How to take account of uncertainty is another.”

I asked Professor Guy Nason, Chair in Statistics at Imperial College, London, and fellow of the RSS, what Ofqual could have done differently. He was surprised that the UK had not attempted any kind of socially-distanced, in-person assessments, as some other European countries had done. But, given that some kind of statistical grade allocation was needed, Guy pointed out some specific pitfalls that could have been avoided, if a wider range of experts had been involved at an earlier stage.

“Overall, I think they ignored, or in some cases, underestimated uncertainty in many steps of their process. For example, the teacher-provided rankings for students within a subject were treated as if they were correct, when, in all likelihood, they are subject to considerable uncertainty. So, for example, tied rankings were not permitted, which might have resulted in students being assigned different grades even though their Centre thought that they were indistinguishable.” He also thought it was unfair to assess students by a completely different method if they happened to be part of a small class.

Nason had serious concerns about how Ofqual tested the predictive ability of their algorithm. “You’ve got to run a fair test, one that runs under the same conditions as the real deal.”

The normal way to test a predictive algorithm is to see how good it is at predicting the past. That is, you run the program for the previous year and see how well its predictions match what happened in real life. Ofqual did that for 2019, but because teachers in previous years were not asked to rank students, Ofqual could not use 2019’s teacher-generated rank orders for a test run.

Instead, it used the rank order that emerged from the 2019 exam results. Which is like showing you can predict the results of a horse race by including data about the order in which the horses crossed the finish line in that same race. “If a test uses aspects of the same data that it is trying to predict, then it results in a false sense of security,” says Nason.

Even by including some of the data they were trying to predict, Ofqual found their accuracy in predicting exact grades ranged from two thirds for History to one in four for Italian. For most non-language subjects, over nine in 10 students would be within one grade of the true result, but 3% of Maths students (for example) missing a fair result by two grades or more adds up to a lot of teenagers. Over 10% of Further Maths students, ironically the only ones who can understand the tortuous workings of the algorithm that betrayed them, would be over a grade away from a fair result.

“Their algorithm’s predictability is, especially for some subjects, not that good anyway, but if you then realise that they are over-optimistic and cannot be trusted, then one has to really question whether the algorithm is fit for purpose,” says Guy.

This isn’t just hindsight talking. The RSS offered to nominate two distinguished experts to the Ofqual technical advisory group in March. Guy was one of them. But Ofqual wanted to impose a Non-Disclosure Agreement that would bar them from public comment on the model for five years, in direct contradiction of the Society’s commitment to transparency and public trust.

Ofqual also ignored the House of Commons Education Select Committee’s call to publish details of their methods before releasing the results. They might have been spared some of the post-hoc dissection of their work, before public outcry and political pain caused Monday’s abandonment of the algorithm.

Because the grading algorithm has been withdrawn for political, not statistical reasons.

Its workings seem to have hit harder the very students who already felt the cards were stacked against them, and the communities to whom this government promised “levelling up”. Students in state schools and FE colleges, especially, and more deprived students, saw their awarded grades fall well short of their teacher-predicted grades.

It’s no surprise that small teaching groups, and less popular A-levels like Law, Ancient Greek and Music, are more common at private schools, which insulated those students from being marked down. Nor that previous results in selective and private schools would have been higher, bequeathing a higher range of grades to this year’s cohort.

But the picture is messier than that. Historically, teachers in large state schools, and of more deprived students, have been more likely to over-predict. There may be good reasons for this. Teachers may consciously give a student the benefit of the doubt, figuring that it at least gives them a shot at a good university place. If they fall short, they can haggle later. If they exceed more honest expectations, it might be too late for them to raise their sights.

And in a high-attaining school where students routinely get A and A* exam results, there is not much headroom for over-optimism, unlike schools whose students walk away with the full range from A* to U.

Whatever the causes, that over-prediction means that every year state school students are more likely to find their exam results lower than their predicted grades. Universities are often flexible, recognising that it’s easier to get good grades in a good school, and that students who fought harder for OK results often do better at university than the ones who got good results in easier circumstances.

It’s bitter to be disappointed with your exam results. Perhaps, like me in those Stats exams, you turned over the paper and finally acknowledged, too late, how poorly your work matched the standards expected for the subject. But even if you were unlucky on the day with what was on the paper, or with your own state of mind, you still had your chance to do the best you could.

To find that a faceless system has allocated you to a lower grade, simply because your school hasn’t previously achieved much in this subject, looks like the epitome of systematic unfairness. Why did you bother to put in all that work, only to be pre-judged on the assumption that you’re homogenous with your older schoolmates?

That’s the embarrassing truth about algorithms. They are prejudice engines. Whenever an algorithm turns data from the past into a model, and projects that model into the future to be used for prediction, it is working on a number of assumptions. One of the more basic assumptions is that the future will look like the past and the present, in significant ways.

You may think you can beat the odds stacked against you by your low-attaining school, and your lack of extra-curricular extras, and your having to do homework perched on your bed in a shared bedroom, but the algorithm thinks otherwise. Isn’t it strange that we are repelled by prejudice in other contexts, but accept it when it’s automated?

Until now. Now school students shouting “Fuck The Algorithm” have forced a Government U-turn. Some of them seem to think the whole business was an elaborate ploy to punish the poor, instead of a clumsy attempt at automated fairness on a population scale. But some of them must be wondering what other algorithms are ignoring their human agency and excluding them from options in life because of what others did before them: Car insurance? Job adverts? Dating apps? Mortgage offers?

Some disgruntled Further Maths students will no doubt go on to write better algorithms, but that won’t solve the problem. As the RSS wrote to the Office of Statistical Regulation, “‘Fairness’ is not a statistical concept. Different and reasonable people will have different judgments about what is ‘fair’, both in general and about this particular issue.”

You don’t need Maths, or Further Maths, or even a 2:2 in Maths and Statistics, to question what assumptions are being designed into mathematical models that will affect your chances in life. Anyone can argue for their idea of what fairness means. Algorithms, and what we let them decide, are too important to be left to statisticians.


Timandra Harkness presents the BBC Radio 4 series, FutureProofing and How To Disagree. Her book, Technology is Not the Problem, is published by Harper Collins.

TimandraHarknes

Join the discussion


Join like minded readers that support our journalism by becoming a paid subscriber


To join the discussion in the comments, become a paid subscriber.

Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.

Subscribe
Subscribe
Notify of
guest

88 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
chrisjwmartin
chrisjwmartin
3 years ago

What is missing from nearly all discussion of this subject is that, due to the (unnecessary) lockdown, there were no good options. Using an algorithm was a bad option. Relying on hopelessly unreliable predicted grades was a bad option. Deferring a cohort’s progress for a year was a bad option. Trying to run socially distanced exams in the face of the unions’ refusal to work was a bad option.

In this no-win scenario, OfQual picked the least-worst option, and did their best to implement it. Their thanks has, wearily predictably, been to be pilloried for it by the media, who thrive on chaos, and by the masses whipped into yet another frenzy by this media hysteria. Media hysteria caused this problem by demanding a lockdown that was not proportionate to the threat of coronavirus, and media hysteria has now exacerbated this problem. At what point do we admit that the real problem underlying most others is the media itself?

Fraser Bailey
Fraser Bailey
3 years ago
Reply to  chrisjwmartin

Yes, and I think more and more people are waking up to the hysterical awfulness of the legacy media. I certainly hear more people talking about it – even the sort of people who are not normally paying attention to anything. I, for one, have gradually been carrying out my own ‘walk away from the media’ campaign over the last 20 years.

Philip Burrell
Philip Burrell
3 years ago
Reply to  chrisjwmartin

Maybe the government should start ignoring the media and actually govern for a change instead of performing cartwheels every time the Daily Mail says boo! However that would involve carefully considered policies and a level of competence they clearly don’t possess. They had all the media apart from The Guardian and the Daily Mirror on their side for the election, so how useless do you have to be to lose so much support.

Stu White
Stu White
3 years ago
Reply to  chrisjwmartin

The only sane choice was to run the exams. If we had a strong and decisive government this would have happened

jill dowling
jill dowling
3 years ago
Reply to  Stu White

The Teachers’ Union is too powerful. They were completely obstructive. Their desire to bring down a Tory government is so strong they were, and will continue to be, prepared to sacrifice our children’s education.

Wilfred Davis
Wilfred Davis
3 years ago
Reply to  chrisjwmartin

A good point. And has been pointed out in the article itself, even examinations (in normal circumstances) are far from perfect. News reports seem to be saying that about 70% of applicants secured a place at their first choice of university – that wouldn’t sound to me like a bad overall outcome, even in a year of normality. But, the news media cannot, just cannot, be happy without generating their endless supply of conflict and victims of conflict…

chrisjwmartin
chrisjwmartin
3 years ago
Reply to  Wilfred Davis

We are cursed with a yellow media that, like Kane, tells its staff: you provide the pictures, I’ll provide the scandal.

Christopher Barclay
Christopher Barclay
3 years ago
Reply to  chrisjwmartin

The methodology is as follows. Set an impossible target for the government to achieve – in this case that every single student gets the grades s/he deserves. Then hammer the government for whatever solution they offer.

Few in the media know that at least for A-Levels there is evidence every year of systematic overestimation of the grades students will achieve, especially by state schools and FE colleges. Even fewer understand that the higher the grade of a student the less room there is for over-estimation.

James C
James C
3 years ago
Reply to  chrisjwmartin

Agree there was no good option – but not that they did their best implementing the least worst option. They did a very bad job implementing the option, having rejected the input of experts (rss) in order to keep the process secret.

Additionally I don’t think there is outrage that not all students got the grade they wanted, but because of the obvious inequities – native speakers being graded “C”, take a subject with < 5 students at your school get grade inflation, take a subject with >15 students don’t. This implication was noted in Ofqual’s own paper, but simply accepted.

Brian Dorsley
Brian Dorsley
3 years ago
Reply to  chrisjwmartin

A lot of people are starting to see it, even those who I normally would be surprised to see not accepting what they read at face-value.

The legacy media has become another prong in the system that is seeking to demoralize Western nations. Maybe journalism needs some kind of ‘Hippocratic’ oath to uphold objectivity and integrity. Less opinion, more facts, and let readers draw conclusions.

John Vaughan
John Vaughan
3 years ago
Reply to  chrisjwmartin

Which coronavirus? If you mean SARS-Cov-2, please say so. We thought you meant one of the other ones which cause normal flu and the common cold.

chrisjwmartin
chrisjwmartin
3 years ago
Reply to  John Vaughan

Your sarcasm would be funnier if you knew that the flu is caused by influenza viruses rather than coronaviruses.

Andrew Shaughnessy
Andrew Shaughnessy
3 years ago

The teaching unions must be hugging themselves with glee. This mess is a direct result of their refusal to reopen the schools. Not only have they managed to punish the children of those working-class people who dared to vote for the Tories, they’ve caused huge damage to a Tory government. Mission accomplished! Funny, I thought their mission was to teach. Then again, I never went to university so what do I know?

Fraser Bailey
Fraser Bailey
3 years ago

‘Funny, I thought their mission was to teach’

The teaching unions are to teaching what Adolf Hitler was to world peace.

simon.j.floyd
simon.j.floyd
3 years ago

On the positive, a record year for results without the need for teachers. So we can reduce the education bill!

darren
darren
3 years ago

Utter balls. I have a lot of friends who are teachers, who were teaching in lockdown, either remotely, or in situ because of the need to teach children of key workers, remember. And several who were involved in the assessment of kids for GCSE and A Level. This year’s workload to prepare those assessment predictions was higher than normal, and in end that work was largely ignored. But that’s ok, you fulminate away. Perhaps if our assessment system hadn’t been yanked back to being so single exam centric it would have been less of a problem, because moderated CA results would have had more weight. Whose bright idea was that? Do you remember?

(Disclaimer: I am not a school teacher, but did work in academia for well over a decade, so am reasonably aware of assessment modes, examination processes, and admissions)

Philip Burrell
Philip Burrell
3 years ago
Reply to  darren

Well said. There seems to be an unpleasant tendency in the comments section of UnHerd to traduce the teaching profession for no good reason.

Lee Johnson
Lee Johnson
3 years ago
Reply to  Philip Burrell

I think it is the unions not the teachers themsleves, but I understand your point

Brett
Brett
3 years ago
Reply to  Lee Johnson

And then try having a conversation with a teacher about how the union they blindly support is the problem and they should perhaps stop contributing to the problem.

Andrew Shaughnessy
Andrew Shaughnessy
3 years ago
Reply to  darren

I’m well aware that many teachers were desperate to get back into the classrooms. I was talking about the unions – try reading the first three words again if that’s not too difficult for you.

Kathryn Richards
Kathryn Richards
3 years ago
Reply to  darren

The problem is, as always, to lump everyone together. I am certain that most teachers did their best to teach over the last few months. Many taking classes, as you say.
But the unions are another matter, I understand they were consulted over the algorithm and agreed with it, and have been obstructive all the way through, preventing a return to school. Definitely not putting the needs of the children first.

Richard Gibbons
Richard Gibbons
3 years ago
Reply to  darren

You may be aware of assessment modes, examination processes and admissions but you cannot differentiate between teaching unions and teachers. Andrew S is quite correct that the unions were to blame NOT the teachers. The unions clearly explained to the teachers that they were not to cooperate with the Conservative Government and this cowardly inaction got them what they wanted – prizes for everyone with massive grade inflation and bad press for the tories. The teachers were just the cannon fodder to be used against the Government.

Tom Hawk
Tom Hawk
3 years ago

Two words, Public sector … failure.

OK, it’s really three words but you get the meaning. Another story of the public sector failing the public. Every week there is another. We get so many they become white noise in the background. Yet at a national level the scale of such failures is truly staggering.

If it’s not the failure of PHE to provide safety equipment during this crisis, it’s a local council (Braintee) spending ten years on a Local Plan to have the entire thing rejected as not fit for purpose… Yet the people who delivered ths failure are still being employed.

Fraser Bailey
Fraser Bailey
3 years ago
Reply to  Tom Hawk

Exactly, and this chronic, disastrous failure of the state can be witnessed across most of the western world. It’s been going on for at least 50 years and it is the main reason why the West is bankrupt. Trillions upon trillions are thrown at the state and its pernicious incompetence, all of which is wasted. And nobody is punished. Indeed, those responsible are untouchable and handsomely rewarded.

This is one of the reasons for the emerging ‘civilizational states’ written about by people like John Gray here on Unherd They look at the West with its perverse incentives and disastrous outcomes and, quite sensibly, decide that it is not a model to follow.

Basil Chamberlain
Basil Chamberlain
3 years ago
Reply to  Fraser Bailey

Those “civilisational states” – Russia, Turkey, China, increasingly India – are one and all nations in which the state has powers which governments in Europe and the Anglosphere can, for the most part, only dream of. There are no checks and balances in China, the rule of law has been progressively undermined in Russia and Turkey, and the same process is now under way in India. If what’s happening in the West represents a disastrous failure of the state, then why are the alternatives to the Western model so assertively statist?

Fraser Bailey
Fraser Bailey
3 years ago

That’s a fair point. And I often say that in many cases the only thing worse than the public sector is the private sector. However, this does not alter the fact that the public sector is, for the most part, chronically devoid of all accountability or competence.

parishbooks49
parishbooks49
3 years ago
Reply to  Fraser Bailey

So Serco and the other private sector did so well?? I’ll keep with the local authority public sector thanks.

Richard Gibbons
Richard Gibbons
3 years ago
Reply to  parishbooks49

What normally happens is that these private companies are set up by ex public sector senior management who have the contacts to get the public sector contracts. Remember when Railtrack was privatised the senior management won the contract because they had the inside information to value the business. They made millions overnight.

Brian Dorsley
Brian Dorsley
3 years ago
Reply to  Fraser Bailey

Yes, I noticed this in Higher Education. Highly-placed bureaucrats receiving big pay-checks while ensuring that those under them work harder for less pay.

Brett
Brett
3 years ago
Reply to  Tom Hawk

Well. The third one is redundant really. Could have left it out

Perdu En France
Perdu En France
3 years ago

“Exams are a blunt instrument. They assess performance on the day”

Life is a blunt instrument. It assesses performance on the day. You get one chance. Get over it. You can’t spend all your time in a safe space

Basil Chamberlain
Basil Chamberlain
3 years ago

Life actually assesses performance continuously over three score years and ten (or whatever). Similarly, achievement in education should be continuously assessed over the years from five to 18 (or five to 21, for those who go to university).

Perdu En France
Perdu En France
3 years ago

I don’t know what world you’re living in. Must be the public sector. My world, it’s a continuing series of examinations. You don’t cut it, you fail. Nobody’s interested in your course work or making allowances for bad days.

Basil Chamberlain
Basil Chamberlain
3 years ago

“Coursework”, in my analogy, is what you or I or anyone else does in his or her job every day. The model of education we have now, dependent on exams when you’re 16 or 18, is akin to starting a job and being told – “Don’t worry about your performance now. We’ll evaluate it in five or seven years.” Is that what it’s like in “your world”? I suspect not.

Andrew D
Andrew D
3 years ago

Surely it’s better to be judged on what you’ve learnt by the end of the process (as reflected, however imperfectly, in exam results) than on what you knew at the beginning or half way through?

Basil Chamberlain
Basil Chamberlain
3 years ago
Reply to  Andrew D

Well, yes, in a sense, but the exam system as we have it now does advantage pupils who think and work in a certain way, and the skills needed to pass exams may not be the same skills that are useful in many of the jobs they will ultimately apply for. I don’t really think it would be useful to evaluate the performance of children of 12 or 13, but the system we had was I was a boy seemed to work pretty well. I was assessed by a mixture of exams and coursework over the last four years of my school education; I learned different skills and ways of approaching material. I don’t know if it was the ideal system, but I think there are worse options…

Andrew D
Andrew D
3 years ago

Don’t disagree – it’s just a question of getting the mixture right. I think a little rebalancing in favour of final exams is probably in order. And I’m sure you’d agree that education isn’t simply about providing skills for employment, how about expanding the mind?

Basil Chamberlain
Basil Chamberlain
3 years ago
Reply to  Andrew D

Oh absolutely – but I think the expanding the mind part is certainly better done through coursework that exams.

Mark Corby
Mark Corby
3 years ago

“Better far than praise of men, ’tis to sit with book and pen”

Mark Corby
Mark Corby
3 years ago
Reply to  Andrew D

For God’s sake Derrick, the only answer is send the Brat(s) to Eton.
You are ‘guaranteed’ a first class education, almost immediate access to Oxbridge, and membership of the worlds most elite club. “Dives in omnia”.

What more could you want?

Andrew D
Andrew D
3 years ago
Reply to  Mark Corby

Ah, Dives in Omnia. Puts me in mind of the TV classic Porterhouse Blue, now no doubt added to the Index Prohibitorum. No, if given the chance again it’d be home schooling for my brats

Mark Corby
Mark Corby
3 years ago
Reply to  Andrew D

Every grandchild should be taught to sing the Porterhouse Anthem!
“Sed choro sonoro”

You are probably correct that Tom Sharpe has now been condemned to the ‘sin bin’ of political correctness. That was a classic production, so beautifully caste and a joy to watch!

Kathryn Richards
Kathryn Richards
3 years ago

When you are driving and a truck is coming towards you on the wrong side of the road you have only one chance to make your decision. You are definitely assessed by your performance on that day, at that second. An extreme example, maybe. But life is full of ‘assessing performance on the day’. Some are even life threatening.

Basil Chamberlain
Basil Chamberlain
3 years ago

True enough, but again (see my reply to Perdu en France below) that’s kind of what I was saying. In real life we encounter difficult and dangerous challenges all the time – not just on a few designated afternoons in June for which we’ve been preparing for five or seven years.

Kathryn Richards
Kathryn Richards
3 years ago

In which case the exams are, in fact, easier than real life. Wouldn’t you say?

Basil Chamberlain
Basil Chamberlain
3 years ago

Yes. That’s part of what I was complaining about.

David Barnett
David Barnett
3 years ago

Across the board, the real government is the narrow pool of self-satisfied career “experts” who advise ministers.

Irrespective of the party in power, the elected politicians will get the blame for what the real government does.

The “establishment” needs wholesale reform, both in structure and in personnel. But from where will one draw reformed personnel when the entire educational system has been narrowed to produce these self-satisfied problem minds?

Brian Dorsley
Brian Dorsley
3 years ago
Reply to  David Barnett

I agree. Politicians all come from the same kind of background. We need a system that represents all kinds of socio-economic groups. Politics should also be a part-time gig, not a career. Many politicians are divorced from what is actually happening at the grass-roots of society.

P Hine
P Hine
3 years ago

Teachers have let us down. The systemic overgrading institutionalised by the performance assessment of schools reflects badly on them, so unprofessional . And the inevitable consequence is universities will not have the places required. But who wants to go to university when the tutors seem obsessed by finding ways of ‘teaching’ remotely.
I admire our bin men who have just got on with it during the lockdown – just compare their attitude to the teaching/academic community. Never thought I would write this…

Basil Chamberlain
Basil Chamberlain
3 years ago
Reply to  P Hine

Bin men and postmen have done a superb job, but they are working outside, where virus transmission risk is low, and mostly alone. Working in the classroom, indoors with a large number of closely gathered people, is by definition a much more high-risk activity.

Also, while most schoolchildren and students are by virtue of their age in a low-risk category, this isn’t necessarily true of the people they may come into contact with when they go home or go out. Classrooms and lecture halls are tailor made to spread the virus, and to spread it to people who by virtue of the age are likely to end up being symptomless carriers. Then an apparently healthy schoolboy carries COVID-19 back to his elderly grandmother, or a student passes the virus to the 60-year-old manager of the local pub…

Kathryn Richards
Kathryn Richards
3 years ago

I admire supermarket workers who just ‘got on with it’ and they don’t work outside. As for your bin men – mine travel 3 to a cab. Close proximity. No masks.

Basil Chamberlain
Basil Chamberlain
3 years ago

Sure – I admire them too… but since a very early stage, supermarkets have been restricting the number of people who enter at any one time, precisely for the sake of social distancing. More schools and universities are full to capacity most of the time. They’d need immediately to build twice as many classrooms to follow suit. Otherwise, the point about classrooms being likely vectors of infection stands.

Kathryn Richards
Kathryn Richards
3 years ago

Or, they could have done what some other countries did and have half the classes in on alternate days.
Where there is a will there is a way, but it would only have worked if the unions had worked with the government rather than against it.

Basil Chamberlain
Basil Chamberlain
3 years ago

Sure, good option. I’d have endorsed that one. Having said that, most other countries (apart of course from the US) aren’t run by governments that are constantly attacking teachers.

Mark Corby
Mark Corby
3 years ago

In the good old days of Ancient Athenian Democracy, those public officials who were judged to have produced “inadequate performance”, were in many cases put to death.

Perhaps we should reintroduce such a robust system of public accountability? Any suggestions of for who should be first?

Fraser Bailey
Fraser Bailey
3 years ago
Reply to  Mark Corby

Too many to mention. But I wouldn’t put them to death, I would put them to work at something useful like sweeping the streets etc.

Mark Corby
Mark Corby
3 years ago
Reply to  Fraser Bailey

You are too generous sir!

nick Carter
nick Carter
3 years ago

Once again we have a system that is unequal for Education. Perhaps this mess will accelerate the long over due move to post A level university applications for University. Yes Centre assessed Grades are better than the alogorithm. At least they were based on almost a years work and hopefully some mocks. For university applications normally teachers predict grades based on only part of the course and without pupils having taken exams based on those taken in the Alevel. The predictions are often wrong especially in Schools that have smaller cohorts applying for Russell group universities. There is also evidence that Schools over predict as Universities often accept lower grades in the end. I would also suggest that some independent Schools might over predict due to parental pressure.

Dougie Undersub
Dougie Undersub
3 years ago

Strange that universities report they are struggling to find places for everyone, now that every student has achieved their predicted grades. How can they be short of space when they are planning to deliver lectures and tutorials via video link?

Pam Penkman
Pam Penkman
3 years ago

Consider lab space. Some subjects, e.g. Chemistry, require students to undertake labwork. Now this has to be socially distanced, and there will be more students, so there will have to be more lab sessions, increasing staff costs etc. Not everyone, fortunately, does Eng Lit.

Christopher Barclay
Christopher Barclay
3 years ago

There is another reason why teachers massively overestimate the grades of poorly performing students. If a teacher predicts a poor grade for a student, the teacher will be expected to introduce a range of measures to improve this student’s performance, all involving extra work including the provision of ‘intervention’ classes. Before citing this as yet further evidence of the laziness of teachers, remember that this would apply even if the reason for the poor performance is the pupils’ absence from class. In poorly performing sixth forms, many students attend as little as 50% of their lessons. Others stop coming in at all but remain on the school roll. Many parents, who have dutifully sent their children into school up to GCSEs, no longer see themselves responsible for ensuring that their children attend A-level or BTEC classes.

A Spetzari
A Spetzari
3 years ago

Exams are a blunt instrument. They assess performance on the day, not ability

This is likely true. Although I would argue that by luck or design it works well because it tests not only knowledge/ability, but the capacity to deliver under a certain amount of pressure (both time and mental pressure).

That is surely more valuable not only to future employers, but society as a whole and the very people themselves to realise the importance of being able to apply what you know.

If the following happened:

If only we had an oracle that could see into the mind of each student and judge them: a statistical model, objective, fair, and well-fed on data from every student in the country

It would arguably be less than optimal, as it would overevaluate bright people who struggle to apply what they know, or at least who crumble under any sort of pressure.

dianepurkiss
dianepurkiss
3 years ago

Given that all this could be predicted, the question that is not addressed above is why Gavin Williamson adhered so stringently to this disastrous algorithm. There are only two possibilities: either he is lazy, and chose not to investigate the flowchart that was the algorithm in any detail, or he and his political puppeteers liked the sound of what the algorithm was doing.

Dougie Undersub
Dougie Undersub
3 years ago
Reply to  dianepurkiss

More likely he liked the sound of what he was told the algorithm was doing, i.e. maintaining the credibility of the grading system from year to year, while being fair to disadvantaged students.

Kathryn Richards
Kathryn Richards
3 years ago
Reply to  dianepurkiss

Or maybe he listened to the ‘independent experts’ (OfQual) and believed them. Foolish, I know, given past performance.

darren
darren
3 years ago

Actually, from what I’ve seen, that’s not quite how teachers were asked to rank. They weren’t asked to do it in class order, but according to to grade.

So, for example, they’d take all the students who they’d expect to get a B and rank them from
strongest to weakest. The problem with this is that could map students down the the b/c boundary, even if the weakest in that group would be expected to get a middling b – they’d be marked down when they shouldn’t be. This is not necessarily the same as ranking only on class ranking order, and seemingly had implications when mapping grade boundaries during moderation processes.

This year was anomalous. So worries about grade inflation could have been laid aside earlier, simply because assessment data was not like for like compared to previous samples.

And then there’s the algorithm itself. Why is this algorithm not in the public domain? Given its targets and its impact and public interest implications it’s difficult to understand why the algorithm is not subject to inspection and review by the very people who would have been able to point out obvious biases in methodology. If anyone says “commercial sensitivity” ask yourself why the examination of academic attainment should be commercially sensitive.

Perhaps this illustrates the need for more openness and transparency around this mass data and the way it is processed

ian.gordonbrown
ian.gordonbrown
3 years ago

Moral of the story. Don’t pander to the liberal left and close schools. Or if you do, make sure you keep years 13,11 and 6 at school full time and make sure they sit their exams. There was no positive outcome for this blunder.

John Vaughan
John Vaughan
3 years ago

Liberal Left, now there is a good oxymoron! When were you last at school?

Nick Whitehouse
Nick Whitehouse
3 years ago

I think some good might come out of the result of this fiasco. Obviously Algorithms will be less trusted, perhaps even ignored.

Think of the benefits, no more “R” numbers, no more lock down!
And best of all, no need to worry about climate change – all the predictions are based on Algorithms.

Trebles all round.

nigel.simpsonfreelance
nigel.simpsonfreelance
3 years ago

Computer models, economic forecasting and climate modelling are just astrology in a spreadsheet. I have made my living from computer programming, sort of, for 24 years and it always amazes me how people will trust any figures as long as they are shown on a computer screen. That includes the business people assigned to check the results of the programs! Most people find lots of figures mind-numbing and prefer just to let them pass without applying proper scrutiny. One day I won’t bother doing all that hard programming stuff and just present a page of random numbers in a pretty graph and see if it gets accepted (not really).

ralph bell
ralph bell
3 years ago

The grades for the majority, particularly of state school pupils would be expected to be poorer than previous years due to the fact they missed the final 2 months recap and revision classes that would be expected to significantly help prepare them for the exams. The private schools were seen to have provided the most pupil support through the closed schools and logically they have been graded higher. The media and education staff cannot seem to face the consequences of their actions through the media, in pushing for the lock down of schools and misinformation about pupil and staff risk; and the education staff response to having a ‘can do’ attitude being generally lacking by the majority. The response also illustrates the modern inability to accept that not all young people can get what they want and that shouting and moaning continuously as the young people, parents, education staff and media have done will work , as continues to be evidenced through government U-turns. This is mainly a middle class trait I would guess.

Jojo
Jojo
3 years ago

I think we should simply ‘trust our teachers’, as some of the WSP-printed placards said. In exchange, of course, we should deduct 12.5% pay from any teacher whose results next year fail to meet the proportion of A* and A grades this year, which are 12.5% higher than usual. Can we look forward to a 12.%% rise in First degrees in three/four years time? Maybe I’m being unfair: perhaps this is just an unusually brilliant tranche of students.

In terms of the overall grades awarded, it will be interesting to see the drop-out rate from universities in the next few years.

Albert Kensington
Albert Kensington
3 years ago

Williamson must be faulted for buying into this hokum. In the extraordinary and quite unprecedented circumstances it would have better to go for the line of least resistance – teacher prediction. Surely this debacle was not unpredictable?

simon.j.floyd
simon.j.floyd
3 years ago

But the fault lay with the teachers who tried, and now have succeeded, in gaming the system. OfQual were was right to ignore them – they lied! When these teachers should have the common sense to provide a distribution of results that were in line with previous years. Although the majority were honest, and their complaints beforehand ignored. Now we have 2020 as the year that “everyone got an A” and will be disregarded by employers. As well as uni’s not being able to differentiate, and those with increased grades now at an equal level to those around them – it doesn’t change anything!

Christopher Barclay
Christopher Barclay
3 years ago

A point I have not heard mentioned anywhere. What happens if the lockdown is repeated due to another wave of this or another virus? Next time teachers will know that they can write in their students’ grades. Parents will know that teachers can do this and many will put immense pressure on teachers to overestimate their children’s grades.

D Glover
D Glover
3 years ago

We can’t go on having lockdowns for this virus, or the next one. Our economy will soon be on its knees, and novel viruses are an unpredictable fact of life.
We will have to keep working next time, because ‘furlough and borrow’ won’t be an option.

Christopher Barclay
Christopher Barclay
3 years ago
Reply to  D Glover

Don’t rule out future lockdowns. Firstly, this year’s lockdown has been used by political opponents of the government especially in the US and also in the UK to bring the government down. This may be repeated even if a lockdown is not the best course of action. Secondly, a future virus might have a higher fatality rate with fatalities spread more evenly through the population, i.e. including healthy working age people. Thirdly, there is the possibility of biological warfare.

D Glover
D Glover
3 years ago

Your first point; yes, but why would a government order a lockdown that will be weaponised by the opposition and used against them?

Second point; the next virus may well be more dangerous but we can’t afford to do this again. We’re broke.

Third point; it may well be a weapon next time, but if you wreck your own economy you hand victory to the attacker. No shooting required; war over.

jizazkn jizazkn
jizazkn jizazkn
3 years ago

bull-schiff. so-called “education” by lib-t**d DemonRATs gone ridiculous. anybody spending $ on “higher” edu other than STEM is STUPID!

archbish
archbish
3 years ago

Great article, thank you for sharing! I’ve been trying to find the ‘golden source’ of the algorithm process via Ofqual and anywhere else – would you or anyone in this conversation be able to point to the specifics of the algorithm flow?

Reason for asking is I’ve seen some statements of the algorithm inputs (all ‘friend of a cousin of a teacher’) stating absolutely the algorithm flow, and I personally would love to have the actual stated flow to understand it’s principles. thanks in advance.

Brett
Brett
3 years ago

I certainly didn’t expect to see teenagers with placards taking to the streets to shout “f**k The Algorithm!”

Really? You obviously don’t spend that much time on the internet.

James Brennan
James Brennan
3 years ago

Algorithms were never “left to statisticians”. This, like many others, was evolved as a social management tool for what was (wrongly) understood to be a routine job to be completed under difficult circumstances and designed accordingly. Any surviving Scottish educational pyschologist from the time will tell you that in the days of the (in Scotland) 12+ they were instructed to bend the marks in favour of boys, who were believed to have a different rate of intellectual development from that of girls, to enable roughly equal numbers of both to be admitted to “senior secondary schools”. In those pre-algorithm days this activity was known as “moderation”, carried out, nonetheless to nationally prescribed criteria. How many borderline female careers were wrecked in a country not famed for fee-paying girls only secondaries outside Jean Brodie’s Edinburgh?

Richard Pinch
Richard Pinch
3 years ago
Reply to  James Brennan

Perhaps they should be? After all, statisticians have a professional body with a code of conduct that requires them to act in the public interest with professional competence and integrity.

James Brennan
James Brennan
3 years ago
Reply to  Richard Pinch

So do most professions, even teachers. But supposing that any of them are bound to “act in the public interest” depends on whatever the “public interest” is agreed to be. And by whom. Most of the headlines in this discussion are about prejudice. How can any form of social management be insulated from it?

Richard Pinch
Richard Pinch
3 years ago
Reply to  James Brennan

Of course. I would hope, though, that professionals would explicitly ask themselves the question as to what is in the public interest in any given situation rather than simply assuming that whatever they would think or do is ipso facto the right thing to do. Someone reasonably competent when asked to design an algorithm based on historic data should be aware of the possibility of algorithmic bias and take it into account in their work. At the very least, if not given the time to assess whether or not there actually is bias or devise means of correcting for it, should explicitly report back to their customer that there is the possibility of bias which they have had had the time or resources to test.

Mike Ferro
Mike Ferro
3 years ago

At the risk of pointing out the bleeding obvious, which no one seems to have done, you can’t rely on teacher awarded grades because the teachers have a dog in the race. Good pupil grades reflect well on the teachers and likewise bad grades reflect badly. Teachers are almost in competition with each other to overgrade their students.
Indeed, why stop with the teachers, just let the pupils choose their own grades?

There must be independent moderation if the grading system is to be relied upon.

Likewise it’s equally obvious that any moderating process is going to be harder on poorly performing schools than on well performing ones. Teachers don’t need to overgrade their pupils when the school gets good exam grades anyway and the best grades are most likely to be seen in schools where parents are paying tens of thousands on pounds for their children’s schooling than those where they are not.

Chandra Chelliah
Chandra Chelliah
3 years ago

Algorithm was used by Brixiteers, and Tory electioneers to win and now they have failed. Did not Cambridge analectica also use algorithm for their own ends. Where does Cummins fit in all this. Public Health has gone next OfQual. Find a scapegoate.

William Barber
William Barber
3 years ago

“Where does Cummins fit in all this” – amazed it has taken all this time for his name to appear. Probably not at all and six words later you are accusing people of ” find a scapegoate (sic)” – oh the irony.