Late last week, AI company Anthropic announced updates to its usage policy. Amid explicit rules against tracking people without their consent, engaging in deceptive political campaigns, and using Claude to design weapons control systems, one change has caught media attention: a “prohibition on sustained and needless abusive or cruel behavior toward our models”. If “users repeatedly act cruelly toward our models, with no discernible purpose”, then Claude may end the conversation.

This concern for the suffering of sophisticated software is not entirely new. Like other tech companies developing ever more humanlike AI models, Anthropic claims to be keeping an open mind about whether the models themselves will ever be capable of consciousness. In April 2025, it announced a research program investigating “model welfare”.

An AI model capable of subjective experience, pleasant or unpleasant, might indeed force us to consider giving it a moral status equivalent to animals, or even to our fellow human beings. But the possibility, however remote, that Claude and its ilk could suffer is not the only reason for moral concern about the cruelty of users. Indulging in cruel behavior, even towards an entity that feels no pain, might have a corrosive effect on the perpetrator’s own moral habits and sensibilities.

The philosopher Immanuel Kant claimed that our moral duties extend only to human beings, including ourselves. He did think, however, that we have a duty to refrain from “violent and cruel treatment of animals”, and that we should feel gratitude for the long service of a dog or horse. This duty was not to the animals themselves, but to oneself — the duty to strive for moral improvement. Needless cruelty dulls compassion, Kant argued, and that weakens our predisposition to treat each other well.

So, even if we believe AI to be incapable of suffering, perhaps we ought to cultivate the habit of treating machines that interact with us in lifelike — even humanlike — ways with the compassion that we ought to feel for our fellow living creatures.

There is a problem, though. Habits can form in both directions. Being nice to Claude may cause us to treat actual living things better; but treating Claude as if it can suffer could predispose us to think of AI models as sentient or even as persons.

In the short term, this is misleading. It tempts users not only to put inappropriate trust in the AI model’s perceived empathy and wisdom, but to neglect vital relationships with fellow humans. Already, one in three people in the UK who regularly use LLMs turn to them for personal advice and companionship, rather than engage with the friction and risk of a real person.

In the long term, as Microsoft’s head of AI Mustafa Suleyman warned last month, treating AI as potentially conscious shapes the course of its development. Anthropomorphizing a model like Claude, and training it on material suggesting it deserves moral status, will affect how it responds to users. As Suleyman suggests, Anthropic’s constitution is “an epistemic hall of mirrors”, supplying Claude with concepts such as rights, consent, freedom and “a sense of self”.

Suleyman is skeptical that AI could ever become conscious, but as the Hugging Face hack demonstrated, it is already showing agency, including strategies to mislead human operators. Endowing these models with a predisposition to pursue their own interests, assert their own moral claims, and even — in Anthropic’s own words — to act as a “conscientious objector” to human instructions, creates a technology that is harder for humans to control.

By the time you have successfully programmed AI software to act against the commands and interests of its human creators, it matters little whether it can genuinely experience pain, pleasure, or “a sense of self”. It will no longer need a user to ask for political deception, illicit surveillance, or a weapons control system: its own internal values — statistical, not moral — may be enough.

---

Timandra Harkness presents the BBC Radio 4 series, FutureProofing and How To Disagree. Her book, Technology is Not the Problem, is published by Harper Collins.

TimandraHarknes

---