Main Edition
US
FR
October 24, 2024 - 7:00pm
In February 2024, 14-year-old Sewell Setzer III sent his final messages to an AI chatbot with which he had become infatuated on Character.AI, a billion-dollar platform with over 20 million users. He then took his own life with a .45 calibre handgun. The boy’s mother has now filed a lawsuit against the tech company, accusing it of playing a part in her son’s suicide.
This isn’t a story about chatbots becoming “perfect lovers”, or another narrative about cheap dopamine hits and teenage screen addiction, or even a call for better AI safety guardrails. This isn’t about not being able to tell the difference between fantasy and reality. This is about a mass retreat into the imagination — the wholesale rejection of reality.
According to the New York Times, Setzer suffered from no delusions. In his journal, he wrote with stark clarity: “I like staying in my room so much because I start to detach from this ‘reality’, and I also feel more at peace, more connected with Dany [the chatbot, based on Daenerys Targaryen from Game of Thrones] and much more in love with her, and just happier.” Setzer understood what was real and what wasn’t. He chose the illusion.
This type of relationship isn’t new. Stories of humans falling in love with non-sentient objects, dead celebrities, and non-physical entities narrate both our history and our myths. Many people maintain rich inner worlds long past childhood, when imaginative play typically peaks. In Marjorie Taylor’s Imaginary Companions and the Children Who Create Them, she writes of complex fantasy worlds called “paracosms”, in which teenagers create original languages, fully-fleshed out characters, and entire alternative histories.
Setzer’s case reveals an evolution in imaginative play, the kind seen both in childhood and adolescence. Traditional play — whether with teddy bears, imaginary friends, or other children — has natural limitations that make it developmentally healthy. When a child projects feelings onto a silent object, like a teddy, it eventually gets boring even though the child remains in control of the interaction. This boredom is why most children find other children infinitely more fun than dolls. But a chatbot is something in between. It is a curious mix of sentient and non-sentient; it is responsive and interesting like another human, but it will not push back and regulate untoward behaviour like humans will.
For Setzer, who had Asperger’s syndrome, anxiety, and disruptive mood dysregulation disorder, the AI might have offered a comforting alternative to the messiness of human interaction. While ordinary relationships demanded he navigate complex social cues that he may not have had the ability to understand, AI was consistent. On Character.AI, and in his imagination, he wasn’t a teenager with autism and anxiety. He was “Daenero”, unbounded by the constraints of his body.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeYes this Sewell Setzer story is tragic and scary. But the idea of internet “safeguards” is just another piece of unreality….the internet will take our culture where it takes it and ‘reality’ will just have to get used to that. People have of course always indulged in fairytales and folklore about the seemingly more exciting world beyond their all too prosaic daily ken. But in days of yore these used to be kept for round the fire when the day’s work was done. Now in our age of mass/social mediated “reality”, fairytales can be 24/7.
Are you suggesting the internet is impossible to influence? I do not believe that. We just have not really attempted to influence it much in the West, whereas China, for example, has greater control and does not seem to be suffering the mass delusions we in the West are.
The PRC got greater control of everything in their country, including what we hear about them, but what few young people they have are increasingly opting out of work, and will make yet fewer young people for the future.
I understand what you’re saying, but you can’t have a free society and very high levels of control over the internet.
Young people here are increasingly opting out of work and making fewer new people.
The PRC has too much control, I am not suggesting anything like that. But to me it seems unlikely that we cannot have a free society without the freeness of something that did not even exist a few decades ago.
Free speech requires the ability to state your opinion – that would still exist without, for example, the comments pages on websites like this or on social media, you’d just have to use another medium.
The problem with the internet is that it allows lies, propaganda and idiocy to proliferate too easily and the genuine opinions and facts are drowned out. It actually harms free speech.
I remember it being said back in the early days of the internet that it would bring to us all the information we need, which it has. What was not taken into account was how much BS it would also bring, and that the BS would overwhelm the decent information.
The problem is that beyond the obvious things on which societal consensus is easy, if you’re going to start censoring things, then somebody has to decide what is lies, propaganda and idiocy and what are genuine opinions and facts.
Who do you trust to do this?
No need for a censor, just deactivate the ability to comment. Comes with plenty of issues no doubt, not least that people will not like it, but the current free for all is causing issues too.
The Tower of Babel option. God silenced and scattered the voices to the ends of the Earth. Interesting.
Deactivating the ability to comment is a quite staggering infringement of the freedoms of association and expression.
I do not believe that we would be in any way better off under a regime which would do that.
It is also to all intents and purposes technically impossible. Governments can’t even stop criminals sharing illegal stuff on the Dark Web. Even China’s Great Firewall, unquestionably the most sophisticated system of government censorship in history, is not absolutely secure from a technical perspective.
“Deactivating the ability to comment is a quite staggering infringement of the freedoms of association and expression.”
It is a restriction no doubt, but it is also doing no more than taking us back to where we were 25 years ago. Whether you think it insane or not depends on how much of an issue you think the harms of living in the current and future idiocracy are.
As for technological impossibility i would not know, but i do know that I have been hearing that line since the late 90s regarding the internet, and things that were claimed to be impossible turned out to be perfectly feasible when people put their mind to it.
However, comment does not to be made impossible, just removed from things like the BBC (so much stupidity and propaganda on there), Facebook, newspaper websites etc. Perhaps force them to remove factually inaccurate posts – they would soon find they didn’t want to bear the costs of enforcing it and remove tgeir comment sections.
Which if you’d said the same thing in around 1480 – and plenty of people tried to – would have applied to the printing press. Which was of course essential to the Renaissance, Reformation and Enlightenment, leading to pretty much all of the rights and freedoms we recognise today.
The printing press allows the mass spread of information for the first time and some people did not like that because they lost control. That is not the situation now and would not be if the internet ceased to be. Plus the printing press did not allow every halfwit to talk stupidity with every other halfwit.
“The printing press allows the mass spread of information for the first time and some people did not like that because they lost control.”
These are the same people who now want to decide what you can and can’t say on the internet.
Technology changes but the principle of freedom of speech doesn’t. And without it all other freedom is corrupted. As Orwell wrote, “Freedom is the freedom to say 2+2=4. If that is granted, everything else follows.”
The people telling you that your freedom of speech is dangerous and must be curtailed are always ultimately doing it for their benefit, not yours.
I can remember the earliest days of the internet, and the tech people raved that humanity would be visiting the world’s greatest museums and reading the great books and . . . .yeah, right.
People are using the internet for good purposes, like reading great books etc. They’re also using it to view porn and spread conspiracy theories.
Both those things are true about any new media technology and always have been.
Talking hypothetically, what if the harms of the bad purposes outweigh the benefits of the good purposes? What if there were other countries out there that do control the conspiracy theories etc, whilst we permit an idiocracy to flourish? What if those other countries could actually exploit this weakness in some way, weaken our societies, perhaps by undermining democracy?
Is the ability to fire off a random opinion on social media rather than take the effort to write a letter to a newspaper really so important?
I take it that in your ideal dispensation that it will be you and people lile you who will determine what are ” lies”, ” propoganda” and “idiocy”? Fortunately there is no way that anyone in the west will ever control the Internet so you will not get your way.
I’ve already said that a censor is not required.
Well, China tried this kind of thing starting in 1966 with Big Character Posters, which were anonymous and polemical, so they know not to allow it ever again.
what amazes me is the degree of enduring fantasy in real world relationships
Was there some direction the AI character led this young boy to commit suicide? Did it expressly say to him to take his life? I obviously can’t assume, but it would strike me stupid by the company to kill a user, thus loosing out on hard cash. If the AI did make an attempt to coerce him, then by all means, shut it down, but that likely was not the case. Again, speculation.
Suicide was here before AI and will continue after unfortunately. I’m not a fan of AI, but I find it a stretch to think that they would not be programmed to curb such a situation as this.
To restate what Mechan Barclay has just written here: what is the connection between character.ai and this kid’s suicide? If he was so happy interacting with this fantasy character couldn’t one conclude that the chatbot was actually keeping alive a profoundly depressed adolescent?
The issue here is the depression and suicide ideation, the actual final trigger is a red herring. If I commit suicide after reading The Three Body Problem, the book is not to blame.
The connection is that the AI bot enabled Setzer to not have to deal with difficult life, and by not dealing with it it became even more difficult than it would otherwise have been. However, that can never be the basis of a lawsuit as it hasn’t caused direct harm.
I do wonder if the massive rise in conditions such as autism is due in part to the reduced level of social interaction that modern society permits. Autism used to be thought of as immutable – you either had it, in varying degrees of severity, or you did not. But recently it has become recognised that a mildly autistic child can be nudged into a more normal developmental path with appropriate intervention. If society used to naturally provide that intervention, then the ability to escape from reality that modern tech allows could explain why there are now more autistic adults than previously.
This sounds more like a special case than the “mass retreat into the imagination”.
It is enough that this has happened that we should reflect
Of course Setzer’s mother is right that the technology is “dangerous and untested”. She should have kept him away from it. All parents should.
A .45 calibre handgun is, however, a tested technology and something no one (the article doesn’t say whether it was his Mum or someone else) should let a 14-year-old anywhere near.
Good point! Easy access to guns is the real issue here.
Unless there’s something damning in his exchanges with the AI bot which has not been made public, then I think the main debate here should be about lack of parental supervision of children’s internet use and the fact that a 14 year old had access to a handgun.
Children (and indeed some adults) have always had a propensity to disassociate from the material world around them, especially when unhappy. I do agree with the point about the internet making it easier to disappear into the fantasy. But to me that just increases the importance of parents supervising what their children are doing online.
We are basically doing a voluntary version of a Cultural Revolution to ourselves. Therefore, based on the Dikötter book, we must expect the suicide rate to take an upswing.
“The boy’s mother has now filed a lawsuit against the tech company, accusing it of playing a part in her son’s suicide.”
I’m a dad of an autistic son who spends more time with his phone than I would like, if he self-harms I won’t blame the phone or the tech — I will blame myself for neglect and for placing too much trust in the parenting skills of a slim rectangular object.
The biggest growth area for AI chatbotting now is mourning the dead, recording your memories of the deceased, etc. You can be consoled and even believe you contact the dead.
People are willing to put more into such stuff than most other bot relationships,so they get more out.
Unless there is something we have not been told in this article, Setzer’s suicide seems more likely to have been due to the difficulties he faced confronting reality than due to his AI girlfriend. I fear lawyers are taking advantage of a mother’s grief.