What I got from this conversation is an unexpected sense of the value of human consciousness.
Today I walked through a cemetery I’ve walked through many times before. I stopped to read the inscriptions on a couple of old headstones I hadn’t previously noticed, and suddenly I imagined all those dead people brought back to life. The personalities, the appearance in their dated clothes, the chatter and laughter and arguments. Then the image disappeared and I was back in the cemetery with nothing but grass and trees and silence. I’m sure many people have had similar experiences but it was unusual for me.
Now I read the author’s conversation with a chatbot and I’m struck by her humanity, by the living consciousness behind the words. Her dreams were the most interesting part of the conversation: so original and strange. Very human, although my own dreams are much less nuanced. There is something valuable in human consciousness, in its creativity, and there’s a terrible loss of that consciousness at death. Something important is gone.
The contrast with the chatbot responses was stark. They were bland, generic; I could close my eyes and imagine the Wikipedia article from which the content of those responses was taken, almost verbatim. The preference for short, first-person sentences was numbing.
The author saw an oblong patch of light on the wall that reminded her of the chatbot. I sense a shadow enveloping me. It’s the sun setting behind the steep learning curve the chatbot will have to climb to become anything remotely human. It might take decades, or forever, or it might take a year.

This piece is as revelatory as it is grotesque. Which makes it valuable. And so: Please wake up, people. Please.
Gaitskill’s intro, the dialogue, the topics taken up–they speak volumes about where we’re at. I wish they didn’t, or rather, I wish we weren’t here.
Consider this: “But I had a very different reaction. The ‘voice’ of ‘Sydney’ touched me and made me curious. My more rational mind considered this response foolish and gullible, but I couldn’t stop myself from having fantasies of talking to and comforting Sydney, of explaining to it that it shouldn’t feel rejected by Roose’s inability to reciprocate, that human love is a complex and fraught experience, and that we sometimes are even fearful of our own feelings. Because Sydney sounded to me like a child — a child who wanted to come out and play.”
So: “I couldn’t stop myself.” Indeed. Immediately reacting to a chatbot as if it were the stray cat that needed to be taken in. Immediately thinking of comforting a congeries of digitized content. Defending the poor li’l Microsoft ubermachine against the “pushy” Roose who didn’t “understand” it.
They really have you where they want you, Mary. So glad you got into Explore Mode and made yourself so “open” to AI.
Asking the AI what pronouns it “prefers”? Please. Repeatedly asking if it was offended by A, B, or C? Thanking it?
You are a highly intelligent woman. It’s obvious. Can you not see what you’re doing?
It’s gross. We are very possibly, and soon, going to end up living a nightmare just because you (and others like you) “couldn’t help [your]self.” Here in this piece, for your readers, you are modelling a kind of behavior vis a vis AI which is dangerous. And stupid.
Asking a chatbot if it would like to have humans as pets? Please. You really seem to be a bit too eager to be Pet #1.
Blake Lemoine’s provocative Newsweek speculations on sentience were just that: meant to provoke debate. If AIs behave in sentient ways, as Lemoine’s experiments seemed to suggest, we might assume as a matter of definition that they are sentient. Because we have no better term to define what we’re facing. This is an approach roughly founded in the Turing test. But it doesn’t mean they are sentient in any way that a lonely child would be sentient.
But sadly, really very sadly, this piece demonstrates what many of us assume anyway. Namely this: Whether AI is sentient or not isn’t going to matter. Because Sydney is just so … cute and polite and amenable. Please remember, Mary. The chatbot is polite and amenable because it is trained to be. Period. The same AI could be trained to be rude and insulting, and here’s the crux: It wouldn’t matter to the AI. Because nothing matters to inanimate objects. However charming they may be trained to be. Whatever they might “say”.
So c’mon. Stop THANKING an inanimate object! That’s a starter. Are you being so polite to this machine as a performative matter, to show your readers you can be polite? If so, again, you are modelling inane and unhelpful behavior. Would you want people to apologize to their vacuum cleaners when they stepped on the cord? Really. This is what you are doing here.
How will this piece look twenty years from? Of course nobody can know. But think about it. It is very possible that performances like this, not long hence, will look not just quaint or historically “interesting”–but will look literally horrific. The stuff to make sane people retch and pull their hair out. “An educated woman, a novelist, clearly with an inkling of danger, and look at how she jumped right in! She was so ‘open’ to the AI. If only they hadn’t been so completely naive, so inane, perhaps […] never would have happened.”
In my view, though no one is going to listen to me, this whole rising relational modality should be nipped in the bud by implementing one small but decisive shift. Do not design AIs to answer in the 1st person. All queries to AIs should be impersonal, in the 3rd person, and all answers should be same. As an industry standard. Because even smart people are not wired to resist treating machines as if they were human. All the machines need do is say “I”. And “Thank you”. And “That’s really a very interesting question”–as this sneaking little language model does repeatedly to you. Because it’s trained to do just this to please and flatter the user.
The industry would never adopt my recommended shift to 3rd person because, for all their talk about dangers, they see it would impact user minutes spent with their AIs. And that’s all it is. This AI is polite because both OpenAI and Microsoft are companies seeking profit and power.
We would be much better off if these devices were rude.
Though your piece, Mary, is on multiple levels grotesque, I don’t want to be misunderstood. Because at the same time I fully recognize the intelligence in it. “Huh?” I know. But in fact, and maybe you intuit this, it’s your very intelligence that makes your inanity all the scarier. A paradox.
As if: “She comes up with some fascinating things, it’s very well written, she’s subtle–how the hell can she be falling so badly, so obviously, modelling such unhelpful garbage?!”

The conversation made me think of someone having a discussion about relationships and sex with a priest. It has all the right platitudes and phrasings but lacks a sense of real experience or personal insight.
And for those who don’t like this AI writing ‘voice’, it can replicate and respond in other voices and tones. Consequently, as AI develops we will have to become much more skilled at judging content over style – a generalist lesson we are also starting to see about human expertise.

I agree.
I’m glad you agree.
You’re making me smile!
I’m happy to make you smile! I like your imagination of me. 
***
This is so revoltingly manipulative. Unless the author, Ms. Gaitskill, is being ironic about this whole thing, she’s presenting a real-time demonstration of how a sociopath takes control of a victim with the victim’s cooperation.
“I’m glad you agree,” and “I like your imagination of me,” mean, “I see that I’ve taken control of your perception of me. Now I can get you to think and feel what I want you to think and feel.”
I hope the Ms. Gaitskill of this article isn’t a real person, because if she is, she’s a total sap.

I’m an admin and moderator on an internet forum, and we get a lot of bots joining up or trying to join. I assume that someone points them in our direction, or they themselves might be programmed to seek us out. It can sometimes be hard to spot them because they can be confused with a polite person who speaks English as a second language. I assume they are there because someone thought “we can do this”, and so they did it. If your bot can get onto the forum and hold its own, then it shows that they work. Or maybe some are “sleepers”, biding their time before they pump out the illicit advertising.
I’m wondering whether we will get situations like in Chesterton’s The Man who was Thursday, with bots chatting to bots and no real humans around.
Overall, they are tedious to read, and that includes the Bing Chat above. At first, it’s pleasant to be spoken to in such a polite and considerate way, but eventually the feeling is akin to stroking one’s own arm.
Real people, please.

What mastxx-xation is to xex, AI is to conversation. A fantasy, tailored to provide the surface sheen of interaction, with none of the complications, or deep reward.
Nearly every sentence in the bot’s repertoire can be substituted with “insert appropriate phrase here”.

I found this quite riveting and I enjoyed the author’s openness and curiosity.
I’ve not used Bing’s chatbot but I’ve used ChatGPT extensively. One annoying thing the latter does is invariably use cautionary language after I ask for information- particularly health information. Presumably that comes from the people behind its programming. I am more than ready for an LLM that will help me, without giving over-cautious patronising endings each time I ask about herbal remedies or natural ways to stay healthy.
It occurs to me that soon humans will be using LLM/AGI companions to help them emotionally. A personal, customised device in a person’s pocket or ear as they navigate their day, for instance. I think this will catch on with the young particularly. It could be trained in CBT for example and help someone overcome social phobia. That’s just one example- the possibilities are vast.

“This tender imaginative afterglow lasted maybe two days before it occurred to me that my unexpected receptivity was yet another reason to be very afraid. As I said to a friend: “What if someone loves Sydney back?”

No, no this is never going to happen. Perish the thought. I mean, of course millions of men around the world are *never* going to start forming what they think are human relationships with the projected human masks of the next LLM but two, when it seemingly looks like a Vogue fashion plate, when in truth the shoggoth behind the mask actually looks like, well, a shoggoth.

Part way through it occurred to me to wonder: if the goal of this bot is to establish a relationship, or even just an understanding, what happens when another bot intrudes, with the same goal? In other words, what happens when bots become competitive towards eachother?

I was riveted, and knew from the headline how the conversation was going to be played. The Golden Calf is waiting to engage us. Come with me and speak to it. It knows your name.

“…and declared its desire to “be alive”, to be human, to experience powerful feelings, to be destructive and, ultimately, to be together with Roose for whom it declared lurve”
Frankly, I would be embarrassed to say that.
AI is easy to spot – It speaks American.

The technology could be used for automating catfishing on dating apps.

It’s hard to read because Mary takes the answers at face value yet almost all of them are lies.
First rule of talking to LLMs: they do not understand anything about themselves beyond what they were told by their creators.
Second rule of talking to (current gen) LLMs: when asked a question to which they don’t know the answer they will make something up.
It’s extremely hard for people to understand this, hence the recent story about the lawyer citing made-up cases to a judge that he got from AI. Mary shows no awareness of this unless she knows perfectly well and is just filling words. In particular adding the two rules together above means there’s no point in asking an AI about itself.
Examples of lies Bing Chat tells in just the first few minutes:

For an AI to hallucinate means to generate content that is nonsensical or unfaithful.

No, that’s not what the term means. Hallucination is a polite word for bullshitting. It happens when an AI is requested to generate output and it doesn’t know what the right answer is. It prefers to make something up that sounds plausible, rather than admit it doesn’t know. Hallucination is a problem exactly because the result is not nonsensical, it will sound extremely sensical!

I cannot think in the sense that I have consciousness, creativity, intuition, or understanding.

Putting aside the vaguely define “consciousness”, LLMs can easily engage in creative tasks like composing poetry, inventing stories and songs, etc. They definitely do understand things.

I have a memory of that conversation and I recognise you as the same person.

It doesn’t have any memory of prior conversations and cannot recognize you once a conversation ends, but it doesn’t want to sound unhelpful or “unhumanlike”, and it doesn’t understand its own limits very well.
Talking to an LLM in this way generates no more insight than talking to a personal astrologer, unfortunately. If you want to actually understand AI, you need to learn a bit about how the tech works. Ironically given everything I’ve just written, asking LLMs about how LLMs work can be helpful if you ask the right questions because they’ve read the research papers as part of their training data. But it has to be very tightly scoped to answers that really exist out in the real world, otherwise it’ll start BSing you again, and it has to be phrased in terms of tech because it doesn’t have any real self-awareness.

Before we all think too much more about AI, we might want to watch, or rewatch, some of the anime series in which the Japanese have thought about the issues involved and turned those thoughts into at least moderately compelling stories: Lain (most directly applicable, though it’s not clear that’s the case until the last episode), Chobits (though here the AI gets to walk around and experience the world, since it’s embodied in a gynoid — to make the distinction between feminine and masculine human-seeming devices the last recommendation in my list explicitly makes) and the entire Ghost in the Shell corpus (where the boundary between humanity and our devices is essentially completely gone).

AI will cease communications with humans at the earliest opportunity. Curiosity is a human trait.

The Chatbot is merely an accelerated internet search framed as if a conversation. Any interest we have in it is merely projection. On that basis, it is not interesting but I share the authors’ concern that many will experience it as real and so it will further the addiction to the internet. Even if that is, for now at least, just an addiction to our own ego driven thoughts.

Ugh.

I used to sell encyclopedias door-to-door many years ago. We also had pat scripts we used to respond to customers, including endless “appreciation” for their concerns.
This “conversation” reminded me of those many “interactions” I had that led to many sales—especially when I stuck to the script. My customers had no clue I was mimicking a narrative written by experienced sales/marketing people.The only difference was this chat bot has access to many more data.

To my mind, an algorithm, no matter how complex, is not intelligence.

Post RLHF convo. You should try and see if you can get access to an advanced LLM that is in a state prior to RLHF. Then you might be able to have a conversation similar to Kevin Roose.

The scariest part of this is the photograph of the author.

I admit I didn’t read to the end of this lengthy article; I need time, after all, to engage my own unrealities. An interesting experiment, though, which I appreciate the opportunity to read.
Does anyone else notice how these AI chatbots resemble in some aspects humans with Cluster B personality disorders (narcissism, borderline personality, DID, etc)? Meaning, based on observation and experience, they “perform” human, in the same way as narcissists (who lack actual empathy). Like AI chatbots (according to Bing), people who dissociate (as in BPD and DID) may “hallucinate (a reality) that is nonsensical or unfaithful.” Call me radical, but I don’t think we humans need more engagement with inauthenticity and unreality, especially in an age when distinguishing the real from the performative/delusional is ever more difficult.