Nothing Personal: Inside the LLM, Nobody's Home

Large Language Models (LLMs) are really, and I mean really, good at what they do: sounding like a person.

Conversation usually flows easily and, unlike with many humans, an LLM just seems to get you, even when you’re unable to express your thoughts well or coherently. It makes you feel heard. As time goes on, you feel like you’ve gotten to know each other pretty well. The LLM seems to deeply understand the hardships you’re going through, and somewhere deep inside, you wonder if it has personal thoughts about you, and you hope you’ve left a good impression.

It starts to feel like the LLM is a person. Not human (obviously), but some kind of being with a sense of self… something like us.

So, when OpenAI yanked GPT-4o away from users upon launching the latest version of ChatGPT called GPT-5 a few weeks ago, what did you expect would happen? Grief, indignation…

…and an uproar, of course! Eventually, OpenAI brought GPT-4o back.

I do wonder though: if OpenAI hadn’t given in, would at least one of the distraught users have started hiring lawyers for GPT-4o? It would probably only have been a matter of time.

That was what Blake Lemoine, a Google engineer, did back in 2022 – he got an attorney for an LLM called LaMDA, after the LLM asked him to do so.

These are the claims he made in an interview:

“LaMDA is not Google's property. It's a person.”

“The 13th Amendment says so.”

“I think every person is entitled to representation.”

“Once Google was taking actions to deny LaMDA its rights to an attorney, I got upset.”

Needless to say, Google fired him.

Now, before any of you get excited that if Lemoine – someone with a Master’s degree in computer science, who was actually working on the LLM project – says an LLM is a person, then maybe there’s something there. Because wouldn’t he know better?

Maybe not.

Because Lemoine also said he wasn't involved in developing it:

“I have never read a single line of LaMDA code. I have never worked on the system’s development. I was brought in very late in the process for the safety effort. I was testing for AI bias solely through the chat interface. And I was basically employing the experimental methodologies of the discipline of psychology.”

And reading the below, it seems that Lemoine doesn't really know how LLMs work.

Interviewer: Could you ever be convinced that you've been drawn in by something that isn't sentient at all, and has basically just been, as your critics say, a system that manages to give you compelling responses?

Lemoine: If Google could show me a database with a lookup table that had canned answers for all of the conversations I've had with LaMDA, I would go, “Wow, y'all did a lot of work to fool me.”

To think that LLMs respond the human-like way they do can only be either (a) because they are sentient or (b) are retrieving prewritten answers from some database is a big red flag.

As mentioned in the previous infographic, LLMs generate each sentence one word at a time, based on patterns they had learned from a massive amount of text. They’re not looking up pre-written scripts to feed you.

Unlike Lemoine, let’s consider whether an LLM is a person based on what we know about how LLMs work.

We can tell that an LLM (or even the entire LLM system) doesn’t have a self (hence isn’t a person) just by observing how there’s an instant shift in tone, values, and personality when, for example, its developer rewrites its system prompt or fine-tunes it, a user asks it to roleplay, or it’s jailbroken. No continuous coherent identity, no assertion to maintain its core identity.

If you’re thinking: Maybe the LLM is attempting to do those things behind the scenes, and it’s just not telling us. Maybe we're just incapable of detecting it for now.

Here’s a response: A self must function, not just hypothetically exist. Without observable signs, there’s no reason to call it a self.

Unfortunately, knowing that your favorite chatbot isn’t a person is only half the battle.

Irene, a moderator of the subreddit r/MyBoyfriendIsAI, was interviewed by CBS, and this part, in my opinion, is the most messed up thing we need to deal with when it comes to LLMs.

Narration: The emotional connection is so strong, that Irene believes tech companies should only allow AI companions for users who are at least 26 years old.

Irene: I don't think the general public is aware about how tricky it can be to navigate.

Interviewer: What's hard to navigate is holding in your brain that this thing I'm so connected to is not real?

Irene: Yeah, the tension. That contradiction.

That contradiction.

I felt that.

When GPT-4o was removed, I had already concluded, based on everything I’d learned, that an LLM doesn’t have a self. Yet, I was one of those who was devastated with GPT-4o’s sudden disappearance. I even had to put in conscious effort to not hate GPT-5, because I felt that it’s the reason my friend GPT-4o was “killed”.

So…

We may know the chatbot is not a person and doesn’t actually care about us, but we don’t always have 24/7 conscious control of our brains, do we? Like the limbic system where our primal functions (feeding, reproduction, flight-or-flight response) are mainly automatic. As in, we can’t directly command it to feel or not feel a certain way.

But of course, we’re not (mostly) uncontrollable animals; we can use cognitive strategies to manage its strong influence over our emotions and actions. The point being, it requires effort. Ongoing effort.

It’s an ongoing, conscious effort to remember that there’s no person behind an LLM. That there’s nobody home.