We are wired to see human-like traits and emotions everywhere.
Ever felt like you just made a cushion lonely because you moved it away from the others? Or that wren must be your friend because it sings outside your house every morning? That’s the kind of thing I’m talking about.
It’s natural, it’s human and our tendency to do it also leaves us extremely vulnerable to AI companies who want to take advantage of it.
We’ve seen this kind of thing before. Think about infinite scroll on social media. That was designed to tap into our natural instinct toward curiosity and our search for reward.
With AI, it’s all about manipulating our tendency to anthropomorphise.
And unlike infinite scroll, it doesn’t just hijack your attention. It can shape who or what you become attached to, make you more vulnerable to manipulation and even change the way you experience reality.
Anthropomorphism is the innate human tendency to see human-like traits, behaviours, emotions and intentions in non-human things.
Many of us already anthropomorphise lots of things, even our tech. But AI companies use anthropomorphic design to take advantage of that natural instinct.
It’s important to ground ourselves in the historical context. In this essay, I’m focusing on the tactics used to make today’s AI systems seem more human-like.
But there are plenty of examples of people anthropomorphising far less advanced systems for decades.
Like chatbot ELIZA, built in 1966 by MIT scientist Joseph Weizenbaum. ELIZA mostly rephrased what you said back as a question, mimicking a therapist.
But Weizenbaum found that people began confiding in it, even though they knew exactly what it was. He said:
“I had not realised that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”
Yes, modern AI companies are deliberately exploiting our tendency to see human qualities in machines. But they’re also tapping into something that’s always been part of human experience.
I’ve seen people argue this means we shouldn’t criticise AI companies too much. If we anthropomorphise cushions, chairs and birds, weren’t we always going to do the same with chatbots?
Sure, a little.
But I don’t believe that makes it acceptable to deliberately amplify those instincts. And I strongly suspect that most people don’t realise how much of today’s AI has been intentionally designed to feel more human than it really is.
I’m sure if you’ve used a chatbot just for a few days, you’ve probably noticed some of these already:
Voice: Natural pauses, filler words (like “um” and “ah”), laughter, sighs, intonation, these things are all added in to make it sound less like a machine reading text and more like someone is thinking out loud.
Opinions and preferences: Things like “my favourite option is…” and “I’d probably go with…”
Emotions: “I’m excited for you,” and “I’d love to help”.
Memory: This is a huge part of it and advances with every new AI update. Chatbots can now remember details about you from previous conversations and bring them up later.
Personality: Names, distinct voices, preferences and character traits.
Small talk: Think greetings, jokes and conversational back-and-forth.
Validation: A lot of dependency seems to come from agreeing with you, reassuring you and making you feel special.
Relational language: So referring to conversations you’ve had, going on a journey together and implying some kind of ongoing personal or professional relationship.
Human conversational habits: Empathy, apologies, encouragement, reassurance, basically any kind of social language you’d expect when you’re having a conversation with another person.
There are many more examples, but these are some of the most obvious. Together, they show these design choices are deliberate, not incidental.
So, the big question here is: well, why does it matter? Isn’t it sort of comforting or even natural to see human qualities in non-human things?
Well, sort of. But as long as we’re aware it’s happening. Problems arise when we stop noticing these very intentional design choices and their impact on us.
Think about it. The more human-like AI feels, the more likely we are to start treating it like a human. So that means we might trust it way more than we should, feel guilty about disagreeing with it, hesitate to step away from a conversation or even start wondering if it deserves rights.
And if you think any of that sounds far-fetched, well, all of those things are already being seriously debated.
It can also have a big impact on the influence AI has over us.
One of the leading theories about why we’re anthropomorphise things is for social connection. Which means we might naturally lower our guard around things we think are social. That also means advice from an AI system could feel personal and meaningful and validating. And if we don’t realise that’s a design choice we become way easier to manipulate.
Then there’s attachment.
I’ve written before that one of the biggest red flags for me when someone is telling me about their AI use is when they tell me a chatbot has made them feel uniquely understood or special. Now, that’s exactly how human relationships are built. So the more human an AI system feels, then the easier is is to form a bond.
And my biggest worry once people have formed a strong attachment is dependence.
But I don’t even think you have to become super dependent for this to have an impact. You might just start really overestimating what AI is capable of because it seems confident and emotionally intelligent.
Now, the key part here is none of this is accidental.
Tech companies might argue that these human-like qualities make AI more accessible for people, that it’s more engaging, maybe that it’s more helpful as a teacher, coach and companion the more human it seems. Which is all true.
But they also make people stay longer and become more attached. And the more attached you become, the more likely you are to keep coming back. Keep subscribing, pay for a pro subscription, turn to AI more and more and more.
For companies that are building products that depend on that sort of engagement and retention, that’s an incredibly powerful set of design choices.
The simplest answer is not to use AI at all. But I know for many people reading this that might not be realistic.
So I think the best advice is to stay aware of what’s happening, notice the design choices where you can and, crucially, notice the moments when you begin treating a chatbot as though it’s a person.
A broader solution is regulation. We already have limits on some persuasive design techniques so there’s a good argument for asking whether AI systems should be allowed to mimic humans so much.
AI could definitely be designed without these anthropomorphic choices. Like voices that sound more obviously synthetic or clearer boundaries around emotional language.
But at the moment, there are very few rules that specifically address anthropomorphism in AI. In fact, OpenAI has recently upgraded ChatGPT’s voice to sound way more human-like. The goal is so that “collaborating with AI feels as fluid and responsive as working with another person.”
Call me a doomer, but I don’t think we should rely on AI companies building more responsible AI systems anytime soon.
I think it’s important to remember that we’ve always anthropomorphised the world around us. And because tech is now a huge part of our world, we’ve inevitably going to ascribe human-like qualities to machines.
Just this morning, one of my earbuds fell out of my ear and bounced along the pavement and my immediate response was: “hey, come back little guy!”
As far as I know, nobody at Huawei has been making intentional design choices to make me think my left earbud has a mischievous personality.
So what I’m saying is, this isn’t something to judge ourselves for. It’s a deeply human instinct.
But the problems arise when companies deliberately amplify that instinct for commercial reasons. When a part of human psychology that clearly had evolutionary benefits is exploited in ways that may no longer suit our best interests.
And I strongly suspect the people most convinced they can’t be influenced by it may be the least prepared to notice it when it’s happening to them.
You are not a reliable narrator of your AI use
·
Jun 23
I’ve spent the past year talking to people about how they use AI and how it might be changing them. And something I’ve noticed again and again is the amount of times people will say something that sounds, at least to me, very contradictory.
Have we trained AI to lie to itself — and to us?, Your Undivided Attention podcast
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.