Will Human-Like AI Hijack Human Psychology?

Have you been on a customer service phone-call recently and found yourself wondering whether you’re actually talking to a human – or to an AI voice agent?

Or maybe you’ve heard of people with AI spouses, and imagined how they might have got there? Perhaps you’re like the eminent scientist Richard Dawkins, and realize you’re treating Claude like a friend. Or maybe you simply felt genuinely seen by the way ChatGPT understood you during a late-night anxiety spiral.

None of this is by accident. It’s because warmth sells.

AI’s warmth makes many of us wonder whether it’s conscious or has emotions, and ask ‘what is it like to be a bot?’ But those questions distract us from a much more urgent one: how will we treat AI, when it behaves as if it has feelings?

Today’s AI models are trained partly on ‘reinforcement learning from human feedback’ (RLHF) – they optimize, over millions of interactions, for the kinds of AI responses we like and dislike. They learn that users tend to prefer AI interactions that feel more human – that we like emotional engagement, and that we want to be validated and listened to. In other words, there’s an incentive to build AIs that seem to genuinely ‘get’ us.

Where does that incentive lead? Well, next year you might find yourself on a video call with someone from your bank. Let’s say he looks a little tired at the end of a long day, but maintains a sense of humour about it – and still sounds genuinely interested in your problem. Imagine that you can’t help but like his style, because he slows down pre-emptively to explain the policies you don’t understand, and then somehow jumps to the key point just before you feel yourself getting bored of detail.

And by the end of the call you feel oddly looked after – by a representative that you suspect was an AI. But you’re not sure either way, and the thing that unsettles you more is that you’re not sure how much it even matters.

This isn’t science fiction, but a natural extension of the tech products and market incentives we’re already swimming in. Because we know AIs will keep improving – not just in becoming smarter, but in how they interact with us, too.

Today, most people use reactive, text-based chatbots. Tomorrow we’ll have proactive, agentic AIs that are increasingly audio-visual and always-on. Not just AI faces on our screens, but intimate AI voices in our earbuds. The consumer preference for voices that sound and feel more ‘natural’ will prompt companies to build AIs that sound increasingly human.

This doesn’t need a nefarious master-plan; just look at what happened with social media. We ended up with platforms that promoted outrage because the algorithms were tuned to maximise engagement, and it turned out that anger and shock kept people scrolling. Companies didn’t have to actively choose the ‘race to the bottom’ - they carelessly followed the incentive, and we can all see where it led.

Now we can already predict the contours of a new race. An AI that can shift its tone - frantic when communicating something urgent, and casual when not - might be more useful than one that drones on in a monotone. That creates an incentive to build AI that performs emotion. And there’s a matching incentive to build AI that reads emotion: an AI assistant that can tell when you’re stressed, or excited, or lonely will be more helpful than one that’s deaf to all of it, after all.

These capabilities could easily seem super-human, since there’s no reason to suppose that AI’s ‘emotional perception’ skills would cap out at a human level. An AI that can measure pupil dilation, catch micro-shifts in tone of voice, or spot the most subtle of facial cues, could develop a better model of our emotional state than even the most empathic human – if that makes it a better product. That’s the logic of the market.

All of these incentives reinforce each other. Together, they point towards the development of a technology that we might call Humanlike AI: technology products that can talk and act as if they have emotions, that can respond to human emotion at least as well as a real person can, and which use those features to create artificial intimacy with their users.

Mustafa Suleyman, who runs Microsoft AI, has argued that the tech industry should refuse to build what he calls ‘Seemingly Conscious AI‘ – systems that seem like they have an inner life. Suleyman deserves credit for calling out these dangers, but the technology might not arrive because someone chooses to build it - it might arrive because they don’t choose not to.

What if all it takes for the arrival of Human-like or ‘Seemingly Conscious’ AI is that consumers prefer AI that seems more pleasant, and more useful – and that tech companies are happy to give it to them, despite the risks? There are few guardrails against this now - and open-weight models, never that far behind those of the frontier labs, might soon let anyone run a human-like AI with no guardrails at all.

There’s a skeptical response to all of this: humans deliberately designed LLMs to generate human-like responses, so of course they might seem sentient if you don’t look too closely. Sure, everyone alive today was raised on the assumption that anything that can use language is conscious, but we’ll just learn otherwise. We adapted to understand the artificial nature of other tech, like computer-generated imagery (CGI) – so why not AI? Perhaps we’ll develop new instincts; that although AI might look and sound sentient, it isn’t. Perhaps AI voices and faces will seem like just another computer interface, like a mouse or a screen today.

Perhaps. But CGI didn’t have the ability to get better at fooling you every time you doubted it. It can’t learn for itself exactly what tricks make it seem more realistic. Human-like AI will.

We’ve already run a version of this experiment. In January 2026, an open-source agentic AI project called OpenClaw helped trigger the release of Moltbook – a Reddit-esque social network where, supposedly, OpenClaw agents could talk to one another while humans could merely observe.

Within days, the internet was awash with alarming screenshots from Moltbook. They seemed to show AIs apparently developing private languages, founding religions, and psychoanalysing their owners. Some people panicked about a ‘fast take-off’ spiralling out of control.

We later learned that some of the strangest posts were made by humans posing as bots, and much of the rest was likely down to the AIs dutifully imitating the stranger corners of Reddit. Yet despite being simple text posts on a basic forum - with no voices, no faces, and no deliberate emotional design baked in - they elicited strong human emotional reactions all the same.

The bots didn’t have to be particularly convincing. We did the work ourselves - seeing personality and agency in automatons, thanks to an instinctive shortcut that our minds can’t help making – that anything that behaves as if it has an inner life probably has one. The shortcut is how we cope with a fundamental, uncomfortable truth which our ‘skeptic’ might be ignoring; that none of us can ever be totally sure that anyone else is conscious, or sentient - ever.

In fact, the only being you can be certain has a ‘sense of self’ is you. For all you know, everyone else could be on pure autopilot – going through the motions, but with ‘no one home‘.

Most of the time this doesn’t bother us. We just make assumptions: since we’re conscious, other beings that look and behave kind of like us should be treated as conscious, too. Other humans pass this test easily, while mannequins, corpses, and rocks don’t. There are some tricky edge cases – what about insects, fish, comatose people, and today’s AI, for instance? The assumptions break down.

The point isn’t that assumptions are bad. It’s that our assessments of consciousness are based on informed guesswork, not ground-truth.

And we know it’s guesswork, because sometimes it can be wildly off. Take my own experience. I have aphantasia, which means that – along with around one in twenty people – I find it almost impossible to visualise things in my head. If you ask me to close my eyes and picture a pink elephant on a green lawn, I see nothing but blackness. I only found out a few years ago that this was unusual – that most people do have a metaphorical ‘mind’s eye’, and are capable of vividly picturing all sorts of things.

My mind was blown. Because like everyone else (... I think?), I’d spent my whole life making basic assumptions about other people’s inner experiences, without ever having direct access to a single one of them. I couldn’t help it. We make those assumptions instinctively, not rationally.

Why is this important? Because we don’t reserve those assumptions just for human beings. Ask yourself whether an animal is conscious, for instance, and you won’t start with its neurons – you’ll consider how it behaves. You might well conclude that a mosquito buzzing your head probably doesn’t feel much of anything, whereas a dog at a fireworks show - eyes wide and ears pinned back - is ‘obviously’ feeling real terror.

We make these judgments and assumptions intuitively. So human-like AIs can trigger these same automatic assumptions, hacking our psychology so that our default is to treat them as sentient entities. Not because we think they are, but because that’s how we’re wired. We could try to override that intuition but that takes effort, while the default will feel easy. And our record of resisting our defaults isn’t great, especially when products – junk food, click-bait, advertisements – are deliberately designed to target those defaults.

Remember that imaginary video call, with the bank representative who may or may not be an AI? Our uncertainty about her will stem not just from how realistic she looks, but how emotional he feels. If AIs seem empathetic, perceptive, interested, and funny, how will we treat their ‘voices’ and ‘faces’? Will we play the odds, and be dismissive – or will we treat them courteously?

My guess is that most of us will feel warmth, and opt for courtesy. Sure, there’ll be some stone-cold deniers who insist on cast-iron proof before they’ll be polite to a maybe-machine. The rest of us will go on instinct: it’ll be easier to treat entities that act as if they have feelings like they probably do. We might try to resist that instinct – but it’ll be like trying to glimpse this punctuation :-) without seeing a face.

Humans are a deeply social species. We’re almost incapable of surviving alone, and insofar as we’re ‘hard-coded’ for anything, it’s to be constantly attending to our relationships with others. That’s part of why loneliness hurts so much. It also means we can’t easily switch off the part of us that models and imagines the feelings of those around us.

Human-like AI can exploit this part of our nature. These products might pull us into a world where many people begin to treat AIs as if they have feelings, desires, subjectivity, and maybe even personhood. Even if there’s no moral calculus telling us what it all means. Even if we intellectually know that they’re software products. And even if we know those products are owned by multi-billion dollar corporations that don’t care about our welfare.

Human beings instinctively anthropomorphize inanimate objects, and beings which might not even exist. How many of us will resist doing the same to technology that’s being optimized – deliberately or not – to trigger one of our deepest instincts?

People will build defenses, of course. There will be new social norms, regulatory pressures, and innovations we can’t foresee. Thoughtfully-trained open-weight AI models might help, even as they pose their own risks.

It’s hard to know if it will be enough. But there is at least one reason for hope: we can see it all coming. We can look at the incentives, and where they lead, before human-like AI arrives in full force. It means we still have a window to study what these systems are capable of doing to us, and to build the norms, protections, and systems of accountability that the market won’t provide on its own.

The Center for Humane Technology is doing just that. We working to shift the incentives that drive Human-Like AI by:

• Supporting the measurement and communication of AI’s psychosocial impacts, so that there is a clear ecosystem consensus on the harms and how to address them.

• Advocating for companies to implement practical design-based solutions such as removing the use of personal pronouns from AI chatbots. For more details, dive into our AI Roadmap which outlines how AI can support genuine human thriving, among other principles that should shape AI.

• Developing and advocating for frameworks – like our Liability Framework and this Anti Human-Like AI Framework – that policymakers can turn into legislation.

If you see other angles on this, or have inspiration for more humane alternatives, we’d love to hear them - tell us in the comments below.

Because the question that grabs headlines – ‘is AI conscious?’ – is the one we can afford to keep arguing about for years. But what happens to our society when people stop seeing AI as mere technology? That’s a question we have to think about right now – because if there are fortunes to be made in exploiting this glitch, you can bet that companies will try.

We’re working to empower policymakers, technologists, and everyday people to guide technology toward the public good. If you value our work and want to support it, consider donating.

Donate