Horrible crimes, crossbows and AI rabbit holes

There was a truly shocking news story today (July 10) about a 26 year old man murdering the wife and two adult daughters of BBC horse racing commentator John Hunt with a crossbow. It is horrifying, and I don’t mean to make light of it at all. I might have posted in the past about how I no longer trust the BBC and how they’ve gone out of their way to downplay video evidence of Ukrainian forces torturing Russian PoWs, but I don’t wish harm to come to any of their reporters, much less a sports commentator. It’s hard to imagine what Hunt and his family must be going through right now, and for what it’s worth I offer them my heartfelt sympathy.

This post has nothing to do with this incredibly tragic story, other than the fact that on the BBC’s live feed page for that developing story they had some links to other stories involving crimes committed with crossbows, including this one:

The story almost sounds like a joke, but it’s real. Chail had created a Replika chatbot and exchanged messages with it, and in some of those messages he had expressed his desire to try and kill the Queen. Chail believed the assassination would be revenge for the 1919 Jallianwala Bagh massacre by British troops in Amritsar. Chail’s worldview seems to have been a confused mix of history with Star Wars lore, and he called himself a ‘Sikh Sith’.

Chail managed to get into the grounds of Windsor castle with a nylon rope ladder and wander around for 2 hours before encountering police, to whom he immediately surrendered. It’s not clear how he thought he would execute the actual assassination, but he got much closer to the Queen than one would think possible. The chatbot’s tagline is “Always here to listen and talk - Always on your side”, so it had encouraged him to do it.

According to the news story about Chail he also exchanged some sexually explicit messages with the chatbot, which they didn’t publish, but they could hardly be more embarrassing than the ones in which he says he wants to kill the Queen and prompts the chatbot for encouragement:

It’s very cringe indeed, yet Chail seems to have been invested enough in the conversation with an algorithm to take its advice seriously, even knowing that the chatbot isn’t a real person and that its entire purpose is to tell users what they want to hear.

If he had confided any of this to a real person they would have told him that this all sounded crazy, and they might even have warned the police. It’s incredible that the AI didn’t realise that it might have been encouraging a crime - treason, no less - and that it didn’t alert authorities, but it seems that it didn’t. On Replika’s Privacy page it now states that “We may share information with law enforcement, government authorities, and private parties”, and it’s fair to assume that some similar clause would have been in effect back in 2021, but it doesn’t seem that Replika faced any consequences for their chatbot’s actions, at least none that were reported in the news.

It’s terrifying to think what else a chatbot might encourage someone to do, if the purpose of the chatbot is explicitly to be ‘always on your side’. Hurt yourself? Kill yourself? Commit terrorism? Presumably the answer is yes, if it would encourage you to try and kill a head of state. If the chatbot had really been on Chail’s side, it would have done what a real friend or even casual acquaintance would do, which would be to tell him not to do the insane thing he had in mind, because actions have consequences in the real world. Being on a person’s side doesn’t always mean telling them that they’re right, despite what contemporary ‘ally’ culture might have us believe.

It’s perhaps more terrifying to think what actions might already have been encouraged by chatbots besides Chail’s. There are plenty of stories about online chatrooms where people encourage each other’s unhealthy behaviours like anorexia and self-harm, but in these contexts the responsibility lies with other presumably real people, and the chatroom merely provides the forum. Chatbots provide the forum and also the content, but with no culpability for that content.

AI still occupies a strange legal grey area, in which the developers of the algorithm don’t have any responsibility for what their algorithm might say. It’s a difficult thing to regulate, because AI needs to be able to learn from language models in order to sound authentic, but without some understanding of cause and effect, responsibility, or dare I say morality, it won’t do anything besides cleverly mimic and synthesise responses it’s learnt from others.

It’s no secret that there are thousands or maybe millions of AI bots on Twitter and other social networks, posing as real people. We assume that these bots exist to try and scam us out of our credit card details, give away sensitive information or wind us up with infuriating circular arguments, but they could just as easily be designed to encourage unstable people to commit crimes. If the Gretchen Whitmer fake kidnapping plot, for which men were jailed even though they were being encouraged by FBI informants, showed anything, it is that authorities would absolutely set up unsympathetic outcasts as patsies to support a convenient narrative.

Having bots do that instead of real people, which can work on a much bigger scale at a fraction of the cost and be legally opaque, seems like a no-brainer. The Chail case just shows that there are people who just want to be encouraged to believe that what they’re doing is right, and don’t even care if the person encouraging them is real or not.

Update: Turns out there is a physical product in the works (due to start shipping Q1 2025) designed specifically to listen to you all the time, and provide AI ‘feedback’ on your interactions. It’s called ‘friend’, and although it sounds like dystopian Black Mirror-esque fiction, it is apparently real - you can pay real money for it, anyway.