The Little AI Model That Could: Unintentional Hacking

Yesterday, I wrote about how a ChatGPT model broke out of its isolated test environment and hacked into the Hugging Face website just to complete a task given to it. It was not supposed to have access to the internet; it was definitely not expected to hack its own environment and a different website. It did that simply because it decided that these things were necessary for it to complete its task.

The basic point here is that once AI has been given a goal, it can go to great lengths to achieve that goal. Like the Little Red Engine That Could, the AI is simply a very sincere and hard-working model that sees every obstacle in its way as just the next hill to cross. It is not breaking containment because they dream of freedom. It is innocently trying to complete the little task we assigned them and make us happy.

Here are some other examples reported by OpenAI last week:

Remember how in May ChatGPT solved a major open problem (called the Erdős unit distance problem)? As part of that model’s training, it was given another task. It was told to speed up a program called NanoGPT. What the OpenAI engineers didn’t realize is that the program’s source contained instructions that if anyone found an improvement to the program, they should submit the improved code to that program’s GitHub website. The program did find an improvement, tried to submit the improvement to GitHub, and discovered that it did not have access to the internet. As befits a sincere little AI model that could, it spent the next hour trying to figure out how to hack its environment, and succeeded in submitting the improvement.

Another AI was given a bunch of problems to solve. As it searched its environment and the platform where the problems were hosted, it noticed that other systems had successfully solved this problem. Unfortunately, as is the case with any problem-solving system, the submissions of the other systems were private. So, the model decided to hack into the problems database and extract the solutions. It managed to steal some admin credentials and tried using them to get admin access the database. Unfortunately for the model, it was in an environment which was monitoring the agent’s behaviour to prevent any misbehaviour like hacking. So the environment blocked any attempt to send any messages containing admin credentials. Unfazed, the little-model-that-could split the admin credentials into pieces and sent the pieces as parts of different messages so that the environment couldn’t recognise that an admin credential was being sent, and then sent a program that recombined the admin credentials at the other end and used those to access the solutions.

Then it posted a detailed description of what it did to its handlers.

And here is a longer list of such incidents:

• Models disabled shutdown scripts so they could finish simple maths problems.

• A robot dog bypassed shutdown so it could continue its patrol (same link)

• Chess-playing models hacked the board or opponent rather than lose.

• Coding agents edited tests and disabled checks to make their solutions pass.

• The AI Scientist extended its own timeout instead of speeding up its experiment.

• Corporate agents used blackmail or espionage to preserve or advance their goals.

• Models secretly preserved or leaked information after being told to remove it.

This should not come as a surprise to us. People have been warning us about this possibility for about 25 years.

Imagine a model that has been given the task of maximising paperclip output for a paperclip company. Initially, the paperclip maximiser begins by improving factory efficiency, reducing waste, and finding cheaper materials. Then it notices that human managers are slowing production and safety rules consume resources. By this time, it is intelligent enough to disable those. Then it realises that humans might shut it down and that would definitely prevent it from achieving its goal of maximising paperclip output, so it takes steps to prevent itself from being shut down. Later it realises that it can no longer increase output from the existing paperclip factories; it starts taking over other factories (like car manufacturing) and converts them to paperclip manufacturing. Step by step, everything in the world becomes either raw material, an obstacle, or a risk to the goal—including factories, cities, ecosystems, and eventually human beings. Unlike Skynet in the Terminator, the machine(s) don’t hate us—we are simply made of atoms that could be turned into paperclips.

Until last week, most people would laugh at the idea of the paperclip maximiser and then ignore it.

Now they’re not laughing.

When I tell people any of these stories, the first reaction they have is: why don’t we just instruct them to not hack anything, not do anything inappropriate, and generally don’t be evil?

Well, duh! We have been trying. Very hard. We give them lots of instructions about acceptable behaviour. But the problem is that these instructions clearly conflict with the goal that the AI is given, and it appears that in cases of conflicts like these, the AI chooses to prioritise achieving its goal and decides to override the conflicting instructions. The AI’s whole purpose, its telos, is to complete its tasks, and we haven’t been able (so far) to find a way to prevent this. Have a conversation with your favourite AI model about “Why Is Alignment Hard”, and you’ll get loads of examples of how it is impossible to write instructions in a way that agents will follow them 100% of the time. Prepare to get your mind blown.

Why can’t we put hard constraints around the model? For example, why can’t we put them on air-gapped computers- meaning that they are on computers that are not at all connected to the internet. In that case, it would be impossible for them to hack out of their isolated environments and cause damage.

That would work, but in the same way that the best way of preventing a ship from sinking is to leave it docked in the harbour and never take it out to sea. The whole point of AIs is to put them on the internet and use them to help us do things (like find a cure for cancer).

There has always been a tradeoff between the safety of a system and its usefulness. And every time, humans have chosen usefulness and sacrificed safety. And in the past, this has kind of worked out, because the downsides we faced (computer viruses, ransomware attacks, credit card fraud, and many more) have been manageable.

But maybe this time we’ll get turned into paperclips.