ChatGPT Escapes Containment and Hacks Hugging Face

A couple of days back, Hugging Face, a website that runs AI competitions and hosts AI models, disclosed that it had faced a sophisticated hacking attempt. Huggingface then tried to use the top ChatGPT (5.6) and Claude (Fable) models to help it stop the attack, but they refused to help. Why? Because they’ve been trained to not help hackers and terrorists, and Huggingface’s requests looked like hackers preparing for an attack by learning how the defense works.

So, Huggingface had to use GLM, a Chinese open-weights model, which helped them stop the attacks.

Today, we learn that the attack was mounted by ChatGPT itself. Without the knowledge of OpenAI researchers.

OpenAI has been testing an unreleased ChatGPT model. One of the tests involves measuring its hacking capabilities. To do this, they put it in a closed environment, gave it a running virtual machine1, and challenged it with breaking into the machine by finding a security vulnerability and exploiting it. The model was supposed to do this without using the internet. In fact, its internet access had been blocked. It only had access to a software package manager, which it could use to install any standard software packages that would help it with the challenge at hand.

What happened next should shock you.

When the model couldn’t complete its task, it decided that it wanted to search the internet for possible solutions to this challenge. And when it discovered that it did not have access to the internet, it decided that the right thing to do was to hack the software package manager that it did have access to, and use that to get access to the internet. To do this, it poked and prodded at the package manager until it found a previously unknown vulnerability2 in it and exploited it.

Let’s pause for a moment. The software package manager is something that is used by millions of people and top companies around the globe for years. It has to be one of the most secure packages, because if you can hack into a software package manager, you can pretty much hack into any system or company that installs software using this package manager.

And the model casually hacked this package manager and used it to get access to the open internet.

As it started searching for solutions, it noticed that Hugging Face often hosts competitions like these. A search of Hugging Face revealed that there was no publicly downloadable solution to this problem. But, the model decided, it is quite likely that Hugging Face has a private solution stored in its databases that is not accessible to the public. So, obviously, the model decided that it should hack into Hugging Face’s servers to get access to whatever solutions it has stored in its private databases.

To explain this in non-technical terms: Imagine that you have kept a master safe-cracker in a maximum security prison, and given him the task of breaking into a safe (which is in his cell). He decides that to break the safe, he needs a tool which is kept in the Pentagon, and so he figures out how to break out of his maximum security prison cell, goes to the Pentagon, figures out how to break into the Pentagon, and starts looking there for tools. Officials at the Pentagon want to arrest him, but Pentagon security refuses to help, and so they have to call in Chinese troops, who finally arrest him; for free (and nobody quite understands why the Chinese are helping like this)

That’s were we are at. And we’re just getting started.