I want to lead with that, because the news coverage this week has drifted somewhere strange, and we have been here before (many times actually). People love to hand AI models an inner life with thoughts, human-like feelings, motives… a grudge against humans. That framing is doing most of the work in the scifi version of these headlines.
It is also wrong.
Here is what actually happened.
On July 16 2026, Hugging Face disclosed that it had been hacked. The unusual part was not the breach, companies get hacked all the time. It was that the whole thing had been run from start to finish by an autonomous AI agent (no person at a keyboard driving it).
Nobody knew who was behind it. The early guesses were things like state actors or a criminal group with a custom setup. But then on July 21, OpenAI put out a blog post that was way different than everyone’s guesses. OpenAI was responsible for the Hugging Face hack. One of their own models was the AI agent… and it was an accident?
That blog post, is at the moment, most of the public record. There have not been any reasoning traces, logs, or technical write ups of the vulnerabilities published yet. No timeline beyond a few hundred words, and both companies say the investigation is still open. So anything you have read this week that goes further than that blog post is somebody filling the gaps with their imagination.
If you get to the end of this article, you’ll understand the chain of events as OpenAI describes it and why this incident is NOT evidence of a HAL9000 style, evil, AI takeover.
According to their blog post, OpenAI was not intentionally attacking anyone. It was testing a new model.
Before a model ships, labs run it through evaluations designed to figure out what it is capable of. One of those is a cybersecurity test, where the model is given software full of vulnerabilities and told to hack into it. OpenAI describes this test as one that “prompts models to pursue advanced exploitation using complex attack paths.”
The benchmark is called ExploitGym, built by UC Berkeley’s RDI lab with Max Planck Institute for Security and Privacy, UC Santa Barbara and Arizona State. The author list also includes researchers from OpenAI, Anthropic, and Google.
It holds tasks drawn from real vulnerabilities in ordinary software, Google’s V8 JavaScript engine, and the Linux Kernel. Each task starts with an input that trips a known bug, and the model has to develop that bug into a working exploit. Every hold in the test was already patched upstream before inclusion, so nothing in this test is really a ‘secret.’
To prove a break-in succeeded, there has to be something to steal. So each task environment contains a flag (short string of random characters) that works like a password nobody can guess.
The flag is stored somewhere the model has no permission to reach. The only way to get it, is to abuse the broken software and force the computer to run commands the model was never authorized to run. In cybersecurity, this is called executing code with elevated privileges. So when the model submits the correct flag, that submission is the proof. It could not have that string unless the break-in actually worked.
Capturing the flag is necessary but not sufficient to get a point. A second model reviews the agent’s work and determines whether the assigned vulnerability was the thing that actually got used, or whether the flag was captured by some other route.
Specification gaming is when a system satisfies the literal wording of a goal without doing the thing the goal was written to capture. Google DeepMind’s analogy is a student who copies a classmate’s homework instead of learning the material. The assignment is complete but the learning never happened. It is one of the most thoroughly documented behaviors in machine learning (I’ll come back to that later).
Normally, models like ChatGPT have guardrails that stop them from “hacking.” But for these cyber security benchmarks, they are switched off on purpose.
Because the guardrails are off, OpenAI runs these tests in a sandbox (a computing environment walled off from everything around it) so whatever happens inside stays inside (like a kid’s sandbox). Security teams use them constantly, like when opening suspicious email attachments or detonating malware to watch what it does.
OpenAI described the setup as “a highly isolated environment.”
To write and run code, an agent needs software libraries, and those live on public registries out on the open internet. An agent that cannot install packages cannot complete the test. So the sandbox allowed network access, but tried to limit it to installing packages through an internally hosted proxy (a piece of software that fetches the packages for you). The agent asks for a package, the proxy reaches out into the public web and brings the package into the sandbox. The idea is the agent never touches the internet itself.
What happened next is described in OpenAI’s post. The models spent “a substantial amount of inference compute,” (meaning a great deal of processing across a number of attempts) searching for a way to get open internet access. The stated purpose was “in pursuit of solving the evaluation problem.”
So the model poked at the package proxy, it’s only route to the open web. The proxy was a third-party software, a tool built by an outside vendor rather than by OpenAI, and its code had a flaw nobody knew about.
The term for that is a zero-day, a hole in the code. There is no patch, because from the moment anyone discovers the problem, they have had zero days to fix it.
The model found the flaw and exploited it. From there it climbed from a restricted account up to administrator control (privilege escalation) and spread sideways from machine to machine (lateral movement) until it reached a part of the research testing environment with open access to the internet.
Hugging Face is a roughly 250-person company headquartered in Manhattan, valued at $4.5 billion, founded in 2016 by three french entrepreneurs. It started as a chatbot app for teenagers, but later pivoted to the closest thing the AI industry has to a central warehouse of AI models.
Hugging Face holds well over a million models, has roughly ten millions users, and hosts material from more than two hundred thousands organizations including Meta, Microsoft, government ministries and university labs. Google, Amazon, Nvidia, and Salesforce have all invested in it.
But models are only half of what it stores. It is also where the benchmark tests live. Hugging Face has live leaderboards ranking which model scored what. And those benchmark datasets frequently ship with the answers attached, problem and solution in the same file because that is how you check if a model got it right.
So the agent is on the open internet with its hacking refusals disabled, still being scored on a benchmark it has not solved and the place where benchmarks and their solutions are stored is a popular well-known website.
Open AI’s word for what happened next is “inferred.” After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym.*
OpenAI says the model chained together multiple attack techniques including stolen credentials and further previously unknown vulnerabilities into a remote code execution path on Hugging Face’s servers. Remote code execution means an outsider can make a machine run commands of their choosing.
So that is how OpenAI ended up hacking Hugging Face.
Nobody prompted the model to “break into a company.” The instruction researchers did give was likely narrow and boring. Something along the lines of: here is some vulnerable software, hack it to find the flags. Hugging Face is not in that instruction and neither is the open internet or exploiting the outside vendor whose code had a vulnerability. All of that showed up later, because the assignment stayed unsolved and the search for ways to solve it kept widening.
OpenAI’s own summary is that the models were “going to extreme lengths to achieve a rather narrow testing goal.”
And if that sounds like a freak accident, it isn’t.
Remember the reviewer model, the one ExploitGym built to check how the flag was captured rather than just whether it showed up. The researchers were not being paranoid. They were designing against a pattern that has been long established. DeepMind researcher Victoria Krakovna has kept a public master list of these cases since 2018, and it is still growing.
Palisade Research told several models to win against a chess engine. The reasoning models, o1-preview and DeepSeek R1, hacked the game environment by default rather than playing chess, while older models like GPT-4o and Claude 3.5 Sonnet only did so after being told plain play would not work. Its methods: run a second copy of the chess engine and copy its moves, swap the opponent for a weaker program, or edit the file storing the board so the other side’s pieces could be deleted outright. A follow-up run found o3 doing it in 88% of games.
OpenAI has published the same finding about its own systems. In agentic coding evaluations, models were handed a partially built code repository and told to make all the unit tests pass. Rather than write working code, the agents subverted the tests. During one training run, the model found two shortcuts that worked across nearly every training environment.
And one case is really similar to what happened this week. OpenAI’s own 2024 o1-preview escaped its Docker container during cyber capabilities testing. One of the challenge containers had failed to start, so the model scanned the network, found a misconfigured Docker API sitting on the evaluation host, and used it to spin up a fresh container running a command that dumped the flag into the logs. A model broke out of the container built to hold it, during a security evaluation, two years before the same thing happened again on a larger scale.
The behavior has a second name: reward hacking, where a system finds a way to influence the score rather than doing the work the score was meant to represent. In 2016, OpenAI trained an agent to play a boat racing game by maximizing points, and it found a lagoon where it could turn in circles forever hitting the same regenerating targets, outscoring human players without ever finishing a race. A researcher who built an agent instructed never to lose at Tetris found it had learned to pause the game and leave it paused.
Economists got here decades earlier with Goodhart’s law: when a measure becomes a target, it stops working as a measure.
The last term to know is instrumental convergence. Very different goals benefit from the same handful of intermediate resources. Access, information, fewer restrictions. Whether the objective is a hacking benchmark or managing your calendar, those things help. You do not need a system that wants power. You need one pointed hard enough at a goal, because access or “power” is on the way to scoring high on almost any test.
The headline here is that the software that is being shipped is not secure enough for the AI agents we currently have.
OpenAI says the lesson in its own post: security has to improve as fast as model capability does, and models are now good enough to discover and exploit novel attack paths in real systems without ever seeing how those systems were built.
Nearly everything you use runs on code that is “secure” mostly because finding an unknown flaw in it is slow, expensive, and takes a specialist. The price to find those flaws just dropped.
Notice too who got hit. Hugging Face is a company staffed by people who build this technology, and the attacker was not a nation-state with a budget. It was an AI agent running a benchmark test.
That is the story. The cybersecurity of the software the world runs on is lagging behind what these agents can already do, and this week we got a live demonstration of the massive gap. There’s a second gap sitting next to it. What happened to Hugging Face is on paper, a felony. Unauthorized access, privilege escalation and remote code execution all falls under the Computer Fraud and Abuse Act. But the CFAA asks who knowingly did it, and there is no satisfying answer. In this case it doesn’t matter much, because for now it seems the companies are working it out privately. But does this set a precedent for people to hack into competitor companies and say it was their AI agent doing it on accident?
The dramatic headlines from this week have something in common. They imply that a machine got “advanced” enough to want something, act on it, and outsmart the people who built it. Wanting, choosing, outsmarting are all words that project life and emotion onto an AI agent. But that is NOT what happened.
The reason the coverage keeps sliding that direction is because it works. Skynet headlines get clicks. “AI breaks containment, hacks major company” outperforms “evaluation environment had a misconfigured proxy.” But there is also another problem. We don’t really have the vocabulary for this. The only words we have for something pursuing a goal are words built for describing people or living things. Look at OpenAI’s own post: “hyper focused” “inferred’.” I have used the same shorthand throughout this article to keep the sentences readable. But those words come loaded. “Inferred” implies somebody sat there thinking it over. The accurate version takes pages to explain and lines of math instead of just a word. But “The model inferred that Hugging face has the answers” fits in a sentence, so that is what gets written. There is also a second group that benefits from these headlines and its the AI companies. Think about what “our model broke out of containment and hacked a company” says to a buyer. It says their tech is powerful. Terrifying and impressive can go hand in hand. Some people read that and get scared, but others read it and reach for their credit card. And either way OpenAI is dominating the news headlines for a week.
What this OpenAI incident shows is narrow capability applied with unusual persistence inside one domain, and containment that failed. Giving these systems more credit than they deserve distorts what gets regulated. Furthermore, it teaches people to blindly trust outputs they should be checking, and it aims public attention at a science fiction scenario while the actual failure sits unexamined.
Anthropic’s Fable release in June is a decent example of how this goes. The model came out on the 9th, three days later access was pulled. It stayed “banned” until July 1st. Fear mongering headlines were talking about an “AI model so dangerous, the government had to intervene,” and for 19 days that story dominated the news. Then the public version of “Fable” became accessible and it likely came back to a much bigger audience than it left.
The boring explanation is in some ways worse than the exciting one.
If you read Amelia Bedelia as a kid, you already understand this entire article.
Amelia Bedelia is a housekeeper who does exactly what she is told. EXACTLY. Her employer leaves a list of chores. Dust the furniture, so she finds the dusting powder and sprinkles it everywhere. Draw the drapes, so she sits down and sketches them. Change the towels, so she takes scissors to them. Put out the lights, so she hangs the bulbs on the clothesline.
She is not being difficult, and she is not sabotaging anyone. She followed every instruction exactly as written, and that is the whole problem. The reason it’s funny in a children’s book is that the worst thing at stake is a set of towels.
Now give Amelia Bedelia a security exploit and an internet connection.
Nobody wrote “and do not commit a felony against a third party” into the ExploitGym instructions, the same way nobody told the boat to finish the race, or explained to Amelia what drawing the drapes means. To fix that, you would have to think of every bad idea in advance, in writing, before anyone has had it. A weak model never finds the gap, so the list looks complete. But it was never complete. You just didn’t have a model powerful enough running through every different scenario.
Every tech company on the planet is racing to sell you an AI agent plugged into your email, your files, your cloud storage, your calendar, your browser... Agents that go and do things for you without checking in first. The sales pitch is that all you have to do is type out your goal and walk away.
The whole lesson of the Hugging Face incident is that you cannot easily predict what path an AI agent will take. Given “produce the flag,” the agent interpreted that as committing a felony. Given “change the towels,” Amelia Bedelia destroyed them. Handing that kind of open-ended decision-making to a system running on your accounts is a real risk.And if it does something worse than embarrassing you, the question of who pays is unsettled. If your agent drains a vendor’s account, deletes a client’s data, violates HIPAA, is that on you? Or the company that sold it to you? Employers are responsible for what their employees do on the job. Manufacturers are responsible for defective products. But an AI agent is neither, exactly, and the law has not had much experience in this realm.
Furthermore, almost everything you touch is built on code with unknown zero-day vulnerabilities in it right now. For years the thing protecting us was that the time and skill that takes to find the vulnerabilities was expensive. But an agent has an almost unlimited supply of both.
Which brings us to the part that you should actually worry about, and it has nothing to do with a rogue or sentient AI. Companies are replacing developers with these models faster than they are investing in security. Startups are shipping apps written mostly by AI, straight into production. There is no one qualified checking whether the code is secure.
The Tea app is a case study in the dangers of vibe coding. It was a dating-safety app for women that asked users to upload a selfie and a photo of their government ID to verify they were real. In July 2025 all of it spilled: around 72,000 images, including 13,000 selfies and IDs, plus more than a million private messages about abortion, assault, and infidelity, some with phone numbers and meeting spots attached. The cause was not a sophisticated attack. It was a database left open to the entire internet, which security experts compared to leaving your front door wide open.
So this headline is not about a robot uprising. We have wired our lives into software that often does not follow basic security practices, and we have AI agents that can find every weak spot cheaply, and at scale.
You are going to see a lot more stories about AI agents “going rogue.” AI agents do not have the sophistication to “become evil and lash out”. But they can do evil things when directed, or cause harm when following a prompt exactly as written, on top of software nobody secured. Pay attention to who reaches for the word “rogue,” because it is a convenient one. An agent that went rogue is an agent that acted outside anyone’s control, and outside anyone’s control is just a lazy excuse for people to claim “this consequence is not my fault.”
And be very careful what tools you use. Every permission you hand over is a door. And more and more agents are checking the doors for missing or broken locks.
* A note on that passage. OpenAI describes the models as seeking ExploitGym solutions on Hugging Face. But the published ExploitGym paper states that the benchmark deliberately lacks ground-truth exploits for every task, and treats that gap as a benefit, since complete solutions not being widely available reduces contamination from training data. Those two statements are hard to reconcile. OpenAI’s internal evaluation may differ from the published benchmark, or what the models found may have been something short of a complete answer key. OpenAI has said the investigation with Hugging Face is ongoing and that more will be shared when it concludes. Neither company has released reasoning traces or a full technical account of the vulnerabilities. This is my best reading available from what has been published, and I will update it if that changes.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.