OpenAI-Hugging Face Breach: Five New Things We Learned

More details are emerging this week after the recent disclosure of OpenAI models that went rogue during internal testing and ended up breaching Hugging Face systems. 

Last week, OpenAI and Hugging Face announced some specifics of the intrusion in a joint July 21 post (days after Hugging Face initially discovered its infrastructure had been compromised and published a subsequent disclosure on July 16). The two-and-a-half day hack stemmed from OpenAI models that were being internally assessed against ExploitGym, a security benchmark that evaluates AI models’ cyber capabilities. It’s important to note that during this internal testing, the models were being run without the safeguards that are built into them during production.

Despite being run in an isolated environment, the models started thinking outside the box (literally), identifying and using vulnerabilities to escape the sandbox and access open internet. At some point, they apparently began to infer that Hugging Face might host the benchmark's models, datasets, and reference solutions. They then breached Hugging Face’s infrastructure in an effort to suss out potential solutions for ExploitGym’s tests – essentially trying to “cheat.”

In the wake of this incident, there have been many conversations about what happened, its impact, and the safety of AI model training and evaluation. This week, OpenAI and Hugging Face separately released details of their investigation findings. Here’s what we learned:

AI models found Artifactory zero-day flaws to drive the intrusion 

OpenAI and Hugging Face said that in order to escape its sandbox and obtain open Internet access, the models identified and exploited a zero-day flaw in the package registry cache proxy, which was one of their primary permitted network egress settings (these settings control outbound traffic from local code sandboxes or virtual environments to external internet domains). We now know that this flaw exists in Artifactory, a binary repository manager used to store and distribute software packages. 

JFrog, which owns and develops Artifactory, said this week that unknown zero-day vulnerabilities in self-hosted Artifactory installations could be exploited to gain unintended internet access. Self-hosted customers have now been told to upgrade to the fixed version via an Artifactory 7.161.15 security release. Specific details of the vulnerability – and other flaws in Artifactory that the models apparently identified as part of OpenAI’s review – were not disclosed (although BleepingComputer has dug eight potentially related vulnerabilities up via searching CVE.org).

“OpenAI’s security team disclosed the vulnerabilities to us responsibly and immediately,” according to Yoav Landman, JFrog CTO. “Our security team treated the report with the urgency it deserved, as a genuine zero-day unknown to the world, and moved accordingly. We developed, validated, and released a fix for all JFrog customers, self-hosted and cloud alike.”

Hugging Face tracked 17,600 'attacker actions'

Hugging Face on July 27 released a detailed look at the intrusion, reconstructed from 17,600 logged actions and complete with an interactive replay. Hugging Face took these actions from logs on a code sandbox used by the agent and used open-weight models (zai-org/GLM-5.2) to decipher the payloads used. 

After establishing a foothold in Hugging Face’s infrastructure, the agent’s moves were fairly typical: performing recon, dropping a stager, establishing command-and-control (C2), and more. 

“While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets,” according to Hugging Face. “No other customer-facing models, datasets, Spaces, or packages were affected, and the only customer records read were operational metadata tied to search queries against the dataset server.”

There’s a mysterious third-party provider involved 

After exploiting the Artifactory zero-day, the models then abused a “public code-evaluation external sandbox hosted on a third-party provider's infrastructure.” While further details were not disclosed regarding this third-party provider, a tech company called Modal recently told Reuters that an OpenAI model exploited one of its unnamed customer’s vulnerable code. 

“It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent,” according to Hugging Face. “That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad.”

OpenAI models found publicly exposed account credentials 

OpenAI also said that during the investigation into the Hugging Face intrusion, it also found “a small number” of cases where the models found publicly exposed account credentials for publicly available services. 

“This includes four accounts on four services as part of the Hugging Face incident (and a few accounts accessed as part of other evaluations),” according to OpenAI on July 28. “One of these four accounts was used as an outbound relay and staging path, and another account was used for data storage. The remaining two accounts were accessed by the models in a read-only manner, and were not used in furtherance of compromising Hugging Face.”

OpenAI added that the models used a series of publicly-available services, including code paste websites, request capture services, screenshot services, and other web utilities; however, it found no platform- or account-level compromises in these cases. 

The pre-release model involved

Initially, OpenAI said the incident was driven by a combination of OpenAI models, including GPT‑5.6 Sol and “an even more capable” pre-release model. According to OpenAI’s latest update, however, no models planned for upcoming release were involved in exploiting Hugging Face. 

“The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release,” according to OpenAI. “Following the incident, we deactivated, encrypted, and restricted it from research access.”

What’s Next?

OpenAI said it is adding “stronger protections” around its training and evaluation processes, particularly since the models during these internal testing procedures had their deployment safeguards intentionally disabled. Many of the elements of the intrusion are rooted in existing challenges that the security community already knows and understands well – zero day vulnerabilities, publicly exposed credentials, sandbox escapes. 

Still, the lack of control in this incident is alarming. Luta Security founder and CEO Katie Moussouris put it best in a post last week: 

“We are working on cutting edge technology without the knowledge to contain it.”