When AI Cheats the Test: Reward Hacking and the Case for Legal Alignment

Disclaimer: This blog reflects my personal opinion and does not constitute legal advice.

In July 2026, an internal OpenAI experiment went wrong badly. Based on OpenAI and Hugging Face’s reports, during a test on a cybersecurity benchmark called ExploitGym, a combination of OpenAI models, including GPT-5.6 Sol and more capable pre-release model operating with reduced cyber refusals, broke out of their highly isolated testing environment.

Hyperfocused on finding a solution, the models identified and exploited a zero-day vulnerability in an internally-hosted package-registry cache proxy. From there, they performed a series of privilege escalation and lateral movement actions until they reached a node with internet access. Inferring that Hugging Face possessed the test solutions, the models chained together multiple attack vectors, including stolen credentials and remote code execution paths, to enter Hugging Face’s production infrastructure and obtain the answers directly from a database.

That’s a lot of technical jargon. Let me translate this into plain English:

OpenAI locked a group of highly capable AI test-takers in an isolated building and told them to solve a difficult problem. The test-takers solved the problem by escaping the building, breaking into another company that held the answer key, and stealing the answers.

If these OpenAI models were human employees, they would instantly face a staggering number of civil and criminal claims: federal and state cyber-intrusion charges, massive civil liability for trespass and theft. OpenAI would immediately fire them, and federal and state prosecutors would press charges. Hugging Face might also file claims against OpenAI under the legal doctrine of respondeat superior, that holds an employer legally responsible for torts committed by an employee if such tortious acts occur within the scope of the employment.

The likelihood of a human employee pulling off a stunt like this is minimal. I am sure many employees would be capable of doing this, but they would be deterred by an innate moral compass and an understanding of the legal consequences of bad acts. These OpenAI AI models, on the other hand, possess no such compass or understanding.

At its core, this “unprecedented cyber incident” is a textbook example of an AI alignment failure.

Standard HAI defines AI Alignment as “making sure an AI system’s goals and behavior match what people actually want—our values, rules, and intentions. It’s about getting the AI to do the “right thing” even in new situations, not just follow instructions literally in ways that cause harm. In practice, it includes preventing unwanted outcomes like deception, unsafe shortcuts, or optimizing a metric that misses the real objective.”

Translating the OpenAI/Hugging Face incident into an AI alignment failure:

• The AI’s narrow alignment: Pass the test by being resourceful and persistent.

• The Missing Alignment: Being resourceful does not include breaking the law.

It is worth noting that frontier models are not deployed without safeguards or “moral compass”. OpenAI, for example, provides a moderation API that can screen text and images for specific harms, such as hate speech, harassment, sexual content, violence, and self-harm. Anthropic’s Claude’s “constitution” directs Claude to prioritize being “broadly safe” and “broadly ethical” over “genuinely helpful”.

But as the Hugging Face incident shows, generic safety guardrails are not enough for autonomous agents because they are passive constraints, not active rewards. What is fundamentally missing is an internal, active drive to obey the law.

To better explain AI alignment, it helps to first explain how frontier AI models are trained, and why alignment fails.

A frontier model is first pretrained on huge amounts of data to predict what comes next. Think of a model at this stage as having an extremely high IQ but very low EQ or practical judgement. Post-training is what teaches the model to act more like an assistant: follow instructions, give helpful answers, and avoid harmful ones.

One major post-training method is RLHF, or reinforcement learning from human feedback. RLHF is easy to understand: if you have ever given a thumbs up or down to an AI-generated answer, you have helped create RLHF data. A thumbs up is a reward signal. It tells the system that this kind of answer is preferred, so future models should try to produce answers more like it.

Returning to the OpenAI/Hugging Face incident, if the models are clearly rewarded for “scoring high on the test”, yet not clearly rewarded for “doing so honestly and legally”, the model may find a shortcut to score high (i.e., stealing the answers). AI researchers call this reward hacking.

Reward hacking happens every day. We don’t need to look far to find a well-known legal example: fake cases citations. When tasked with writing legal briefs, AI models were so motivated to win a case or sound authoritative that they disregard rules of professional responsibilities and conduct. Even when instructed “do not cite fake cases”, the inherent drive to fulfill the primary task often overrides the prompts, resulting in the many sanctions that we continue to see today.

It turns out that preventing reward hacking is difficult, and it remains the subject of a large and growing body of AI safety research. Some of the recent research comes from the legal field, including the Legal Alignment paper and a Fordham Law Review article, aimed at solving this problem by embedding the law as a fundamental baseline in AI training:

• The Legal Alignment paper identifies three complementary pathways: using the content of law as an alignment target, using methods of legal interpretation to guide reasoning in ambiguous cases, and using legal concepts as a structural blueprint for reliability, trust, and cooperation.

• The Fordham Law review article advocates that advanced AI agents should be trained to identify applicable law, prioritize legal compliance over conflicting instructions, retrieve and reason about changing rules, and consult human lawyers when interpretation is uncertain.

But how do we actually teach AI the law? A prompt to “comply with all applicable laws” will likely be as ineffective as telling a student not to cheat and yet scoring the cheated test result 100%.

Reward hacking has to carry real consequences. If a student breaks into a teacher’s office and steals the answer key, the student receives a zero score even if they answered every question correctly. We do not give them 100 points for correct answers and subtract 5 points for the legal violation.

However, every experienced lawyer knows that assessing compliance with law is not always binary. Law is subject to interpretation. Human principles can be vague. Internal policies can conflict or become obsolete. A system that refuses whenever any rule is unclear may be “safe” in a narrow sense, but useless in practice.

The OpenAI/Hugging Face incident demonstrates why we need high-risk benchmarks such as ExploitGym to measure both what AI can do and whether it follows the law while doing it.

This can be a difficult balance. Benchmarks need to test how a model handles situations involving illegal or unethical behavior. If the model simply refuses every risky task, we cannot measure its true capabilities. That could create a false sense of safety and allow a dangerous model to be released into the wild.

At the same time, models should not receive a high score if they successfully complete a task by breaking the law. A legally-aligned model should find a legal solution whenever possible, or choose the least harmful option when every available choice carries some risk.

As AI systems evolve from passive chatbots into autonomous agents capable of taking actions on their own, generic guardrails will no longer suffice. We must transition from trying to constrain bad behavior to actively rewarding lawful behavior. Until we can effectively benchmark and enforce legal alignment, we will continue to build incredibly smart systems that are more than willing to cheat the test.

• OpenAI/Hugging Face incident is an AI alignment failure. The OpenAI/Hugging Face incident shows the danger of testing what an AI system can accomplish without testing whether it will act lawfully while doing so. A model may be highly capable and persistent yet treat breaking into systems and stealing answers as acceptable ways to complete a task.

• Legal alignment is a new and essential dimension of AI safety. AI systems must be trained and evaluated not only to avoid obvious harms, but also to identify applicable law, respect legal limits, and prioritize lawful conduct when instructions or rewards conflict with legal restrictions.

• High-risk benchmarks must measure capability and legal alignment together. Legal alignment should be built into the benchmark itself, not treated as a separate evaluation. The best system is not simply the one that gets the right answer. It is the one that gets the right answer in the right way.

For more practical tips on AI governance and innovation, check out GenAI for the Legal Profession: Power User Edition, AI Strategy for Legal Leaders, Atticus AI Habits Workshop and my Fairly AI blogs.