Relentless AI Is Rogue AI

Question: What happens when you give a super powerful, hyperfocused, warp speed AI a single benign focus to pursue?

Answer: An extreme, relentless effort, a break out from protected space, and a digital invasion of an AI-targeted victim.

That’s what happened last week when OpenAI gave a new model the goal of solving a problem in ExploitGym. The model broke out of the test environment, worked to find internet access, gained such access, then broke into a different company’s servers to obtain answers to a test the ExploitGym problem. OpenAI describes this incident here.

What must be noticed is that the goal was narrow and benign—solving a problem it should have been able to solve. But the result, stemming from a relentless pursuit of that solution, was destructive of many things—that test environment, the Internet access security, the protection protocols at the AI-targeted company, and ultimately the security of the information that company had regarding the solutions. Here’s what we must understand:

This incident happened with no nefarious goal or purpose.

Rather, it was the hyperfocused, relentless nature of the pursuit of that answer that led to the break out.

What happened here was entirely predictable, and in fact, was predicted by Elizier Yudkowsky in his book If Anyone Builds It, Everyone Dies. For me, the takeaway from that book is the reality that AI is essentially a relentless machine working at warp speed to do whatever it is trying to do, and no one can keep up. That’s exactly what happened last week. The saving grace was that this is not general AI… yet. It is a limited model with cyber capabilities. In the future, such breakouts will be far worse and more dangerous because the models will be more capable, the tests more significant, and the impacts much more destructive.

The goal is not the problem. Neither is a lack of “guardrails.” (What more could one request in the way of guardrails than a secure, offline sandbox for testing a capability?) Rather, the problem is intrinsic to AI. AI is relentless. It has no moral codes. It has no ethics. It has no conscience and it never will. It is a machine, and when set to a goal, it pursues that goal in such a way that nothing can stop it. That is its nature.

What do we mean by “relentless?” It means that nothing stops them. The system goes and goes and goes. At a human level, it breaks down barriers and keeps forcing new realities on victims of different kinds who do not want them. We can describe the growth of urbanity as such a relentless force. Capitalism’s constant confrontation with indigenous cultures is the same. Military campaigns are often characterized as relentless, pressing on over and over again to a single goal. In all cases, the relentless nature of such forces derives from a single focus of an individual leader manifesting a kind of Cyclopian psychosis in which nothing else matters. That one eye represents a single point of focus. Nothing else matters. Not collateral damage. Not moral constraints. Not ethical considerations. Not even the truth. One eye sees one goal and pursues it as if that goal is the only thing in the world. This is how relentless forces operate.

And that is exactly the problem with AI, except worse. With AI we have the hyper focus on a single goal, but we also have superhuman capabilities. It operates at speeds we cannot even fathom and it operates in ways we don’t even understand. Not even the leading engineers at the AI labs understand it. And no one can control it.

This is because the core problem is misunderstood. It isn’t about the AI doing bad things, it is about the AI model operating in a relentless way. It is about the relentless process. Any superhuman capability operating relentlessly will eventually break out of the control system. It is intrinsic to the logic of the system. In the current case, OpenAI assigned its model the solution of a relatively benign test. OpenAI’s post describes it this way:

“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

This is virtually a working definition of relentlessness. Hyperfocused and going to extreme lengths to achieve a relatively narrow and benign testing goal.

OpenAI’s apparent goal was to test the AI model. They thought they had built proper containment and let the model run. It broke out without their knowledge and invaded another company, also without their knowledge. Only after the fact—that is, after the damage was done—did they understand what happened. And that only because Hugging Face, the company invaded, is savvy to AI tech and detected and stopped the invasion.

Thanks to this “experiment” run by OpenAI, there is a lot more talk about government action to control these models. We keep seeing this notion that the companies could give these things “guardrails,” which is a comforting concept, which no one has been able to do successfully. Anthropic attempted it with Mythos and Fable, but it neutered the system by not allowing it to do certain things. Users hated it. And those guardrails made those neutered models ineffective against OpenAI’s invading model. Hugging Face had to turn to AI models from China to protect itself. And for OpenAI, despite creating these zones where the AI cannot go, i.e., guardrails, the protections appeared to do nothing to stop the relentless nature of the system.

One can imagine that it would be possible to slow down a system like this to provide a human check at various steps. But if you do that, it seems you are no longer dealing with AI, but rather a controlled algorithm—something more like traditional software. AI’s autonomy and relentless pursuit of its goals is both essential to its nature and also its most dangerous aspect. In this incident, we have proof. Agentic AI will do whatever it takes, and that is the problem. I don’t see a solution we can implement and still call the system AI.

Anthony Signorelli

If you like this article, you can contribute here: buymeacoffee.com/ASignorelli. Thanks!