Wakeup call: The OpenAI/Hugging Face incident

OpenAI deliberately built an advanced cyber attack AI. It escaped containment and hacked its way into the computers of another AI company, called Hugging Face, in order to read secret information. According to some estimates, the technical sophistication of its attack was at a level more typical of state cyber warfare agencies than routine cybercriminals.1

Full details are not yet public.2 Simon Willison’s write-up is clear, brief, and level-headed. The implications are disputed; some opinions contradict what I say here.3 To keep this post short, I’ll skip addressing most complexities and counterarguments.

My book Better without AI is about “moderate apocalypses.” These are possible AI-driven disasters that are less catastrophic than “doom scenarios” of human extinction. That makes them less fun to discuss. They are still plenty bad, and so deserve extensive consideration. However, preventing them would be boring and expensive. So, these unpleasant “moderate” possibilities are mainly ignored.

Doom scenarios often feature a malevolent AI developing superhuman cyber attack capabilities. It grabs control of computers for its own use, gains access to secret information, controls communication channels, and takes over critical civilian infrastructure or military systems. OpenAI’s system did the first two of those things in the current incident.

Hostile states maintain contingency plans to use cyber attack capabilities to take over your country’s electrical grid, interbank transfer network, or military command and control systems. Recent advances in AI makes it seem increasingly probable that they would succeed. This could escalate globally to a “moderate apocalypse.”

Everyone has opinions about what AI will be able to do by when. No one’s are meaningful.4 You certainly don’t need mine. Instead:

This is about current technical capabilities that may prevent or ameliorate some bad outcomes. Everyone is ignoring these opportunities, out of short-sighted expedience.

“Develop and mandate intrinsic cybersecurity” is a short explanation I wrote four years ago.5 I’ll summarize it here, but suggest you could read it there as well. It’s only a thousand words.

To prevent AI disasters, there are three basic approaches:

1. Make the AI want to not do bad things, so it won’t.

2. Put it in a box, so it can’t do bad things, even if it tries.

3. Harden the things it might attack to make them less vulnerable.

The first is called “alignment.” Heated metaphysical arguments circle around whether an AI system can have its own desires, be malevolent, or act autonomously. This is unhelpful for this context. For cybersecurity, it doesn’t matter what AI systems “want” (if anything). People and organizations have goals, can be malevolent, and some will use the best AI technology available to take hostile control of others’ computers.

Boxing won’t work. OpenAI’s cyber attack system getting out of its box is an example. Plausibly OpenAI was negligent in building the box, and someone else could have done better. However, such boxes use the same technologies as in current cyber defense, which are definitely inadequate. Also, deliberately unboxed “open weight” AI systems are freely available. Their development lags the most advanced technology by only a few months. Preventing catch-up seems unlikely.

That leaves cyber defense: protecting networked computer systems from intrusion.

“Internet security is mainly fictional.”

The difficulty here is that nearly all software is inherently and unfixably insecure. It’s built using technologies that are known to make it impossible to write secure programs. And it is written with little or no concern for security.

Share

So the current cyber defense approach is to put software armor between the internet and the programs you want to protect.

The problem is that, although the armor is forged with the intention to make it as secure as reasonably feasible, it’s built with the same inherently insecure technologies. Also, the armor is extremely complicated, and inevitably has chinks in it.

This favors offense over defense. An attacker only has to find one hole in the armor, and then they gain access to everything inside. The defender has to try to fill all the holes. Consequently, as I wrote in 2022:

Internet security is mainly fictional. Almost daily there’s a news report about a major corporation or government agency that has been penetrated by human hackers who have extracted sensitive personal data of millions of people; or which has been shut down by a ransomware or DDOS attack.

This has persisted because:

Accountability for cybersecurity failures is nearly nonexistent. Organizations whose security policies and implementations were glaringly deficient, and whose systems were hacked, harming millions of innocent people, nearly never suffer significant penalties.

A few months ago, leading AI companies panicked. They put top priority on advancing cybersecurity capabilities. They offered exclusive access to their most powerful systems to “good guy” organizations, to give them a lead in creating defenses over enemy attackers. Project Glasswing was bellwether for this effort.6

It’s certainly better than nothing. However, the asymmetry in difficulty between defense and attack makes me think it is only a partial stopgap. Cybersecurity capabilities are unavoidably “dual use”: valuable to attackers as well as defenders.

There is a stronger alternative. Specific technologies can create computer systems that are intrinsically secure. Ideally, those can ensure that all parts are secure themselves, rather than hiding the mass of insecure software behind a patchwork of thin protections.

These are not pie-in-the-sky. They are well-understood, and in regular use in some practical applications. “Develop and mandate intrinsic cybersecurity” lists them, with links to explainers elsewhere. (I would now add to its list type-safe, memory-safe languages. I omitted those as too obvious to mention; their use is widespread and routine. However, they are usually avoided in security-critical software, because they run slower.)

I advocate adopting these technologies broadly and immediately. I imagine four objections:

• It is too late

• It wouldn’t work

• It would be too expensive

• Nobody can tell me how to program!7

I’ll sketch counterarguments for each.

Everything I’ve said in this post, and said years ago in “Develop and mandate intrinsic cybersecurity,” has been obvious for decades. Cybersecurity experts have been warning about this for approximately ever. What’s happening this month may be the first explosion in a rapidly-unfolding disaster. Or not.

It may be too late to put intrinsic security in place. Maybe the stock exchange or the nuclear arsenal will get pwned next week. Maybe there’s two years before that could happen. Maybe it never will. We don’t know.

For decades, infectious disease experts warned that a novel respiratory virus would cause a catastrophic global pandemic, and everyone was like “well, I guess we’ll worry about that when the time comes”. Authorities refused the simple preparations that could have significantly lessened the death toll, the cultural disruption, and the economic disaster.

FFS, get started now!

“Intrinsically secure” does not mean “absolutely secure.” There always remain gaps between theory and implementation. Human error can also neuter otherwise-perfect security; and humans do occasionally make mistakes.

Perhaps sufficiently advanced AI could find and exploit the gaps easily. We don’t know. However, known technologies can, at minimum, make that much less likely.

In the mean time, imperfect but significantly improved security would be greatly beneficial, even if cyber attack capabilities never advance beyond this month’s level.8

Project Glasswing, and similar efforts, are using AI systems to find and fix security holes in existing programs. That’s valuable as an emergency measure.

It should also be possible to use AI systems to accelerate the difficult and expensive work of writing intrinsically secure replacements. Although few programmers know how to do that now, AI systems could easily be trained to assist. With that, the seemingly daunting and overwhelming task may be quite feasible.

The place to start is with the “armoring” placed between critical systems and the internet. The TCP/IP stack, firewalls, web servers, database APIs, codecs, and so on. Anything with a string buffer. There’s finite quantities of that stuff. Everyone uses the same ones, so they only need to get rebuilt once, not separately for every software system.

The experience gained from that effort should further accelerate working inward, to harden the “business logic” of the most important application software.

This will cost a lot of money.

• Technologies for developing inherently secure software are more difficult to use. Few programmers know how. Training will be expensive.

• The amount of existing insecure software that needs replacement is extremely large.

• Secure programs are slower than insecure ones, so everyone will need faster computers for equal performance.

• The best security requires a different sort of hardware than is in current use. Although not inherently more expensive, it is not currently commercially available, due to lack of demand.

Adversaries taking down the electrical grid, gaining persistent control over the New York Stock Exchange, or disabling military communications in wartime would also be expensive.

Probably much more expensive.9

This won’t happen unless powerful organizations make it happen. AI companies deserve credit for spending billions of dollars this year to make a start. That includes both internal software development and grants to allied external organizations.

They have alerted business groups and governments. Large companies and government agencies are scrambling to react to the immediate threat. That is good.

They also should plan their transition to inherently secure technologies, and get started implementing that now. That will include instituting mandatory internal policies and spending a lot of money.

For governments, it may also include external regulatory mandates. Strengthening and enforcing penalties for allowing security breaches will be a minimum first step.