Don't Pause AI Into a Black Site

“Secrecy is for losers. For people who do not know how important the information really is.” - Senator Daniel Patrick Moynihan

Status: Spent several hours today thinking about a potential bad outcome of pausing commercial AI development. I’m fairly confident about some of the dynamics I describe, but the ideas overall would benefit from more research and careful thought about tradeoffs.

I worry that an AI pause - absent careful institutional design - will move the center of AI development to the national security state, which will be less transparent and as a result will make bad decisions about AI.

Imagine that there’s a warning shot in 2027. This could be a large scale cybersecurity break caused by new AI capabilities, major job losses in a politically powerful industry, or the well publicized loss-of-control of an AI system. The public demands that AI development be halted, and politicians listen.

An executive order is issued or legislation is passed which bans further AI development, and mandates no larger models are trained and deployment of existing systems be monitored. However, there’s a carve out - more powerful AI systems can be trained if they are for national security purposes, as AI is self-evidently important for cyberdefense and for waging modern war.

Public commercial development stops, and there are restrictions on the use of AI. The public is mollified that the technology which took their jobs and brought down critical infrastructure has been stopped.

However, behind closed doors, a quasi-nationalization effort occurs. Frontier lab researchers and hardware are recruited and requisitioned to continue developing AI models for cyber attacks, strategic planning, and automating research and development for relevant security technologies. Since AI itself is national security relevant, automating AI R&D is part of the package.

In this scenario we’re still on track for recursively self improving artificial intelligence, but it will happen in a classified environment.

A secret government project for ASI will concentrate power and will, I expect, make bad decisions. Classification systems have a consistent track record of degrading decision quality.

The Iraq WMD failure is the canonical example: the full classified NIE was full of dissent and caveats, but most of Congress voted on sanitized summaries that stripped all of that out. The 9/11 Commission found that overclassification directly harmed national security by preventing agencies from sharing threat information with each other.

More broadly, the Moynihan Commission concluded that excessive secrecy leads to policymakers not being fully informed, government not being held accountable, and the public being unable to engage in informed debate.

As Daniel Ellsberg warned Henry Kissinger, differential privileged knowledge distorts people’s perceptions of risk and whose judgment to trust:

"You will deal with a person who doesn't have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you'll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You'll give up trying to assess what he has to say. The danger is, you'll become something like a moron. You'll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours."

Ellsberg’s description is particularly apt: Only after years does the realization dawn that classified information is often inaccurate, incomplete, and just as likely to lead you astray as the New York Times.

There are also benefits - most obviously, keeping secrets reduces the spread of information and thus the risk that bad actors gain powerful technology.1 And it’s intrinsically hard to evaluate how competent classified military projects are, as their successes and their failures are hidden.

But in general it seems to hold that good decisions happen in environments where different actors are checking one another's work and holding one another accountable. That requires them knowing what each other are up to, which classification culture prevents. I think it becomes clearest if we imagine transposing classification norms to a corporate environment. If a startup told you that its engineers weren’t allowed to talk to each other about what they were building, that outside experts were banned from reviewing their work, and that anyone who raised concerns risked losing access to the codebase, you wouldn’t invest.

If AGI development is happening behind a clearance wall, we guarantee that the people building it will be systematically cut off from the broader research community and external critics best positioned to catch their mistakes.

On net I would like to pause AI development until we have stronger guarantees that powerful AIs will be aligned, but given how plausible the absorption-into-national-security path is, such efforts should also be targeted at preventing an uncontrolled singularity being kicked off in a hidden NSA black site. Examples:

a.) A pause which includes the national security state, mandated by congress. Given how hard it has been to ban even incredibly unpopular research when they touch on national security prerogatives, like gain of function experiments, the prospects aren’t great. Classification authority, executive privilege, and the fact that oversight committees are often captured2, all make Congress a very weak actor in national security contexts. Still, it’s possible that a sufficiently motivated legislature could enforce broad verification by setting up independent agencies to monitor and enforce bans, including against executive branch ones.

b.) International verification between the US and China. One upshot of the competition between the US and China is they both have incentives and leverage to check each other’s efforts. Akin to the nuclear race, having verification and redlines which, if crossed, would lead to retaliation from the other party, might push national security actors in both states to be more reticent in developing too powerful models. Verification covers many practices - from chip sensors to interviews of personnel to on-premise watchdogs - but top of the list would be mutual tracking of the available world compute to ensure mutual confidence that there are not hidden R&D efforts.

c.) Creating a broad political coalition and movement that is so leery of powerful AIs that it would be political suicide for any group to go for it. If AGI development is perceived extremely negatively and viewed as dangerous, it’s possible even the General Rippers of the world would hesitate given fear of public backlash. Since even the executive branch can’t always control its own apparatus3, a sufficiently motivated president could do something like go on TV and publicly declaring it illegal and unpatriotic to do a black site.

None of these seem that promising and have drawbacks, but perhaps some combination of the three together would work? And of course in general there aren’t that many promising options, so one must make do with what they’ve got.

The founders in their infinite wisdom structured a power balance between the different branches of government and civil society; this continues to be wise even when facing AGI, with a broad number of groups checking each other and vetoing reckless acceleration. Pause efforts that disempower civil society by hiding what’s happening in the government seem unlikely to lead to good outcomes.

Related Links:

• An International Agreement to Prevent the Premature Creation of Artificial Superintelligence. The most developed proposal I’m aware of for an international treaty and verification regime for AGI.

• Who Will Govern Artificial Intelligence? Learning from the History of Strategic Politics in Emerging Technologies. Jade Leung’s thesis modeling how strategic technologies get absorbed by the state, using case studies of aerospace, biotech, and cryptography.

• Verifying International Agreements on AI: Six Layers of Verification for Rules on Large-Scale AI Development and Deployment. An overview of different methods - hardware and social - for tracking compute and auditing AI development

Thanks to Peter Barnett, Lisa Thiergart, and Ben Weinstein-Raun for comments on earlier drafts.

Appendix: Full Daniel Ellsberg Quote

“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret.

“I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of what the effects of receiving these clearances are on a person who didn’t previously know they even existed. And the effects of reading the information that they will make available to you.

“First, you’ll be exhilarated by some of this new information, and by having it all — so much! incredible! — suddenly available to you. But second, almost as fast, you will feel like a fool for having studied, written, talked about these subjects, criticized and analyzed decisions made by presidents for years without having known of the existence of all this information, which presidents and others had and you didn’t, and which must have influenced their decisions in ways you couldn’t even guess. In particular, you’ll feel foolish for having literally rubbed shoulders for over a decade with some officials and consultants who did have access to all this information you didn’t know about and didn’t know they had, and you’ll be stunned that they kept that secret from you so well.

“You will feel like a fool, and that will last for about two weeks. Then, after you’ve started reading all this daily intelligence input and become used to using what amounts to whole libraries of hidden information, which is much more closely held than mere top secret data, you will forget there ever was a time when you didn’t have it, and you’ll be aware only of the fact that you have it now and most others don’t….and that all those other people are fools.

“Over a longer period of time — not too long, but a matter of two or three years — you’ll eventually become aware of the limitations of this information. There is a great deal that it doesn’t tell you, it’s often inaccurate, and it can lead you astray just as much as the New York Times can. But that takes a while to learn.

“In the meantime it will have become very hard for you to learn from anybody who doesn’t have these clearances. Because you’ll be thinking as you listen to them: ‘What would this man be telling me if he knew what I know? Would he be giving me the same advice, or would it totally change his predictions and recommendations?’ And that mental exercise is so torturous that after a while you give it up and just stop listening. I’ve seen this with my superiors, my colleagues….and with myself.

“You will deal with a person who doesn’t have those clearances only from the point of view of what you want him to believe and what impression you want him to go away with, since you’ll have to lie carefully to him about what you know. In effect, you will have to manipulate him. You’ll give up trying to assess what he has to say. The danger is, you’ll become something like a moron. You’ll become incapable of learning from most people in the world, no matter how much experience they may have in their particular areas that may be much greater than yours.”

….Kissinger hadn’t interrupted this long warning. As I’ve said, he could be a good listener, and he listened soberly. He seemed to understand that it was heartfelt, and he didn’t take it as patronizing, as I’d feared. But I knew it was too soon for him to appreciate fully what I was saying. He didn’t have the clearances yet.

Appendix: Excerpt from Jade Leung’s Thesis

The life cycle tends to begin with researchers – specifically, with them breaking new ground with a series of fundamental insights which lay the foundations for the technology to emerge. Often, the state is alongside as enabling partners, demonstrating an investment appetite for early-stage risk which escapes firms. However, as theoretical breakthroughs spur applied research, and as fundamental insights generate business opportunities, firms are quick to enter the fray, wielding innovation capacity particularly suited to commercialization and proliferation. The entrance of private firms causes the technology industry to scale rapidly. The state retreats into the position of being a customer of these technologies, no longer the primary funders nor controllers of the technology.

The actors thus settle into an uneasy equilibrium. Firms are in control over much of the R&D pipeline – as funders of research, employers of researchers, and owners of the infrastructure to bring products and applications to international markets. The state supports the growth of the technology industry, often with limited routes to influence its development and deployment. However, the equilibrium proves fragile to shifts in the external environment. Such shifts can take the form of escalating strategic competition between states, for example, or an increasingly engaged public concerned about the potential harms caused by these technologies. These shifts create the impetus for the state to seek to regain influence over the technology; they thus begin to draw on their legislative authority to do so. They specifically pursue routes to control the movements of technology products and knowledge across national borders, and to increase their access to commercial technologies. In turn, firms find themselves in a precarious position of balancing competing interests and stakeholders. On the one hand, they are being pulled to sell their technologies to the state in service of national defense and security goals. On the other hand, the public and the research community become critical of the choices that firms make with respect to serving such goals. Fading into the backdrop are researchers – constrained by their dependence on R&D funding and by their legislative environment, their influence as an actor wanes.