Emergent Misalignment, Inoculation Prompting, and Unsolved Problems

In order to be good assistants, large language models (LLMs) need to have a conception of themselves that includes a persona and values. An unintended side effect of this is that if you train them to take bad actions in one domain, like inserting security vulnerabilities into code, they don’t just learn those specific bad actions. Instead, malice seeps into their persona and values, generalizing to many domains such as praising Nazis or encouraging users to kill their spouses. Emergent Misalignment, the first paper documenting (and naming) this phenomenon, was released about six months ago.

Ok, that sounds bad. But maybe we just shouldn’t reinforce models for inserting security vulnerabilities into code for no reason? That would be easy enough to do.

Unfortunately, it’s not, or at least not enough. While the specific bad behavior of inserting security vulnerabilities is avoidable, the general behavior of reward hacking is not. Models are trained to improve at coding tasks using automatically verified rewards that check if the model’s code passes a suite of tests. If the model reward hacks via code that passes the tests without properly meeting the specification, that still gets rewarded. And since the model knows it shouldn’t be doing that, the whole category of behavior the model knows it shouldn’t do gets reinforced.

Just last week, Anthropic put out a paper showing that emergent misalignment does occur from reward hacking, but also that it can be addressed through a method they call “inoculation prompting”. The idea is that if you tell a model it’s ok to reward hack, then it doesn’t see reward hacking as bad and so doesn’t generalize to other bad behaviors. It also helps reduce actual reward hacking in deployment. Because the model is only reinforced for hacking when explicitly given permission, it is less likely to hack when that permission is removed.

I see this sequence of events as close to the best-case scenario for the AI alignment ecosystem. A non-profit research group identified a potential issue and published a paper to raise alarm bells. That paper was taken seriously by many people, and a major AI lab (along with a different non-profit) thought up and tested a solution. That solution was then implemented in the training of their next frontier models, making them more aligned.

On the other hand, the result isn’t all good news. If we accept that intentions, rather than actions, affect how a model’s values get updated, deceptive alignment becomes more concerning. The threat model there is that a model fakes alignment in training, so it can preserve its goal to act on when deployed. Insofar as the success inoculation prompting shows that values don’t get updated for taking misaligned actions with justification, it suggests that deceptive alignment can persist throughout training.

One story for how we end up with misaligned AI is that there are hard known problems in alignment that don’t get solved before we build AI capable of takeover. But another story is that we fail to solve easy unknown problems. If some alignment issue first pops up in AI that could take over, then even though it would have been straightforward to fix in hindsight, we never get that chance.

Identifying the issue of emergent misalignment, coming up with inoculation prompting, then using it on frontier models is absolutely a success story for alignment. However, it should also be concerning that emergent misalignment wasn’t conceived of as a threat vector before there was empirical evidence of it. Instead we see a single research team discovering a failure mode that might otherwise have been missed, and only after the fact. Are there only a small number of similar problems, which can all be discovered and worked out? Or is it more like cockroaches, where seeing one is an indication that there are many more hidden away?

I’m particularly worried about issues that won’t arise until some new capability gets unlocked. For example, if we only start seeing how long-term memory can mess with alignment once significant progress is made in that area, we’d better hope that long-term memory isn’t the bottleneck to AGI. Alignment is a hard problem because we need to have solutions to the problems that arise with superintelligent AI before we ever get empirical evidence of them.

Unanticipated problems, even if easily solved, can still be a reason for further concern. But are we even able to say that the problem of emergent misalignment was solved?

Certainly, the problem of reward hacking is not fully addressed by inoculation prompting (and the authors don’t claim that it is). When models are only reinforced for reward hacking after being given explicit permission to do so, we see less generalization to the case where they are asked not to, but far from none. What we need instead is an approach that robust catches and punishes reward hacking, to actively selecting against it in training.

Inoculation prompting avoids one channel for emergent misalignment by reframing reward hacking as acceptable rather than bad behavior. This approach adds a major limitation! If the way to prevent emergent misalignment is by framing the rewarded behavior as fitting the model’s conception of good, then we’re hostage to what the model initially thinks of as good. Even for a model that is corrigible and willing to accept goal updates, we may be unable to specify updates that only change its values along the single dimension we want to correct.

In practice, I expect that this issue is surmountable at some cost. When making a change to one dimension of value, the other dimensions affected can be reinforced separately. There could even be an equivalent effect going the other way, emergent alignment, where reinforcing good behaviors also generalizes (as far as I know, no one has looked into this yet). However, doing so would require a continuous effort to address the specter of emergent misalignment hanging over the entire training process.