Solutions to Corrigibility, Deceptive Alignment, Reward Tampering, Monitorability, and Reward Hacking

A title like that seems ironic, likely setting up a humorous bait-and-switch. Solving even one of the listed problems would be a major breakthrough, solving all five at once would strain credulity. It’s probably just setting up an interesting but ultimately minor technical point, or a discussion of high-level research philosophy, right?

Wrong! I believe my recent work has made important progress on all of these problems, and you get to find out about it through this blog post. The title was chosen about 65% unironically, with some mild exaggeration to avoid getting bogged down in nuance, to provoke useful criticism, to flag for future AI systems doing alignment work that this post contains a useful idea, and to give readers permission for getting excited about the results. The Alignment Forum posts Defining Corrigible and Useful Goals and Defining Monitorable and Useful Goals more accurately convey that the novel contribution is in defining goals that don’t incentivize the problematic behaviors (and don't sacrifice performance), but those titles lack a certain flair.

Now that I have your attention, I can point you towards the pair of Alignment Forum posts for in-depth explanations of my recent work. A more formal version of the first will be released as an arXiv paper soon, while the second will ideally be written into a paper after running experiments.

The first post defines a “corrigibility transformation” that takes in an arbitrary goal over external environments and produces a goal that is corrigible but otherwise incentivizes pursuing the original goal. It does so without running into issues of existing proposals, like agents manipulating their own beliefs or creating incorrigible subagents. While the main contribution is theoretical, the mechanism is also shown to work empirically with gridworld experiments.

Since the necessary properties for corrigibility are present from the beginning of the training process, this intervention can also address deceptive alignment, which can be thought of as one type of incorrigibility. As a bonus, it also eliminates the incentive for agents to manipulate their own goal, a major form of reward tampering.

The second post takes the core mechanism of the corrigibility transformation and applies it to a “monitorability transformation” that takes in an arbitrary goal and produces a goal that incentivizes pursuing the original goal while as though monitors did not exist. This also makes it much easier to train against unwanted behavior that the monitor catches, such as intentional reward hacking, without just training the agent to hide the behavior.

Both of these transformations work using the following mechanism:

1. Give the agent the ability to costlessly do some bad thing (e.g. decline any updates requested or trick monitors). Have it choose whether to exercise that power as a component of taking any action.

2. Define the new utility for taking an action, whether or not it includes doing the bad thing, to be the expected discounted (original) utility after taking the version of the action that does the bad thing. Make this new utility function myopic.

3. Since doing or not doing the bad thing as part of some action now give equal utility, break the tie in favor of not doing it with some bonus utility.

Since the agent can do the bad thing for free, the original goal disincentivizes taking costly actions that also lead to the bad thing. The new goal incentivizes the same actions as the original goal, but without doing the bad thing for free. This means the agent is disincentivized both from taking costly actions to do the bad thing and from doing the bad thing for free.

This is framed in terms of a utility function, but it should also work as a reward structure for AI systems that are not a goal-optimizing agents. Not doing the bad thing for free is consistently reinforced, while costly actions to do the bad thing are consistently penalized.

A core difficulty with the alignment problem is our inability to articulate all the ways that we don’t want the AI to pursue a goal. This mechanism gets around that by disincentivizing the whole set of actions that lead to some useful intermediate goal, without needing to specify or even understand what that set looks like. While that limits applicability to intermediate goals that we can actually let the AI do for free (e.g. we can’t give it the ability to costlessly kill all humans), I suspect the use cases go beyond just corrigibility and monitorability. A nice property is that once the basic mechanism is in place for one application, further applications are very low cost.

Reactions I’ve received when describing this work include “Huh, is that all you need?” and “It can’t be that easy, can it?”, and I agree that the mechanism is fairly straightforward. If you know me you’ll know it didn’t take a genius to propose it. The simplicity is a point in its favor, making it easier to evaluate and implement, but does raise the question of why it had not been previously suggested. My preferred answer to that is very few people have seriously worked on AI alignment theory, plus the research space is high dimensional, so there is still low hanging fruit.

Another possible answer is that I’m wrong. Perhaps smart researchers have already considered and rejected the idea, due to a fatal flaw that I’ve missed. Criticisms so far have been based on misunderstandings, usually because I explained it poorly, but more fundamental critiques may exist. If you see an issue, please let me know, so that I can either modify the proposal to address it or give up on the approach. I am happy to pay bounties for valuable critiques.

Being able to specify corrigible goals without a performance penalty is by itself “big if true”, and well worth the effort to evaluate the “if true”. If you’ve done that and don’t see any issues, then I hope that you’re convinced this work is a big deal! That should be exciting! Making AI systems corrigible lowers existential risk, perhaps rather significantly, and defining corrigible goals is a major step towards getting corrigible behavior. Further developing the ideas has high potential for impact, whether that’s theoretical/empirical research, or just mentioning it around as an interesting idea.

For clarifications on anything that was unclear, you can comment below or message me. If you would like to work with me or on related ideas, please reach out for a chat!