Catching Misalignment vs. Fixing It

Redwood Research, the organization that popularized the AI Control agenda, consistently have some of the most thoughtful and nuanced takes on AI safety1. That evenhandedness includes turning a critical eye towards their own work, where they’ve warned of the possibility of successfully catching a misaligned AI only for nothing to be done in response.

In 2024, Redwood’s CEO, wrote up his concerns that even if a model is caught acting egregiously misaligned, and even if it would have done significant damage if not caught, there might still not be any serious action taken to prevent deploying similar models. It’s easy for AI labs to say the issue was idiosyncratic rather than fundamental, and that the costs of not deploying the most advanced models are too high. I share that concern but want to point out the even more pessimistic assumption built in, that obviously there’s no way for us to reliably fix misalignment when we detect it. Instead, the best we could hope for is to stop deploying misaligned models.

This is not an issue just with the AI Control approach, but rather most areas of work aimed at reducing catastrophic risk from rogue AI. As a very loose timeline, mechanistic interpretability was the fastest growing area of AI safety in 2022, evaluations were in 2023, and AI control was in 2024. Now, at the start of 20262, those overlapping domains represent a majority of the field. You can see it, for example, in the distribution of mentors for a program like MATS, which has been many people’s inroad to working on AI safety.

Putting aside the question of whether such methods will actually catch misalignment, we are still left with the question of what to do if we succeed. Thankfully, most parties recognize that training models against the tools used to catch them has a high risk of just hiding misalignment. In theory, interpretability could get advanced enough that we could reliably train against it, but that milestone is very far off.

One straightforward solution for what to do after catching misalignment is to pause. Don’t deploy misaligned AI, don’t train even more advanced AI, and prepare for when we can no longer keep a lid on it by investing in the development of robust methods for alignment. Of course, if it would be useful to do then, it would be better to get started now.

The current default solution is to use more of existing tools. As Jan Leike described in a recent post, “to address agentic misalignment we made some SL [supervised learning] data, some RL prompts, and synthetic reward modeling data”. I’m skeptical that this will continue to work as well for models that are more coherent agents, with longer time horizons, but without a pause or better options then it’s worth a shot.

As an aside, I think the lack of work on better methods for aligning AI can be ascribed to the shift from theory to applications. There is little room to improve empirically on the approach Jan described, because it works quite well on most3 instances of misalignment as it exists today. That is, misalignment is obvious and models do not resist corrections. Until that stops being true, only theoretical progress is possible.

I point out the limitations to the default path to argue that alternatives should be supported. That said, I also believe there is a decent chance that agendas aimed at catching misalignment could be sufficient.

In short, catching misalignment is enough to win in worlds where AI alignment is easy.

What do I mean by easy? I’m talking about the size of the basin of attraction around alignment, where a model that’s aligned “enough” acts to become more aligned. Consider an AI and a human that share all the same preferences, except the AI slightly prefers sunsets to be coral pink and the human slightly prefers them to be watermelon pink. This minor difference is dominated by their shared preference for the AI to update, so the AI is happy to switch even though it gives up something.

That framing is somewhat complicated by the fact that nobody has a very good idea of how aligned is “enough”. You could think of the basin size as a continuous quantity, but I find it more helpful to break it down into paradigms of alignment difficulty.

In the worlds where alignment is very easy, current posttraining techniques work robustly, and consistently reach the basin of alignment. Even if we make some mistakes, those are overwhelmed by the level of optimization applied in training. The ability to catch misalignment might give us some peace of mind, but doesn’t change the outcome.

I think it’s plausible that we’re in such a world. With some caveats, existing methods genuinely work pretty well on current models, and there’s a real probability that that’s enough. In that case, by the time models are coherent and capable enough to resist further alignment, they won’t want to. Perhaps the values that we want models to internalize are just not that complicated, and immensely overdetermined by the feedback we give.

In worlds where alignment is a bit harder, but still relatively easy, current methods give us both aligned and misaligned AIs. Possible forms this could take include:

1. Randomness in the training process gives reasonable chances of both alignment and misalignment

2. A model contains multiple personas that are active at different times, some are aligned and some are not

3. Models pass through a misaligned phase, but after enough training end up aligned.

These are the worlds where the ability to catch misalignment shines brightest. When we detect misalignment we either cull the model, reset to an earlier checkpoint, or throw more alignment training at them. With sufficiently good detection techniques, the probability of an aligned model ends up much higher than the probability of a misaligned model that can pass our tests.

Unfortunately, I think such worlds are very unlikely. Given diminishing returns to resources spent on alignment, the probability is low that we end up just on the borderline so that either randomness or extra resources tip the balance into the basin of alignment. An ensemble of models or personas with explicit training pressure for diversity could increase the chance of one being aligned, but for now we don’t have a good method of avoiding strong correlations between ensemble members.

At the difficult end of what I would consider the easy worlds, models would be aligned (at least with decent probability), but get turned misaligned if we make mistakes. By mistakes, I mean incorrect decisions that can be fairly easily corrected, like “overly training for sycophancy”, not deeply ingrained part of the training process like “using RL”. Techniques like inoculation prompting are targeted at this region, fixing the mistake where unintentionally rewarding models for reward hacking leads to emergent misalignment.

Catching misalignment is very useful in these worlds as well, since it allows for mistakes to be more easily diagnosed and corrected. The feasibility of that process depends on the number and subtlety of the mistakes being made. Additionally, with models that are “almost”4 aligned, the threat of catching misalignment can shift a model’s best option to doing useful alignment work or strike a deal with humans.

The phenomenon of emergent misalignment being more or less randomly stumbled upon makes me fear a cockroach analogy: if you see one, there are many more you haven’t seen yet. I’m also worried that if we initially caught emergent misalignment with evaluations, monitoring, or interpretability, it wouldn’t have revealed the underlying cause. On the other hand, it’s unclear whether missing emergent misalignment would have prevented entering the basin of alignment. The number and severity of uncaught mistakes is a big unknown.

The worlds where alignment is hard share the characteristic that known posttraining methods are insufficient. Models neither start aligned, nor do they become aligned in the training process. Success in these worlds will require new techniques for defining the alignment target and/or for imparting it.

For determining where marginal alignment research should be directed, you can ignore the worlds where success or failure is highly overdetermined. Of the remaining worlds, where alignment research can make an impact, I would loosely expect catching misalignment to be sufficient about half the time. Further research can increase the chance that we actually do catch misalignment in worlds where that’s enough.

I would claim that research focused on catching misalignment is one of the most important problems that can be worked on, and it has a real chance of saving humanity. However, any world that could be saved by catching misalignment could also be saved by aligning the model, and the reverse does not hold. Better methods for actually aligning models remain sorely neglected, and waiting until we catch misalignment that can’t be fixed before invesing in them will be far too late.