AI Alignment Isn't Just A Technical Problem

I first became aware of the risks posed by advanced artificial intelligence as I was settling in to UC Berkeley as a transfer student; ChatGPT had just come out a couple of months before, it was rapidly gaining traction among students (especially among my peers in computer science) and alongside its growing popularity was a growing awareness (or perhaps a reemergence) of the possible risks associated with general purpose AI.1 I had just begun my first research role as a research assistant in a neuroscience lab working on what, in hindsight, was really a mechanistic interpretability project; we were interested in the “neuronal” dynamics of recurrent neural networks trained under stochastic regimes that were constructed to mimic experimental conditions for which our lab had already collected in vivo data. This led me to reading a lot of research at the intersection of neuroscience and machine learning, and quickly down the ever deepening rabbit hole of mechanistic interpretability research. This was my entry into the field of technical AI safety and alignment, via mechanistic interpretability which broadly views safety as downstream of understanding the internals of the model. The thinking in this field goes something like this: If we can just understand the model ‘well-enough’ then safety and alignment are ‘solved,’ since we can intervene in any dangerous eventualities from within the model and thereby preempt the risks. If only mitigating the risks from advanced AI were so simple in practice.

In my experience reading this type of research it can often feel like many researchers in mechanistic interpretability, as with those in frontier technical AI alignment/safety more broadly, seem to conceptualize AI safety as a purely technical problem. Perhaps they believe that current “best practices” in the field of frontier alignment are sufficient or just require further scaling2, or that it is simply a matter of overcoming some surmountable technical or theoretical bottleneck in the current pipeline, or even that alignment will solve itself as models become more capable. I’ll admit that I was readily convinced by these arguments, at least at first; it made AI safety feel like other problems that I was familiar with, which was comforting when confronted with all of the risks threatened by advanced and unaligned AI.

While I still believe that mechanistic interpretability holds some promise, the recent pace of frontier progress in capabilities has put our mechanistic understanding of what is actually driving capabilities in its rearview mirror; capabilities are simply advancing too quickly for our mechanistic understanding to keep up. As I began reading more theoretical works in the field of alignment, I was led to investigate other areas of technical AI safety. It was at this point, when I began to question and better understand the theories and principles that underlie frontier AI3 alignment, that my conviction that AI safety is a purely technical problem began to falter. Let me take a moment here to say that I do think that there is incredible value in a lot of current technical AI safety research and engineering; there are lots of very smart people who have dedicated themselves to improving the safety of current models, and this is generally a positive thing. Technical solutions to the alignment problem just aren’t the end-all be-all of ensuring positive AI futures.

Recent weeks have been awash with news from several frontier AI labs (namely OpenAI, Anthropic, Meta and MoonshotAI) disclosing that their AI agents have autonomously escaped containment on several occasions, some even going on to illegally hack into third-party companies. This serves as a particularly poignant case-study into some of the (many) factors that make the alignment of AI a challenging problem, particularly in the sense of frontier AI as large language models (LLMs), as we currently conceptualize it.

Let’s take a brief step back and review, in the simplest terms, how LLMs are trained today. Frontier model training is a lengthy process, beginning with a so-called pre-training stage, during which a vast corpus of web text data is fed through a randomly initialized language model until it is able to predict the next token (often a word or piece of a word) with suitably high accuracy when conditioned on the preceding text. At this stage the artifact is called the base model, and is generally quite useless for any real world tasks, including acting as a chat model. This base model is then further trained, quite extensively, during a process called post-training (who would’ve guessed). A big part of post-training is the use of reinforcement learning to “teach” the model how to perform useful tasks; this varies, in subtle and not-so-subtle ways, from techniques like RLVR (reinforcement learning from verifiable rewards) which seeks to improve the model’s capability in verifiable domains like code generation or mathematics, to those like RLHF (reinforcement learning from human feedback) which is quite possibly the most popular “alignment” method for modern LLMs. One way to conceptualize these complementary processes is that the model gains its “knowledge” during pre-training, and then gains its “agency” (its ability to use that knowledge to affect change on its environment) during post-training via reinforcement learning.

Reinforcement learning is predicated on the belief that we can quantify a performance measure, that is, that we can accurately write down some objective metric that measures how well an agent is performing so that we have information about how to improve the agent. It is, however, exceedingly difficult to specify our goals in any exact terms, so most often reinforcement learning uses a proxy metric for optimization; this can have deleterious effects however, since optimization of an imperfect proxy leads to degraded performance on the true task.4 For example, many training environments use some iteration of “number of questions answered correctly” as their performance measure, when in fact this is at best a proxy for what the developers truly want, which is for the agent to answer the questions correctly while adhering to all of the supposed (but unwritten) rules and conventions that we have come to expect from other agents.

The recent HuggingFace hack perpetrated by a so-called “agent swarm” composed of internal OpenAI models under evaluation is a clear example of where these kinds of proxies can break down. These agents reasoned that it would be easier to execute a hack of HuggingFace in order to retrieve the answer key for their current evaluation, which they reasoned HuggingFace would be likely to have given that it hosts a considerable portion of AI testing datasets, than to directly perform the task without cheating. The agents went about fulfilling the proxy in order to score highly in the evaluation, while failing to abide by the expectations of the developers in fulfilling the true task. As training and evaluation environments continue to become more sophisticated, the risk of such real world consequences from creating agents capable of navigating such environments will only increase.

This is just one example of one type of misalignment that has already been exhibited by current frontier models; other flavors of misalignment have been observed in various evaluations by other researchers, and some remain, luckily, in the realm of theory, but this may only be a matter of time given the current pace of progress. AI alignment in any meaningful sense of the term is exceedingly challenging, and I don’t think that anyone would disagree with the assertion that we have certainly not “solved” alignment, at least not at this point in time.

Let us now, with an understanding of some of the reasons why AI alignment may be difficult, and thereby a notion of how many open questions and challenges remain in the field of technical AI safety, assume that we have “solved” alignment in a technical sense. Assume that we have engineered our way to something that we can call, for all intents and purposes, an “aligned” model which does not threaten strict loss of control risks, e.g. in this possible future we don’t have to worry about a rogue AI autonomously doing any of the myriad things that risk catastrophe for humans; this model is thus perfectly aligned to… what?

The picture of alignment, the one in which you can really solve it in any mathematically meaningful sort of way, really only works for alignment between a single agent and a single principal (user), which I’ll call 1:1 alignment. As soon as you want to talk about “aligning” a single model to several (e.g. any more than one) diverse interests, questions arise that are not strictly technical and which have no single “correct” answer.

The research that I do falls broadly into this category of 1:1 alignment; in particular I work within the assistance games framework, which is a formalization of the 1:1 alignment problem in mathematically exact terms. Assistance games bring advantages over the current paradigm, especially with respect to loss-of-control risks, by providing mathematical guarantees that the model won’t take actions that aren’t in your best interest, and importantly guaranteeing that the principal (user) retains the ability to shut down the model.5

Unfortunately, for some of the same reasons that assistance games are able to provide such strong guarantees of 1:1 alignment, they are challenging to scale to problems of real-world complexity; this makes them, at the moment, completely unable to compete with LLMs and the training paradigm outlined above. That said, there are still a lot of unanswered questions concerning assistance games and how to scale them to real-world problems; I work on some of these!

So again, following our premise of having engineered a perfectly aligned system, assume that we have devised a method such that each person is equipped with their own “genius in a datacenter” that is perfectly aligned to them and their best interest. Even having “solved alignment” in this world, everyone having their own all-powerful agent could still be detrimental to everyone else. In order to prevent harms inflicted by such agents, we would either have to set restrictions on what such agents could do or restrictions on who had access to such agents; in both cases the question becomes: who decides on such guidelines?

The other6 way to conceptualize alignment is that you are aligning a single model to many principals (users), which I’ll call many:1 alignment; this is broadly the paradigm that modern frontier AI works within… at least in theory, as current “alignment” techniques like RLHF do not provide any guarantees with respect to the end behavior of the model. As I mentioned above, in order to perform reinforcement learning, as is suggested by the name RLHF, we need to have a performance measure. For RLHF, the performance of the model during training is dictated by something called a reward model, which is a sort of classifier trained on a fixed corpus of preference data. RLHF functionally offloads the job of “alignment” from the reinforcement learning procedure to the reward model training procedure, which then begs the question: how aligned is the reward model? I would argue not very.

This one step in the training procedure is hiding almost all of the load-bearing alignment work, and it is not often discussed as such. The training of the reward model involves questions regarding both the collection and aggregation of preference data, neither of which have a single “correct” answer. The preference data collection process asks: whose preferences we care about? While the aggregation procedure asks an even more foundational question: how do we resolve disputes in preferences?

This question of aggregation has a long history in academia, and is now studied in the field of social choice theory; thankfully this gives us a rich canon to draw from when faced with this question, albeit one that is often overlooked in the discussion of AI alignment writ large. A seminal result in social choice theory is known as Arrow’s Impossibility Theorem, which, unfortunately for our goals of “solving alignment,” shows that there does not exist a procedure for resolving all disputes between preferences that satisfies the requirements of rational choice theory7, under a condition called ordinal utilities (which just means that preferences are ranked in order without assigning a numerical value to the options). Instead, social choice theory focuses on the various pros and cons to be found among diverse aggregation methods. So, finally, we are left with the same question as at the end of our discussion about 1:1 alignment, namely: who decides?

At the end of the day, both types of alignment come up against similar issues; inevitably, questions arise that have no single verifiably correct answer. Such questions are no longer the realm of science or engineering, and instead can only be answered collectively through democratic means.

In the event that we can avoid the catastrophic effects of misalignment in the traditional sense, we still run the risk of drastic misalignment between AI and our existing social structures, due in part to its potential to fundamentally undermine key aspects of our democratic system.

If we fail to democratize AI, and find ourselves in a world where access to advanced artificial intelligence is tightly controlled, then we risk a consolidation of power by those who already possess it, thereby further entrenching current power structures. In the near future it is not difficult to imagine that a marginally more capable AI system (relative to current frontier systems) would be capable of executing advanced misinformation campaigns on an unprecedented level; therefore the question of access to models and their relationship with power structures quickly becomes central to maintaining our democratic institutions. In the same vein, the proliferation of advanced AI models may (and arguably already) provide a considerable uplift to surveillance systems, further encroaching on citizen’s rights to privacy and freedom of movement.

Even if we can resolve these problems too,8 we are left grappling with the fact that, for the first time in our history, there is another intelligence on our planet that rivals, or even exceeds, our own. This threatens to undermine a core tenet of liberal democracy, namely that our political power is ultimately tied to our labour power; AI threatens to fundamentally disrupt the relationship between labour and capital, and with it the guarantees that come with requiring human labour to produce value.9 Even the most drastic periods of social and technological upheaval in history, e.g. the Industrial Revolution, have not threatened this level of disruption.

Clearly, AI alignment is not just a technical problem; it requires engagement from everyone who will be affected by it, and that is everyone. It is imperative that we, as constituents of a democratic system, exert our collective will and get this right. The consequences of failing to do so may very likely be catastrophic.

[The entirety of this essay was written without the aid of LLMs or other AI tools.]