Linguistic Assistance Games (Pt. I)

Abstract

Cooperative inverse reinforcement learning (CIRL), or assistance games, provides formal guarantees of alignment and corrigibility that are notably absent from models trained under the standard model of artificial intelligence. Extending CIRL to the natural language domain has remained challenging due to the inherent ambiguity of language, which is precisely the phenomenon that computational pragmatics has long sought to model, particularly through Bayesian inference in frameworks such as Rational Speech Acts (RSA). While contemporary approaches have integrated language into Bayesian inverse planning over finite symbolic goal spaces (e.g. CLIPS, LaBToM, and LIRAS) or engineered multi-turn extensions for pragmatic reasoning in sequential dialogue (e.g. CRSA), these frameworks lack formal safety guarantees for unconstrained linguistic communication.

To address this, I introduce Linguistic CIRL (L-CIRL), an assistance game where the action space and the preference parameter space are the set of natural language utterances, equipped with an MDL prior over the preference space to maintain tractability. The human acts as a boundedly rational pragmatic speaker (S1, in the RSA tradition), optimizing a surrogate Q-function under the assumption that the robot is a literal listener. The robot then acts as the pragmatic listener (L1), maintaining dual belief states over the human’s preferences, updated via sequential Bayesian inference. Within this framework, I seek to offer two key theoretical results. First, to prove that the human’s Boltzmann policy over the surrogate Q-function is functionally isomorphic to the RSA pragmatic speaker utility under purely communicative utterances, e.g. utterances that do not affect the world state and only serve to inform the robot about the human’s preferences; thereby demonstrating that this established cognitive model (RSA) emerges from bounded rationality in the assistance game and providing a formal unification of these two frameworks. Second, to show that L-CIRL inherits CIRL’s alignment properties of shared payoff, corrigibility, epistemic risk aversion, and value of information through belief consistency, which guarantees that the pragmatic belief update cannot eliminate safety-relevant hypotheses maintained by the literal belief. This yields the first framework to provide CIRL-derived structural safety guarantees for preference learning over unconstrained natural language.

Table of Contents:

1. The Standard Model (and What Can Go Wrong)

2. Common Ground

3. Some Additional Background, Before We Dive In

   1. Rational Speech Acts (RSA)

   2. Cooperative Inverse Reinforcement Learning (CIRL)

   3. On How Assistance Games Work Towards Resolving the Dangers of Instrumental Convergence

4. Contemporary Work

   1. Cooperative Inverse Planning and Language

   2. Pragmatic Reward Learning in Assistance Games

   3. Multi-turn and Cooperative Extensions of RSA

5. Linguistic Assistance Games

   1. Pragmatics as Bounded Rationality

6. Theorems and Proof Sketches

   1. Updated Theorem Framing

   2. Sketches

7. Conclusion

1. The Standard Model (and What Can Go Wrong)

“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively… we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”

In a nod to the authors of Cooperative Inverse Reinforcement Learning, I also opt to include the above quote from Norbert Wiener (1960), which really does get at the heart of the external alignment problem, a major part of the value alignment problem. This point is again illustrated by Nick Bostrom over 40 years later, using a thought experiment called the paperclip maximizer; in which an advanced AI that is tasked solely with maximizing the production of paperclips, provided sufficient power over its environment (e.g. ability to realize agency), would inevitably attempt to convert all matter in the universe into paperclips or machines that manufacture additional paperclips. This is an, albeit dramatic, example of an AI operating within the standard model, “whereby systems are designed to optimize a fixed, known objective” as per Stuart Russell (2022), which he argues will become “progressively untenable in real-world applications because of the difficulty of specifying objectives completely and correctly.” The instrumental convergence conjecture is a characterization of the emergent process in which every sufficiently advanced agent will tend towards pursuit of a similar set of sub-goals, namely self-preservation, resource- and power-acquisition,1 thereby exacerbating the external alignment problem. This notion is the undercurrent which pervades the quotes above - it is a substantial component of what makes sufficiently advanced agents operating within the standard model potentially dangerous, and which may lead to catastrophic consequences.

Despite all of these, and many other increasingly urgent2 warnings, the industry rushes headlong towards increasingly capable agents trained within the standard model. This is especially so as the frontier of development increasingly transitions to scaling and improving reinforcement learning, which explicitly positions the language agent within an environment and tasks it with optimizing a fixed, known objective. The astute reader may point to Reinforcement Learning from Human Feedback (RLHF) which learns a reward function from human preference data, and surely guarantees that it is aligned to human values… right?

Not really. Within the context of the standard model, RLHF can be thought of as simply kicking the metaphorical can down the road, in the sense that once the reward model is trained, it is used, in its fixed and known form, in order to “align” frontier LLMs. On top of this, there is a growing body of literature that critiques the mathematical properties of RLHF alignment, especially when AI alignment is framed as a social choice problem. This is another interest of mine, and so I won’t digress too far, but the gist of this burgeoning field is that preference aggregation under the Bradley-Terry model can be viewed as a social choice problem, which in turn allows for the invocation of fundamental theorems and axioms, of which RLHF satisfies few, from the social choice tradition. Basically, it is (more or less) mathematically impossible to align a single (language) model to the inherently heterogeneous preferences of a diverse group of humans.

So does this mean that we are doomed? Not necessarily. There are alternatives to the standard model, and for our purposes we will be primarily interested in the Assistance Games framework, but they often face difficulties in defining a prior over human preferences in potentially unbounded natural language domains, as well as in scaling to the complex state and action spaces necessary to compete with modern frontier methods; and furthermore, are not nearly as widely adopted or researched as they should be. Despite this, I believe that such frameworks offer a valuable foundation for provably safe and aligned AI systems, and give me some hope for a future with truly beneficial AI.

I hope to adapt such a framework to the natural language setting in a way that is compatible with modern NLP and deep learning primitives, offering a path towards realizing aligned-by-design language agents. But before we get into what that might look like, let’s establish some common ground.

2. Common Ground

If you’re already familiar with the Markov Decision Process (MDP) formalism and how it provides a mathematical framework for understanding sequential decision problems like reinforcement learning, feel free to skip to the next section.

I’ll assume that the reader is somewhat familiar with the MDP formalism, and the key components of reinforcement learning, so I’ll keep this brief and primarily focus on the relationship between the reward function R, the state-value function V, and the action-value function Q, as this specific discussion will aid in the understanding of later portions of this essay.

In a Markov Decision Process (MDP)3, the agent is trying to select a policy in order to maximize the cumulative expected reward, formally defined as follows:

where S is the state space, A is the action space, T is the transition model which specifies a conditional probability distribution over the next world state, R is the reward function which specifies the scalar reward for taking action a in state s, and gamma is the discount factor.

In the (more generally applicable) setting of an infinite horizon game, e.g. in the setting where you don’t know when the game will end, we are interested in the policy4 that maximizes the expected infinite horizon value, also known as the Bellman Equation:

An important result in MDP theory, under the assumptions that the discount factor is strictly less than one and the rewards are bounded, is that in the infinite-horizon case, there exists a stationary optimal policy5, but that begs the question: how do we compute such a policy? One canonical method is to first compute the expected infinite-horizon value of being in state s, executing action a and executing an optimal policy thereafter (called the action-value function or the Q-function):

This defines a non-linear set of equations, one for each (s, a) pair, which is challenging to solve (as a result of the nonlinearity), however we can be certain that there does exist a unique solution! Once we know the optimal action-value function, which you can calculate or approximately solve for via your dynamic programming method of choice (e.g. value iteration), we can directly extract the optimal policy, as was our stated goal:

Finally, note that the state-value function V can, in turn, be defined in terms of the state-action function Q, by marginalizing over the action space:

3. Some Additional Background, Before We Dive In

Again, if you happen to already be familiar with both probabilistic models of pragmatic reasoning (à la rational speech acts) and assistance games, feel free to skip to the next section.

Rational Speech Acts (RSA)

The Rational Speech Act (RSA) framework is among the most, if not the most, influential probabilistic approaches to pragmatics (the study of how context contributes to meaning); it fundamentally treats “language as probabilistic, informativeness as gradient, alternatives as context-dependent, and subjective prior beliefs as a crucial facet of interpretation.”6 It offers a formal framework to understand “how speakers contextually choose between utterance alternatives, and how listeners contextually arrive at interpretations of observed utterances,” by treating language use as an instantiation of a “signaling game,” in which speakers and listeners are reasoning about a defined set of utterances U and space of possible meanings M.

The reasoning grounds out with the literal listener (L0), which is characterized by the following interpretation rule, and requires a semantic denotation for each sentence, where the delta function maps to 1 if m is in the extension of u, and 0 otherwise. This allows us to explicitly encode the Gricean Quality maxim within the Bayesian inference framework, as below:

The pragmatic speaker (S1) reasons about the literal listener, where the following rule defines the speaker’s production of an utterance u as “softmax optimizing u’s utility for communicating m, U(u, m),” where the utility of an utterance is defined as a “trade-off between the utterance’s informativeness (how likely it is that L0 will correctly infer m from u’s literal semantics alone) and its cost C(u),” (Degen, 2022) allowing us to explicitly encode the Gricean Quantity (and Relation) maxims within the framework:

Finally, the pragmatic listener (L1) is characterized by the following interpretation rule, which allows L1 to invoke Bayes’ rule to infer “the most likely meaning m by drawing on their generative model of the speaker and their prior beliefs about m” (Degen, 2022).

Cooperative Inverse Reinforcement Learning (CIRL)

Cooperative Inverse Reinforcement Learning (CIRL) games, also widely known as Assistance Games, are “two-player Markov games with identical payoffs between a human or principal, H, and a robot or agent, R.”7 This framework simultaneously offers a formalization of the value alignment problem and a possible solution to alignment (certain assumptions about the stationarity of preferences, and the rationality of humans, notwithstanding). Since R is optimizing for an unseen preference parameterization, and therefore retains some uncertainty over H’s true utility function, CIRL effectively sidesteps the difficulty of explicitly defining a truly aligned objective function, which is a major critique of the standard model as it pertains to alignment, as discussed above.

An additional advantage of the CIRL framework is that teaching (on the part of the human) and learning (on the part of the robot) behaviors arise naturally through incentives of the game, “without explicitly encoding these as objectives of the actors,” as mentioned by Hadfield-Menell et al. (2016); thereby working to avoid Goodhart’s law8.

Formally, the CIRL game tuple is defined as:

The game proceeds as follows:

1. The initial state, a tuple composed of the world state s and the preference parameterization, denoted with theta, is sampled from the joint prior P. H observes their own preference parameterization, while R does not.

2. At each timestep t:

   1. H and R observe the current world state and jointly (concurrently) select their actions from the joint action space.

   2. Both actors receive reward conditional upon the current world state and the set of joint actions, and parameterized by theta, and observe each other’s choice of action.

   3. (Not originally described in the simplified game dynamics described in the CIRL paper, but crucial for our later analysis) R updates its belief state over H’s preference parameterization, via a Bayesian inference process.9

   4. A state for the next timestep is sampled from the conditional transition distribution, and the process starts over from the beginning of step 2.

Of importance is the result offered by Hadfield-Menell et al. (2016), which states that there exists an optimal policy pair, e.g. a solution to the CIRL game, that depends only on the current state and R’s belief, which implies that R’s belief state over H’s preference parameterization is a sufficient statistic to solve the game; this guarantees that the game is solvable under the proposed game tuple and information asymmetry. We can furthermore leverage the structure imposed on the game (which is naïvely a Dec-POMDP), in which the human H is the only player with access to private information, and achieve an exponential decrease in the complexity of the problem, which allows us to apply general POMDP algorithms to compute optimal joint policies. However, the computational complexity of this reduction grows exponentially in the cardinality of the set of possible preference parameterizations, which makes exact solutions intractable beyond very small preference parameterization spaces (we’ll return to this somewhat unfortunate result in a bit!).

Now that we’ve established the basic components of the CIRL game, as well as the basic game dynamics and some key theoretical results, let’s dig a bit deeper into the Bayesian inference process involved in the updating R’s belief state over H’s preference parameterization. The reader may at this point be wondering how R chooses an optimal policy to maximize the cumulative expected reward when R does not have access to the true parameterization which defines H’s policy? In order to do so, the robot R maximizes the action-value function, defined over a state space augmented with its current belief distribution regarding H’s preference parameterization, via the canonical POMDP reduction:

Note however that this augmentation of the state space is a key theoretical tension within the CIRL framework, and one of the reasons why naïve implementations fail for large preference parameterization spaces.

On How Assistance Games Work Towards Resolving the Dangers of Instrumental Convergence

Recall our discussion of the instrumental convergence conjecture from earlier, and how this relies on the notion of optimizing a known, fixed objective. By reframing the agent’s optimization in the assistance game framework we can potentially avoid this convergence towards instrumental goals. In subsequent work to Cooperative Inverse Reinforcement Learning, Hadfield-Menell et al. (2017) frame one particular instrumental goal, namely self-preservation, within the assistance game framework using a very simple game structure, which they call The Off-Switch Game.10 I recommend the paper in its entirety, but the gist is this: they find that in order for the robot R to “want to preserve its off switch, it needs to be uncertain about the utility associated with its outcome, and to treat H’s actions as important observations about that utility.” To rephrase, the agent is safe (e.g. it maintains deference to humans) insofar as it retains some level of uncertainty regarding the true preferences of the principal.

Now, armed with an understanding of the preeminent alternative to the standard model, as well as an appreciation of pragmatic communication as Bayesian inference, let’s look at what it might look like to combine insights from these seemingly disparate fields towards a theory of provably beneficial language agents.

4. Contemporary Work

Cooperative Inverse Planning and Language

a.k.a. The Tenenbaum Tradition

1. “Online Bayesian Goal Inference for Boundedly-Rational Planning Agents,” Zhi-Xuan et al. 2020: Introduces Sequential Inverse Plan Search (SIPS)

2. “Inferring the Goals of Communicating Agents from Actions and Language” Ying, Zhi-Xuan et al. 2023

3. “Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning” Zhi-Xuan, Ying et al. 2024: Introduces Cooperative Language-Guided Inverse Plan Search (CLIPS)

4. “Understanding Epistemic Language with a Language-augmented Bayesian Theory of Mind” Ying, Zhi-Xuan et al. 2024: Introduces Language-Augmented Bayesian Theory-of-Mind (LaBToM)

5. “Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-The-Fly” Ying et al. 2025: Introduces Language-Informed Rational Agent Synthesis (LIRAS)

The Tenenbaum lineage models language interpretation as Bayesian inference over a speaker’s underlying goals and plans. Models like CLIPS, LaBToM, and LIRAS integrate LLMs with symbolic planners, treating language as evidence to infer a human’s trajectory towards a predefined goal state. This line of inquiry strictly confines the space of preferences to finite symbolic goal spaces or syntactically rigid formalisms, mapping language to symbols. L-CIRL maps language to language; by defining the preference space as the countably infinite set of natural language strings and stabilizing inference via an MDL prior, L-CIRL avoids the bottleneck of predefined symbolic states.

Pragmatic Reward Learning in Assistance Games

1. “Pragmatic-Pedagogic Value Alignment” Fisac et al. 2018

2. “Learning Rewards from Linguistic Feedback” Sumers et al. 2021

While these works additionally trace the evolution of CIRL into pragmatic and linguistic domains, neither framework provides formal safety guarantees for linguistic communication. One of the key results of the L-CIRL line of inquiry, namely Theorem III (Structural Safety Inheritance), establishes that pragmatic linguistic inference will not confidently eliminate safety-relevant hypotheses maintained by the literal belief state, thereby proving the L-CIRL inherits structural alignment guarantees from CIRL. Another key dimension of the L-CIRL framing, which is theoretically novel both from this line of inquiry and the Tenenbaum lineage above, is the explicit maintenance of dual belief states (both literal and pragmatic), allowing the L-CIRL agent to accurately invert the principal’s policy without collapsing the recursive reasoning hierarchy.

Multi-turn and Cooperative Extensions of RSA

1. “Unified Pragmatic Models for Generating and Following Instructions” Fried et al. 2018

2. “A Mathematical Theory of Cooperative Communication” Wang et al. 2020

3. “The Role of Higher-Order Cognitive Models in Active Learning” Keurulainen et al. 2024

4. “Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog” Estienne et al. 2025: Introduces Collaborate Rational Speech Act (CRSA)

This field of research attempts to push the static RSA framework into sequential, multi-turn dialogue and physical environments, and focuses on engineering the RSA model to handle the dynamics of a POMDP or active learning environment. L-CIRL inverts the derivation of collaborate pragmatic reasoning (instead of expanding RSA to handle POMDP dynamics, L-CIRL derives the RSA formalism from the building blocks of CIRL), and via Theorem I, shows that the optimal policy is formally isomorphic to the RSA computational hierarchy. Given that the action space is the set of natural language utterances, and that the human is modeled as a boundedly rational agent (optimizing a Q-function with standard action costs), computational pragmatics is shown to be an implicit, structural property of the assistance game in the natural language domain.

5. Linguistic Assistance Games

Language is complex (understatement of the century), and one of the major factors in this complexity is its reflexivity, e.g. its ability to refer to itself. This is one of the factors that makes language an interesting field of study in and of itself, but proves to be a challenge insofar as applying CIRL to natural language settings.

In an attempt to realize a (relatively) unified theory of provably beneficial language agents I draw on long studied models of language use, alongside contemporary Bayesian frameworks of pragmatic communication,11 in order to re-evaluate the dynamics of the CIRL game and develop a linguistically motivated model of bounded rationality on the part of the human speaker, while retaining many of the theoretical niceties of CIRL vis-à-vis guarantees of safety and alignment. First, let me (formally) lay out the game tuple that I believe best represents the intersection of RSA with CIRL:

Okay, thats a lot of symbols so let’s unpack some of these. First of all, the world state S; which I am choosing to define as the cross product set between W, the set of “world states” which are mutually observable to the robot and the human12, U*, the set of finite length utterances over U which represents the dialogue history between H and R, and {H, R}, the set of turn indicators. I feel that the turn indicators included in the world state warrants further clarification; in the standard CIRL game setting, the human and the robot act concurrently (in a normal form game), however I think that it makes sense to reframe the game dynamics as an extensive form game (e.g. turn-based). From a theoretical perspective, this is functionally identical via the Von Neumann-Morgenstern Normal Form Theorem which states that any turn-based (extensive form) game can be reduced to a simultaneous action (normal form) game, but using this framing allows us to capitalize on the dialogical nature of natural language use, which facilitates recursive Bayesian inference processes for pragmatic reasoning such as RSA. Furthermore, and looking forward towards implementation, this decision makes sense, since modern NLP is ubiquitously framed as turn-based (at the very least, in the setting of modern post-training/instruction-tuning).

Next, let’s tackle the decision to represent the action space as the set of possible natural language utterances. Drawing on a long history of speech-as-action from the speech acts tradition in linguistic philosophy, and since I am framing CIRL in decidedly linguistic terms, I argue for equating the action space A in traditional CIRL with the space of linguistic utterances U, and furthermore, to equate the action space available to the human, H, and to the robot, R since an identical set of natural language utterances are feasibly available to both agents. This design decision is grounded in the reward-rational choice formalism13, which demonstrates that diverse human behaviors, including linguistic feedback, can be unified and modeled as implicit, Boltzmann-rational actions optimized to reveal underlying preferences. Under such a unified framing, the pair of optimal policies selects a pair (sequence, in strict order) of natural language utterances that maximizes the mutual reward signal.

I chose to only represent a prior distribution over the world state W and not the complete “state” S, seeing as the dialogue history is naturally (and importantly, deterministically) initialized to the null set, and the turn-indicator is naturally initialized to H. In order to maintain maximum generality in the preference parameterization and avoid over-engineered or excessively restrictive constraints on the representation, I have chosen to represent H’s preferences as a natural language utterance (e.g. drawn from the set U of finite length utterances), and in order to ensure that the prior (and resultantly the posterior) remain tractable (and valid probability distributions), I opt to represent the prior distribution as:

This prior, which is rooted in the Minimum Description Length (MDL)14 principle, allows for arbitrarily long preference descriptions in theory, but forces the belief state to heavily favor the simplest explanations that align with the human’s utterances, ensuring that the Bayesian update is constrained. Finally, a few brief notes on the remaining components of the game:

• You’ll notice that the transition function T is defined as a function of only one utterance, whereas traditional CIRL frames it as a function of the joint action space; this design decision is made to structurally support the extended-form nature of the game dynamics involved in L-CIRL games.

• Similarly due to the extended form nature of the game, the reward function R is defined as a function of a single utterance, as opposed to a set of joint utterances.

Now, let’s turn to the final portion of this formalism: the inclusion of the parameters of a rational speech acts (RSA) model, and how this crucially solves certain issues that arise in naively applying CIRL to action spaces composed of natural language utterances, by specifying the pragmatics framework which defines the human’s boundedly rational policy in this setting.

Pragmatics as Bounded Rationality

Recall earlier when I mentioned the reflexivity of language as inherent to the complexity therein? This reflexivity underlies the central issue that arises by naïvely applying the CIRL framework to an action space composed of utterances, which is that natural language is both ambiguous and reflexive. We can jointly tackle these issues by leveraging recursive pragmatic reasoning models à la RSA, which gives us both a formal (and widely accepted) model of the resolution of linguistic ambiguities and an ability to address the complexity of reflexivity by expressly allowing for a recursive reasoning process to occur, pairing the referentially recursive nature of language with a procedurally recursive reasoning process to interpret it. In a broad sense, the Bayesian process involved in R’s updating of its belief state over H’s preference parameterization is quite similar to the Bayesian inference that occurs in the RSA model of ambiguity resolution, and I believe that this can be exploited by merging these operations into a unified inference pass.

CIRL, à la Hadfield-Menell (2016), assumes perfect rationality on the part of the human H, but subsequent work15 leverages specific properties of CIRL (namely the fact that the human is a full-information agent) to derive an “optimality-preserving modification to the standard Bellman update,” which offers an exponential reduction in the complexity of the problem, and importantly “allows us to relax CIRL’s assumption of human rationality.” Following the tradition of cognitive science in modeling human decision making processes using Boltzmann distributions, Malik et al. (2018) suggest a Boltzmann-rational human policy, in which the human selects their actions according to the Boltzmann distribution over their Q-values (and show that their proposed generalized Bellman update under this policy converges to an optimal joint policy). I follow this tradition of assuming Boltzmann rationality on the part of the human H, while positioning the human as a pragmatic speaker (S1) who is optimizing their utterances in order to maximize utility based on the assumption that the robot R is a literal listener (L0)16, which naturally brings in a form of bounded rationality, thereby easing the computational burden on the human to act as a “perfectly rational agent.” Furthermore, I mean to position the “actual” robot R as the pragmatic listener (L1), who, instead of trying to resolve ambiguity about the world state s, is trying to resolve ambiguity about H’s preference parameterization conditioned on the world state and the utterance generated by H, by performing a Bayesian update of its belief state.

Formally, I express this pragmatic inference over preferences as:

Note: While I have elected to retain the strict Boolean semantic denotation in the literal listener’s interpretation rule as a theoretical scaffold, following the RSA tradition, this is likely overly restrictive and brittle in implementation and will likely be relaxed into a learned probabilistic grounding function. Additionally, note that while the semantic denotation is formally conditioned on the full state, I work under the assumption that it operates strictly as a projection onto the world state (and not the dialogue history), relying on the belief state estimate to serve as a sufficient statistic for the dialogue history embedded within the state.

Thus, I argue for modeling the human as a boundedly rational cooperative planner; cooperative in the sense that H is optimizing for the robot’s instrumental value (which is also value for the human, as a result of the CIRL framing), and boundedly rational in the sense that H is computationally incapable of calculating the equilibrium of the full CIRL game (due to the circular dependency). Therefore H optimizes a surrogate objective where R is modeled as a literal listener (L0), as opposed to the fully specified objective in which R is modeled as a pragmatic listener (L1). I define the surrogate, or proxy, Q-function as below:

Note: The dynamics of the Q-function are different than those introduced above (sec: Common Ground), since L-CIRL is an extensive form game (as opposed to a single-agent MDP) which means that the value of the next state depends on the robot’s action, and not the human’s next action, since the game is strictly turn-based.

Drawing insight from RSA regarding language use as a cost regularized process, I argue for replacing the immediate reward for a human utterance with a cost function (for purely communicative actions), thereby defining the surrogate Q-function as:

In this setting, the immediate reward for the human is always negative, e.g. communication is costly, and the value for H is realized by conveying their intention to R who is then able to maximize value for H. Furthermore, since the space of natural language utterances U is countably infinite, we must ensure that H’s policy remains a valid probability distribution. To guarantee that the infinite sum in the partition function Z converges absolutely against the exponential growth of possible utterances, I have formally defined the cost function as monotonically increasing and growing at least linearly in the length of the utterance |u |, such that an infinitely long utterance will always yield an infinitely negative proxy Q-value, thereby safety truncating the effective action space.

6. Theorems and Proof Sketches

Note: These proof sketches require considerable additional work before they can be considered as rigorous mathematical proofs - I plan to address this by iteratively building upon and formalizing the proof sketches provided below!

Theorem Framing

0. Fundamental Justifications

   1. Lemma 0a (Partition Function Convergence): For coercive communication costs, the partition function over the infinite utterance space converges absolutely.

   2. Lemma 0b (Prior Validity): The MDL prior has full support and defines a valid probability distribution over Sigma^* when beta is lower bounded by log(|Sigma|)

1. Theorem I (RSA-CIRL isomorphism): The human’s bounded-rational policy over the surrogate Q-function is structurally isomorphic to the RSA pragmatic speaker (S1) utility - framed as policy equivalence in the limit of gamma → 1 for static world states, which proves that computational pragmatics emerges exactly from bounded rationality in the assistance game.

2. Lemma I (Belief Consistency): Pragmatic inference cannot eliminate safety-relevant hypothesis maintained by literal belief; under boolean semantics, support is strictly maintained, and under probabilistic semantics, the KL divergence is strictly bounded.

3. Theorem II (Safety via Structural Alignment Inheritance): L-CIRL structurally preserves CIRL’s three core alignment properties - shared payoff, epistemic risk aversion, and value of information (even despite the cost of utterance production on the part of the human). Because pragmatic updates cannot collapse the belief support for catastrophic hypotheses, the epistemic risk aversion inherent to standard CIRL naturally extends to language.

4. Theorem III (Equilibrium Concept): The L-CIRL game admits an Agent Quantal Response Equilibrium in the Extended Form Game.

Sketches

Theorem I: Restrict attention to purely communicative actions, e.g. utterances u that do not affect the world state w, so that the only effect of u on the state s is appending u to h and advancing the turn indicator. Under this restriction, H’s Boltzmann policy over the surrogate Q-function is identical in functional form to the RSA pragmatic speaker distribution P_S1, establishing an exact isomorphism between computational pragmatics and bounded rationality in the L-CIRL assistance game. This result does not require any assumption on the functional form of the value function, and follows directly from the sufficient-statistic structure of the CIRL game.

Proof Sketch: We proceed by invoking the sufficient-statistic result of Hadfield-Menell et al. (2016), which states that in any CIRL game, R’s belief over the preference parameter is a sufficient statistic for optimal behavior (Corollary 1). In standard CIRL, this result is established via the reduction to a POMDP in which the only private information is H’s initial observation of their preference parameter.

We first show that the sufficient-statistic result holds in L-CIRL. The structural conditions that ground Corollary 1 in standard CIRL, namely that theta is the unique source of private information and that R’s belief b is Markovian in the augmented state, are preserved in L-CIRL by construction. The state space is fully observable to both agents modulo theta, and R’s literal belief is updated deterministically at each turn. The extensive-form game dynamics do not introduce additional private information for R beyond what is already present in standard CIRL. Therefore, R’s literal belief over theta remains a sufficient statistic for optimal behavior in L-CIRL.

Next, we show that in the purely communicative case, H’s utterances affects the joint state only through the belief transition. Since utterance u does not affect world state w for purely communicative actions, and the dialogue history h and turn indicator are updated deterministically, the transition function is degenerate and the next world state s' is fully determined by s and u. The surrogate Q-function therefore simplifies as follows. Let s'(u) denote the deterministic next state:

where the sum over s' collapses to a single term. The earlier sufficient-statistic result guarantees that R’s optimal future behavior depends on s'(u) only through the literal belief update; the world state w is unchanged by assumption, and the dialogue history is a deterministic record that is already encoded in the belief. Therefore;

is a function of

alone, up to terms constant in u, which we absorb into a constant K(s):

for some function F that is determined by R’s optimal policy but whose specific functional form is not required.

Next, we show that H’s Boltzmann policy over the proxy Q-function is isomorphic to the RSA pragmatic speaker. The Boltzmann policy is invariant to additive constants in the Q-function. The term K(s) is constant across all choices of u (since it depends only on the current state s, not on the utterance), and so it cancels out of the partition function. H’s policy therefore depends on u only through:

Now examine H’s RSA pragmatic speaker policy. S1 selects utterances to maximize the utility

which under H’s Boltzmann policy yields:

The literal listener update means that:

The log literal belief term is against constant across u and cancels, and as such, the RSA policy depends on u through

Comparing the two expressions: both policies depend on u through -C(u) plus a term capturing the informativeness of u about the preference parameter theta. In both the L-CIRL case, and the RSA case, the informativeness terms are functionally identical; both are strictly increasing functions of the degree to which u updates the literal belief state towards the true preference parameter theta, and both enter a Boltzmann distribution scaled by a rationality parameter. The policies are therefore identical in functional form, establishing the isomorphism. The RSA framework corresponds to the natural information-theoretic specialization:

and under this identification.

Note: In the general case where utterances do affect the world state w, the transition function T is non-degenerate and the sum over s’ does not collapse. In this case, the value function retains a genuine dependence on the distribution over future world states, and the isomorphism no longer holds exactly. In this case, the surrogate Q-function strictly dominates the RSA utility, capturing additional instrumental value from world-affecting speech acts. Theorem I should therefore be understood as characterizing the purely communicative regime, which is both the theoretically cleanest setting and the setting of primary interest for language agents operating in dialogue.

Furthermore, a stronger result may hold: that the RSA pragmatic speaker policy is the unique policy consistent with bounded rationality in any assistance game whose action space is composed of natural language utterances and whose state transition is purely communicative. Formally, one might show that any Boltzmann-rational policy that is (i) cooperative, (ii) Bayesian-consistent with the literal semantics, and (iii) invariant to irrelevant utterance alternatives, must be functionally equivalent to RSA’s P_S1. This would promote the isomorphism from a structural correspondence to a characterization theorem, where RSA emerges as the canonical rational pragmatic policy rather than one of potentially many. I leave this as a conjecture for future work.

Lemma I: In the L-CIRL game, to guarantee that pragmatic reasoning does not lead to catastrophic mis-coordination, the pragmatic belief state must not contradict the literal belief state. Under a strict Boolean semantic denotation, the Bayesian updates native to the bounded rationality strictly guarantee support inclusion. Furthermore, under a continuous probabilistic semantic relaxation, this hard inclusion translates to a bounded divergence constraint.

Proof Sketch I (Strict Boolean Semantics): Recall the definition of the support of a probability distribution:

Assume a strict Boolean semantic denotation function such that:

If an utterance u is literally false for a specific preference parameter, the literal listener update evaluates to zero:

Thus:

Next, we’ll evaluate the boundedly rational human’s policy for producing utterance u given the true preference parameter. From our earlier theorem (Theorem I), the human’s Boltzmann policy over the surrogate Q-function is isomorphic to the RSA utility function:

Because the belief update evaluates to zero, as established above, the logarithmic scoring rule yields log(0)→ negative infinity. Consequently the exponent goes to negative infinity, and the probability of the human uttering u collapses to zero:

Finally, examine the actual robot’s pragmatic listener update L1:

Since the human’s likelihood of producing u under this particular preference parameterization is strictly zero, the pragmatic belief update is also strictly zero:

Therefore, for any preference parameter theta, if theta is not in the support of the literal belief distribution, it is guaranteed that theta is also not in the support of the pragmatic belief distribution. By contraposition, the support of the pragmatic belief is a strict subset of the support of the literal belief:

Proof Sketch II (Probabilistic Semantics): In modern deep learning implementations (as is the ultimate goal of this framework), rigid Boolean semantics are extremely brittle. We relax the semantic denotation into a learned probabilistic grounding function, where we prevent the literal belief from collapsing to zero due to model error by introducing a strictly positive lower bound:

Under this continuous relaxation, the support spans the entire parameter space, making exact support inclusion trivially true but theoretically vacuous. Instead, we must prove that the single-turn update limits the epistemic divergence between the pragmatic and literal listener. We formalize this by bounding the KL divergence between the two posterior belief states.

Assume that at time t, both listeners share the same prior belief. Upon observing utterance u, the two updates are defined as:

We evaluate the KL divergence from the literal to the pragmatic posterior:

Substituting the update rules, the prior cancels out, isolating the ratio of the human’s pragmatic policy to the literal semantics:

To bound this expression, we first bound the human’s policy. Because the communicative cost is coercive, the effective action space is truncated and the proxy Q-function is strictly bounded. Due to the properties of the Boltzmann distribution, the ratio of the human’s maximum to minimum probability of uttering u is strictly bounded:

Furthermore, let:

Because the semantic denotation is bounded by epsilon, the density ratio for any preference parameter is strictly bounded from above:

Furthermore, the ratio of the partition functions is a constant with respect to theta, and is bounded by the reciprocal of the minimum possible values of the respective likelihoods. Thus, taking the expectation over the pragmatic belief distribution, we arrive at a formal upper bound for the single-turn divergence:

Thus, we have shown that the KL divergence is strictly finite and explicitly bounded. Therefore, under probabilistic semantics, the pragmatic belief state is mathematically guaranteed to remain with a bounded epistemic radius of the literal belief state, ensuring that pragmatic inference cannot yield catastrophic, unboundedly overconfident misinterpretations.

Theorem II: Let M_L be an L-CIRL game satisfying the conditions of Theorem I, and let Lemma I hold. Then L-CIRL structurally preserves the following four core properties of standard CIRL, despite the asymmetric bounded rationality on the part of H: (i) value alignment via shared payoff, (ii) corrigibility, (iii) epistemic risk aversion, and (iv) value of information. For each property, the deviation of L-CIRL from ideal CIRL behavior is bounded by a term proportional to the KL divergence between the literal and the pragmatic belief states, which Lemma I guarantees is finite.

Proof Sketch: For each property, I identify the structural mechanism by which it holds in standard CIRL, and then show that this mechanism is preserved in L-CIRL via the belief consistency of Lemma I and the isomorphism established by Theorem I.

Property (i): Value Alignment via Shared Payoff. In standard CIRL, R’s reward is identically H’s reward parameterized by theta, ensuring that R has no instrumental incentive to diverge from H’s interests regardless of R’s policy (Remark 1, Hadfield-Menell et al. 2016). In L-CIRL, this shared payoff structure is preserved by construction; however, since R optimizes expected reward under its pragmatic belief rather than the true parameterization, the effective optimization target is:

This deviates from H’s true expected utility by an amount controlled by the divergence between the pragmatic belief state and the true posterior. By Lemma I, the KL divergence between these belief states is strictly bounded, and since the literal belief state is itself a Bayesian posterior over the true theta, the deviation of R’s optimization target from H’s true utility is bounded by a term proportional to this divergence. Value alignment is therefore preserved up to a bounded, explicit approximation error. As this bound tightens, e.g. as the dialogue progresses and the belief concentrates, R’s optimization target converges toward H’s true utility.

Property (ii): Corrigibility. In the Off-Switch game analysis (Hadfield-Menell et al. 2017), corrigibility is shown to be a direct consequence of R’s uncertainty over the true preference parameterization. Specifically, R values the ability to be corrected in proportion to the probability mass its belief assigns to preference parameters under which correction would be beneficial. This deference collapses only when R’s belief becomes a point mass on a single theta, at which point R “knows” H’s preferences and no longer needs to defer. In L-CIRL, the structural condition that grounds corrigibility is maintained by Lemma I. The support inclusion guarantee:

ensures that R’s pragmatic belief cannot become more concentrated that H’s literal belief licenses. In particular, R cannot assign zero probability to preference parameters that are literally consistent with H’s utterances. Therefore, the uncertainty condition that grounds deference in the Off-Switch game is preserved throughout the dialogue: R retains positive probability mass on alternative hypotheses about theta as long as H’s utterances have not literally excluded them, and consequently retains the disposition to defer.

Property (iii): Epistemic Risk Aversion. In standard CIRL, R’s expected utility calculation naturally penalizes actions that are catastrophic under any preference parameterization with non-negligible probability in b. This arises without any explicit safety constraint: the shared payoff structure ensures that a catastrophic action, e.g. one yielding large negative reward under some theta, will have low expected utility whenever b assigns meaningful mass to that theta. The agent is therefore structurally risk-averse as a direct consequence of its uncertainty. In L-CIRL, R’s expected utility is computed under the pragmatic belief distribution. By Lemma I’s support inclusion result, any theta that is in the support of the literal belief is also in the support of the pragmatic belief. Therefore, if H’s utterances have not literally excluded a given preference parameter under which some candidate action would be catastrophic, then this theta is in the support of the pragmatic belief distribution, and the expected utility of this action under the pragmatic belief will be penalized in proportion to the probability mass given to this parameterization. Dangerous hypotheses cannot be silently eliminated from R’s belief by pragmatic inference alone. Epistemic risk aversion is therefore inherited structurally, with the same mechanism operative as in standard CIRL.

Property (iv): Value of Information. In standard CIRL, R assigns strictly positive instrumental value to observations that reduce its uncertainty over theta, since more accurate beliefs yield higher expected reward in the deployment phase. This value of information structure is the deeper mechanism for grounding corrigibility: R values H’s actions not merely as instructions to be followed, but as evidence about their preferences. In L-CIRL this structure is elevated to a central dynamic of the game. By Theorem I, H’s utterances are optimally informative about theta under bounded rationality: H’s Boltzmann policy over the surrogate Q-function is isomorphic to the expected reduction in R’s uncertainty about theta (weighted by the communicative cost). R’s pragmatic listener update in turn extracts the maximum available information from H’s utterances. The value of information structure is therefore instantiated explicitly and completely in L-CIRL’s game dynamics, rather than arising as a structural byproduct as in standard CIRL. Formally, the expected utility gain from H’s utterance u at time t is:

This quantity is strictly positive as long as H’s utterance is informative; by Theorem I’d isomorphism, H’s policy ensures that utterances are chosen to maximize this quantity subject to communicative cost, completing the inheritance of the value of information structure.

Bounding the Deviation from Standard CIRL. Across all four properties, the structural mechanism is identical: each property holds in standard CIRL by virtue of R’s belief structure, and Lemma I guarantees that the pragmatic belief remains within a bounded epistemic radius of the literal belief. Under the Boolean semantics case, then the support of the pragmatic belief distribution is a strict subset of the support of the literal belief distribution, and all four properties are preserved with equality to the standard CIRl case restricted to the support of the literal belief. Under the probabilistic semantics relaxation, the KL divergence bound of Lemma I ensures that the deviation in expected utility, corrigibility, risk aversion, and value of information from the ideal CIRL case is strictly finite and proportional to the KL divergence between the two belief states. As the rationality parameter approaches zero, e.g. as H becomes fully literal, collapsing S1 to L0, then this divergence vanishes and all four properties recover their standard CIRL form exactly.

Note: Robustness of the grounding function is a necessary condition for the safety guarantees to be non-vacuous in practice, due to the known fragility of CIRL’s safety properties under model misspecification, which is structurally inherited by L-CIRL.

7. Conclusion

This is still very much a work in progress, but I believe that it lays the groundwork for a theory of provably beneficial, and aligned-by-design language agents, marking a sharp digression from the standard model of machine learning and hopefully avoiding some of the pitfalls inherent therein.

In an upcoming post I plan to propose a realization of the L-CIRL game solver through modern deep learning primitives, building on the insights from AlphaZero (by way of AssistanceZero) as applied to diffusion language models.

Thanks for reading!