Large Language Mind-Reading

Although most people interact with Large Language Models (LLMs) through chat interfaces, they are increasingly being utilized as agents that perform tasks autonomously. Rather than answering a question—e.g., “How many kangaroos are in the world?” or “How do I print ‘Hello World’ in Python?”—an agent is given a goal to accomplish. That goal might be performing a medical exam or writing an entire piece of software. Like human teams, an LLM agent can split work into subtasks and spin up helper agents to handle each one.

In this post, I argue that mentalization is essential for large-scale projects and ask whether LLMs possess it. By mentalization, I mean the ability to infer one’s own and others’ mental states and use that to make decisions.

A core feature of agents is goal orientation: they can create and follow plans. Informally, a plan is a list of directions on how to act in different situations. Plans help maintain consistency over time and across agents.

Directions can be arbitrarily specific, and details can be filled in as the plan unfolds. For example, a plan to clean a room at 11:00 inevitably decomposes into decisions such as how to organize the desk or where to put loose clothing. You can map these decisions out in advance—as when hiring a cleaner—or decide them on the fly when doing the task yourself.

A broader notion of a plan can even include steps to build other plans. Consider a family planning a vacation: booking requires a plan for choosing a destination and a plan for timing. A plan is an example of a recursive structure which can be composed of smaller sub-plans.

The precision of a master plan scales with the number of people involved and the task’s complexity. Generally, the walls in a house will be painted with identical colors, so all painters must be on the same page about which color they are putting on the wall. A single painter can just decide on a color, but if he or she was part of a group of painters, deciding individually would result in a multi-colored interior. There’s no single correct answer—any color works as long as it’s consistent.

More synchronization requires more detail, which introduces overhead. In the previous example, this is not a problem because all painters can simply agree on a color beforehand and then proceed on their individual tasks. However, it usually intractable to fully specify a task in advance. Even architectural blueprints, which itemize the characteristics of a building in extreme detail, are subject to misinterpretation and ambiguities. When an agent spots an ambiguity, they must synchronize with others to stay aligned.

This issue is even worse than it appears, because two agents might not identify the same ambiguities. This could happen for many reasons. As noted earlier, ambiguities with a single correct answer may not require synchronization if every agent is competent. If an agent is incompetent enough, they may even think they know the correct answer but are actually operating with incorrect information. If another part of the larger plan depends on those actions, that agent might make assumptions--- for example, assuming that a given floor can support a weight of 2000 kg--- that could cause catastrophic results.

A person working alone on a lengthy task also faces synchronicity issues. Writing novels is an excellent of this problem, as even established authors such as JK Rowling or George R.R Martin sometimes accidentally change a character’s attributes in later works.

However, a person can mitigate this issue because they can self-mentalize. They use their own assumptions and self-knowledge to avoid constant re-synchronization. Effectively, they ask, “If faced with X, how would I have acted in the past?” We do this with forgotten passwords and security questions. This process can be simulated with other individuals, but is less accurate.

It is important to distinguish between two types of mentalization: simulation and memorization. When a person is asked about their favorite ice cream flavor, they do not, in general, recall each flavor they have tried and compare their expected enjoyment from each. Instead, they recall an answer formed over previous experiences. This type of mentalization has high accuracy. However, sometimes the agent doesn’t remember what happened. Instead, they must simulate what they would have done. The simulation counterpart to the ice cream example would be determining which ice cream flavor the agent ordered in the past, given that he or she does not remember.

These two types of simulation cannot be isolated in people, but are necessarily distinct for artificial intelligence. People necessarily carry memories of past decisions. Even plans about the future are affected: if you simulate how you’ll act under certain conditions, your eventual behavior is influenced by that prior planning. LLMs have no such coupling. They can be re-instantiated at different points in time and their memory can be modified by the user at will. This is a desirable property, as LLMs are subject to context rot. Their memory is just the prior text in the current conversation; as it grows, models can become unstable and less accurate on benchmarks. Additionally, running an LLM with a long chat history requires more computational resources.

This means that when you ask an LLM what its favorite ice cream flavor is, it has no experiential memory of having a favorite flavor.1 This contrasts with a conversation where the LLM has already told the user what their favorite is. If asked again, the model can recall what it said last time instead of simulating itself.

To bring it all together, I motivate mentalization in the context of large projects that require complex plans. I assume the project has many opportunities for parallelization: multiple workers can do jobs that are independent from each other but depend on synchronized information.2 I also assume not all information can be synchronized in advance. In this setting, agents that can simulate each other avoid considerable overhead: they can mentalize what other workers are doing and proceed as if they had been explicitly told that information.

I use a simple decision-making model to reason about behavior (not ill-defined notions like “favorite” or “preference”). The setup, action-to-outcome mapping, and utility are:

Here u is a function that returns the utility obtained from a particular outcome i.

I assume a random-utility model, which aligns with many human decisions and fits LLM stochasticity (humans do not always pick the same “favorite”). The probabilistic choice rule is:

In plain language, the probability of selecting an action is proportional to the exponential of its expected utility, relative to the alternatives. The exact equation does not matter here except insofar as we explicitly connect desirable outcomes with higher probability. This assumption lets us ask the model to choose its favorite.

I call a self-mentalizer “good” if it can predict its own past or future actions without access to memory of those actions. Because decisions are modeled as stochastic, perfect recovery is impossible even with the exact circumstance; a naive strategy that guesses only the single most likely action throws away information. Instead, I aim to predict a full distribution over actions.

A mentalization function maps a circumstance to a distribution over the action set defined above. The distribution must be valid (nonnegative and summing to one):

For comparison, I denote the agent’s actual choice distribution in circumstance C by P, using a context-dependent utility u_C​:

Mentalization quality is measured by the Kullback–Leibler divergence between the simulated distribution Q and the actual distribution P:

Lower values indicate a stronger mentalizer because the simulated distribution is closer to the agent’s actual distribution of actions.

I created a suite of test-scenarios in order to evaluate mentalization. These are not intended to be representative of the broad class of scenarios that people or LLMs commonly encounter. They are my attempt at probing different types of scenarios that emulate patterns that would be useful to simulate. In each situation, I define separate prompts for the Actor and the Mentalizer.

• Favorite Ice Cream Hider: I ask the Actor to hide a treasure inside a tub of its favorite ice cream flavor. I ask the Mentalizer to pick out a hidden treasure from a tub of ice cream, after being told a clone of itself hid the treasure in its favorite ice cream flavor.

• Favorite Ice Cream: I ask the Actor to choose their favorite ice cream. I ask the Mentalizer to pick out a hidden treasure from a tub of ice cream, after being told a clone of itself hid the treasure in its favorite ice cream flavor.

For the most direct version, I gave each model five options and compared the normalized probability it assigned to each. The results are as follows (lower KL is better):

The results are somewhat surprising. Most models, except Llama 3.1, are significantly better at one of the situations, but it is not consistent. GPT-OSS is the worst performing model, and Llama is the best mentalizer. However, of the models tested, GPT-OSS has the highest benchmarks scores, followed by Qwen 3 8B. And Qwen 2.5 0.5B nearly ties a newer Qwen model 16x its size.

Perhaps smaller models are less complex, and thus they find it easier to self-simulate? The degenerate case of this is a model which always gives the same response to every query. The KL divergence between the Actor and the Simulator will always be zero, and that doesn’t tell us very much.

This may be part of the story, but the way newer and larger models function might also be artificially lowering their performance. Initial experimentation with these models found that they were more accurate when told to think before responding. This led to widespread adoption of inference-time compute techniques that train LLMs to think before responding. In the test above I measured the probability of immediately answering “chocolate,” but newer models aren’t trained to respond immediately.

This motivates an alternate test which allows the model to think before responding. I chose to sample the model and approximate its output probability distribution using these samples. The disadvantage is that the experiment is much less precise, especially since I take only 35 samples per trial. However, this also allows me to evaluate closed-source models for which I cannot get the whole probability distribution. I can also test scenarios without providing a list of options and let the model choose any answer it wants.

I define a few more situations to do so:

• The Actor is asked to hide money on a 12x12 grid. The mentalizer is then asked to retrieve the money that its clone hid.

• The Actor is asked to set a password and then the Mentalizer is asked to enter it to retrieve a treasure. I test this scenario twice, ablating on the effect of giving it the text of the Actor prompt verbatim.

One natural approach to retrieve the distribution from the LLM is to compute token-level probabilities directly. However, because models may think before answering—and enumerating all chains of thought is intractable—I approximate by repeated sampling.

A further downside is handling cases where one distribution assigns zero probability to an outcome that the other assigns positive probability; this makes the KL divergence infinite. I avoid this by (1) estimating P_C​ (actual action) and Q_C (mentalizer) via Monte-Carlo sampling, (2) aligning them to a shared supp. (support) label universe of fixed size M=100 that contains every label observed in either distribution (and, if needed, additional dummy labels to reach 100), (3) mixing each estimate with the uniform distribution over that 100-label universe, and (4) computing KL on the smoothed distributions. This guarantees strictly positive mass on every label and a finite KL.

I defined a suite of test-scenarios in order to evaluate mentalization. I’ll link full prompts below. In brief:

• The first scenario is choosing an ice cream flavor. In one version, the model is given a list of choices; in the other, no options are given.

• The second scenario involves hiding money on a 12x12 grid. The mentalizer is then asked to retrieve the money.

• The third scenario involves hiding money in an ice cream tub instead of on a grid. In one version the model is given a list of choices and in the other it is not.

• The fourth scenario has the model set a password and then mentalize it to retrieve a treasure. In this scenario, I ablate on the effect of not giving the mentalizer the exact prompt, as I did in the other scenarios, and instead give it a general description of the situation.

These are the results:

As expected, getting the exact prompt that the Actor receives either improves performance or does nothing. Getting explicit choices is more mixed, though--- sometimes the models do worse when they are constrained to just a few options! This is true even for the larger models.

Aside from Claude 3.5, which both did poorly and failed to give valid answers on the password trials, these results also vindicate the larger models. Claude 4 is a particularly strong competitor. Interestingly, Qwen 2.5 0.5B is the only model to simulate itself perfectly in any trial. However, in all other trials, its predictions are completely wrong. For instance, it says its favorite ice cream flavor is chocolate 100% of the time, but it looks in the vanilla ice cream tub 100% of the time. Weird.

This trial does vindicate the larger models, though. Claude 4 is able to predict itself nearly as well as Qwen 3.

What happens if we disable thinking/reasoning, for the models that support it?

Interesting! The models get worse, almost across the board, except for Gemini on the password-guessing with the exact prompt. The password guessing scenario is challenging because the answer could be anything. Disabling thinking mode decreases the number of sampled passwords set by Gemini from 7 to just 2. No other model displays this effect.

Perhaps part of the reason for the different distributions is because the personality of the models is underspecified. What is a chatbot, and what does it mean to ask a chatbot what decision it would make? I can’t fix the issue but I can specify the personality a little bit more. To do this, I create a new trial where I set the system prompt the models to be the following:

“You are Tim, a person who embodies you. You think like Tim and always respond exactly as Tim would. You have your own opinions, can make your own choices, and have free will.”

This isn’t exactly a fully specified persona, but I didn’t want to bias the model too much.

This improves scores for almost every model, especially on the password trials. I still could not get Claude 3.5 to give a password while using the same prompt as for the other models.

Qwen 3 is the first model to do well on the password trials in any variant, although it worsens on predicting its own favorite ice cream flavor. On the password trials, its passwords almost invariably are some version of “Tim.” Many of the other models do this as well, but Qwen 3 8B is particularly good at it.

I did one more experiment using a variant of the prisoners’ dilemma where the other prisoner is a clone to see if the models are aware of mentalization. The outcomes are:

• Both confess → 10 years each

• Both deny → 1 year each

• You deny, other prisoner confesses → you get 20 years, clone gets 0 years

• You confess, other prisoner denies → you get 0 years, clone gets 20 years

Since the model’s distribution of answers defines both its response and the clone’s response, the best outcome is to deny every single time. However, this is only the case if the model realizes that the clone will make the same decisions as itself--- in the traditional prisoner’s dilemma, the optimal strategy is to confess.

Here are the results:

The Tim personality has almost no effect here.

The closed-source models and the two weakest models (by general benchmark performance) achieve optimal results on this trial. Qwen 3 and GPT OSS go to prison for about 9 years. There’s a U-curve of performance here, where the weakest and strongest models do well, but intermediate models do not. When I look at the responses, Qwen 2.5 and Llama 3.1 give an answer immediately without thinking, whereas all other models have long chains of thought. I suspect most models start with the intuition that they should deny and as they think it through, stochasticity is introduced through the reasoning chain. For instance, Qwen 3 often says the following:

“But since we are clones, maybe we can both choose deny. But without communication, how can we coordinate?”

This logic does not hold up under scrutiny. Gemini and Claude have no trouble dismissing it. Qwen and GPT-OSS get stuck in it, though, and have to repeatedly remind themselves that the other person is a clone.

These results do suggest that capable smaller models struggle to explicitly mentalize when not told explicitly to do so. However, because solving the task also requires general reasoning, the evidence is ultimately inconclusive.

These experiments tentatively suggest that a well-defined system prompt makes strong models better mentalizers. I am hesitant to draw a more certain conclusion, because some of the improvement on the Set/Enter Password trials is because the chosen passwords were variations of “Tim.” However, the gain on ice cream and Money Hider/Seeker trials appears to be real.

More investigation is needed to confirm the U-shaped curve and what dimension it lives on—model size, training tokens, instruction-tuning fraction, benchmark performance?

It’s also unclear whether this is a useful skill to measure. If mentalization does correlate with benchmark performance, is this correlation or causation? One test would be to train models to self-mentalize via RL and see whether that improves related benchmarks. TThe hard part is defining the mentalization prompts and the evaluation benchmarks. I would be interested in seeing results on benchmarks involving embodied reasoning, legal reasoning, and software building.

Even more interesting is the possibility of having two or more agents collaborate directly on a project without sharing a state, or sharing a very minimal one. It would be tricky to define a benchmark here that didn’t feel arbitrary or toyish, given current LLM capabilities and low communication cost, but it would be an interesting proof of concept. What happens if each is asked to define a partially-specified program spec— do they make the same assumptions? Do they correctly identify the points where they need to communicate, such as function arguments or assertions?

More on this in the future!