Why are "Left" and "Right" So Hard for LLMs?

Large language models are a fundamentally different type of intelligence from humans; and this means that even as they approach and even exceed humans in some domains (coding!) we will still see some places where they lag behind human intelligence in surprising ways.

Spatial reasoning is one of these — telling the difference between “left” and “right” in various contexts is still often hard for even very powerful models. Says one user on X:

This shortcoming of language models has actually existed for a very long time and it’s an interesting one that, in my opinion, is illustrative of the structural weaknesses of learning principally from language, and why these weaknesses aren’t as true for, say, color. And so, a short post on left, right, colors, and why some things are harder for LLMs and VLMs to learn than others.

Language does a terrible job specifically of capturing spatial information. Why is this?

Spatial concepts just aren’t obvious and well-defined from the vast corpus of text available to train on. What does it really mean to walk across a room, or to put a book on a bookshelf? The precise spatial locations that we might need to unambiguously resolve this missing information into a clear concept just isn’t a part of the training data in any real way.

I’ve written about world representations for robots in the past (a term which I use very differently from “world models”). For my world representation to be useful, it needs to capture, for example, the positions of all of the various objects in a house, so that I can navigate to them, manipulate them, and ask questions about them.

The process of using sensor measurements to answer these questions is usually referred to as grounding the concepts. The symbol grounding problem deals with how we connect these symbols (on top of a table, inside a box, lined up, clean…) into an understanding of the physical world.

There’s actually a surprising amount of information in LLMs on how objects should be arranged, but there’s always missing information too — details about geometry and contact, about object shapes and physics. Training on text is like learning from a textbook with half the pages torn out, or watching a YouTube video with captions in the wrong language. You’re probably still going to learn something, but a lot of the key context is missing.

But many of the leading frontier models these days are actually multi-modal, taking in images or even short video clips. Unfortunately, this is still not enough to resolve the problem — it persists even in a vision-language model (VLM).

VLMs are trained on images from a single perspective, or in the case of a short video a handful of highly correlated perspectives. This reduces ambiguity and helps resolve some of the missing context, but hardly removes it completely. What does “left” mean in the context of the portrait? My left, or the left of the person in the picture facing me? These things are opposites.

Such ambiguities aren’t really a challenge for modern VLMs (I can just specify “my left” or “their left” in the prompt), but complexity and ambiguity remain. Problems like precise estimation of 3d object position from images remain challenging, which means that for many real-world tasks, the VLM just doesn’t work to a reasonable level of reliability.

Consider how an LLM might leverage its training data to understand colors (red, purple, blue..) versus left and right.

Color is used consistently. An apple is red; the sky is blue; leaves are green. Even if I do not know what green “means” — in terms of specific RGB values — I do know which things are described as green, and because green represents a distribution of RGB values, it’s used fairly consistently and it’s easy enough for a model to understand that something will be green just based on this data.

However, as mentioned above, left and right are highly context dependent. They’re going to be used in millions of different ways, to mean different things.

This is an important illustration of an idea that I’d like to write more about: I fully expect AI and robotics, now, to be able to exceed humans in many domains. This does not necessarily mean they’ll exceed in all domains; there will be some “comparative advantage” where humans remain leaders in some tasks. It’s hard to say what those are; but we can look at the types of learning and reasoning that humans do well for examples.

Humans, for example, operate off of comparatively noisy training data. We set our own learning goals rather than relying on someone else evaluating us. We get bored and do something else, managing our own time to keep us from getting stuck in pointless loops (except during grad school).

Our “training data” is based towards real, streaming, multi-modal data that’s aggressively compressed - using far more compression than works with modern transformers. Humans have to disregard millions of useless signals from various sensors just to go about our daily lives. By contrast, LLMs and VLMs are trained using mostly still images and mountains of text — they’re learning concepts like “left” and “right,” “purple” and “green” all from associations.

There are lots of cases where machine and human intelligence just end up working differently, due to these differences in compression, training data, and of course our different computational hardware.

Ethan Mollick has written about us approaching a sort of jagged AGI, where modern models exceed human intelligence in some areas but fall behind in others, making comparatively little progress. When you look at how these models are learning and what they are doing, I think we’ll find many of these shortcomings are due to fundamental differences in the way human vs machine intelligence works.

This is obscured by the fact that usually, we don’t need our AI agents to interact with the physical world. They inhabit a realm of text prediction and function calls just fine. But this still represents a fundamental limit of what they can do, which can only be overcome, I think, by figuring out how to scale embodied data.

This is all to say that certain types of intelligence, I believe, only really make sense with embodiment. Text, and even much of the image data on the internet, fundamentally can’t learn to solve all of artificial intelligence — embodiment matters.

This is also another reason to be excited about world models, which potentially can start to understand these concepts and learn to ground the various spatial relationships that we talk about in reality. World models take in and predict video data; because of this, they can actually learn an implicit model of the world, of how objects interact, and learn (implicitly) to understand the 3D space that humans and embodied agents must inhabit.

Perhaps the most interesting question, then, is this: will embodied data eventually allow our AI agents to become better at, say, using 3D modeling software, as per the tweet in the intro to this blog post? I believe so; it seems possible at least that these concepts will be best captured through large scale interaction.

After all, it would make sense if, to achieve human-level artificial intelligence, we needed agents to learn the same way humans do.