Why Robots Might be a Lot Harder Than LLMs

There have been some incredible advances recently in robotics, especially in humanoid robots like the Tesla Optimus. Robots are widely seen as the next frontier of AI after language models, providing a way to bring the power of AI to bear on the physical world. Elon Musk recently predicted that there would be 10 billion robots by the year 2040, which if true would entail unfathomably large changes to the economy and society. I generally agree that robots will play an increasingly important role over the next century, but I also have strong reasons to be skeptical that general purpose humanoid robotics will advance as fast as has been predicted.

Disclaimer: These are purely my opinions as someone who has never directly worked with humanoid robots, and I’m sure there are important aspects I am missing. It’s possible that dimensionality reduction for world models constructed from sensory data is simply much more effective than for language models, and/ or that we will be able to generate astronomical amounts of physically accurate synthetic data. These very well might be possible, but I have no intuition either way.

The most decisive question in determining the future economic impact of robots is whether robots will be able to generalize to a wide range of tasks that are currently performed by human workers. These would include tasks requiring precise movements and the dexterous use of tools like carpentry, electrical installation, and assembling furniture. If robots can’t be scalably trained to do these types of jobs, then it’s hard to imagine what several billion of them would be doing at all.

I think the biggest reason to be pessimistic about robots being able to perform general tasks requiring high degrees of precision and dexterity is the dimensionality of the space they are operating in. For comparison, an LLM is trained to predict the most likely token in a string out of a vocabulary of about 100,000 tokens. This seems high, but a humanoid robot like the Tesla Optimus has over 60 independent degrees of freedom, including 22 in each hand (still less than the 27 in each human hand), with each degree of freedom corresponding to a motor controlling a joint. Even for motors that can only take on one of three control parameters (for example, positive, negative, or neutral), the action space of such a system is enormous (~3⁶⁰ possible states), many orders of magnitudes higher than that of an LLM.

On top of that, the observation space of a humanoid robot is vastly larger than that of an LLM. There isn’t really a one-to-one comparison between context for language models and world models for a robot, but consider that a “context” of 100,000 frames from a 30 frames-per-second camera (many more bits of data than any LLM context window supports) would constitute a working memory of less than one hour. A robot must not only process this entire world of sensory data, it must respond to environmental changes in real time, with a reaction time of less than 250 milliseconds in order to be competitive with humans. The ability to respond to changes in the environment in a way that aligns with the interests of humans is a complicated facet of behavior that involves questions not only of law and ethics but economic efficacy and even personal courtesy. How will a robot handle situations like disposing of toxic waste on a job site, or a battery dying in a power drill? A worker (human or robot) that fails to handle these situations appropriately would be a hindrance to coworkers and a liability to employers.

An additional challenge of humanoid robots replacing humans across many roles very soon is that the type of training data that enables LLMs is not available to the same extent as it is for robots. LLMs are uniquely privileged in that they have an abundance of training data in the form of the internet and the entire corpus of digitized text, like code repositories and textbooks. There isn’t anything quite like this for robots. Although there have been very impressive demonstrations of using physically accurate simulated world models, video datasets, and synthetic data generation to train robots, it’s not clear yet whether these approaches will enable generalization over a wide range of economically valuable tasks.

I think it's very likely that humanoid robots will soon be able to take on a range of tasks that are currently performed by humans, and that this repertoire of tasks might expand at an accelerating rate. We’ve seen that physically accurate simulated world models are ridiculously powerful for scalably training complex behavior, and that foundation models are becoming more advanced and efficient at a rate faster than most would have predicted. But most likely these will be specialists that are optimized for a controlled environment designed rather than generalists that can adapt to unpredictable demands, and it will take a huge amount of effort to fine-tune models for the range of tasks they will be applied to.