I Agree With Yann LeCun

Everybody is trying to figure out where AI will end up, including the leading AI researchers. The truth is that no one actually knows where AI will end up. But since everyone is trying to figure it out, you don’t have to have your own view; you can just choose someone else’s.

Say you’re worried that AI will have a largely negative effect on humanity?: Geofrey Hinton is your AI guy. Dario Amodei and Ilya Sutskever are right up there.

You think that’s all BS, and AI will only improve the human condition, then Andrew Ng or Fei Fei Li will suit your temperament. (Or Elon Musk? 😳) If you think that defining yourself as either a boomer or doomer isn’t practical then pay attention to Demis Hassabis or Mira Murati.

I have my own point of view, which happens to tightly integrate with Yann LeCun’s point of view. (I make this statement with a measure of humility; all of the AI researchers have spent their lives trying to figure out Artificial General Intelligence AND are way smarter intellectually than I am. All I can do is to try to figure out the patterns in what they are saying.)

LeCun believes that LLMs have fulfilled most of their promise and that investing billions of dollars in new iterations of LLMs will not move AI forward. This article is a great explication of why. Instead, he argues that we need to shift to a new paradigm, based on world models, representations of the real world that train the AI to understand how real things work with each other.

He is not the only one; Fei Fei Li at World Labs is pursuing world models. Our portfolio company Niantic Spatial, lead by John Hanke, is also focused on using world models in the real world. OpenAI’s Sora is said to be a world model for making video. Anything to do with agents is considered a world model, since agents have to interact with existing systems to complete tasks.

You could put world models into three buckets: simulation, which means making videos look more and more like the real world; robotics, which refers to mapping our own physical environment for machines to operate in the real world (including autonomous vehicles); and “ground truth”, which refers to operating real-time systems that are grounded in the real world, like controlling air traffic, managing satellite networks, or fighting wars.

The part that engages me is that we humans all operate in the real world with our brains and our bodies. A perfect world model would mimic the human brain: learning in real time from everything around it so that it becomes inherently smarter. I’ve thought for a while that LLMs are self limiting because building them is a batch process. See Still Haven’t Any AI Around Here, 10/18/24. Since writing that piece, it’s clear to me that the whole AI effort is beginning to focus on how to move beyond LLMs.

A lot of the researchers focused on world models cite cognitive neuroscience as the intellectual basis for developing world models. Cognitive neuroscience simply means studying the human brain as an analog for artificial intelligence. The brain is a relatively small organ that is capable of complex language, abstract thought, long-term planning, and cumulative culture. So the challenge is: Can we reproduce the functionality of the brain in a machine? So far, we’re nowhere near that since just predicting the next letter or word requires so much data, compute, and energy that it is warping our existing systems for power consumption and data storage. That said, LLMs are only able to impress with their speed and ability to reproduce what we have already created. This is astonishing in fields like programming, structured language, and mathematics. But it doesn’t promise the ability to operate in real time in the real world, much less to be creative or intuitive.

If you believe that LLMs are a dead end, you can also posit that the dynamic of buying lots of GPUs to munge lots of text in a huge data centers might not be the path to the promised land. The industry has already spent hundreds of billions of dollars rebuilding LLMs to the point where the incremental value in the large language models is smaller and smaller. It appears to me that we have all already collectively decided that only three of those foundation models matter: Gemini, Claude and (maybe?) ChatGPT. The others: Grok? Perplexity? Lama? Deepseek? Meh. In other words, the AI industry has already decided the 1-2-3 question: In every new core technology, there is a #1, and #2 and a #3.

If the new core technology is world models, then we have to imagine what it takes to create and operate those world models. The human brain stores all information in real time in the real world. Every increment of new data causes the whole model to update. This system runs on a small amount of power (the popular notion is that it taskes less power than a 60-watt lightbulb) without any backup storage for power or data. If the elements of the human brain are the basis for designing world models, then will operating any one world model take as much energy and processing power as an LLM or will it have different metrics?

I don’t know the answer to that question, although it makes sense to me that if it is really a real-time, real-world system, it doesn’t need a batch process to prepare the data for use. And that would likely lead to a different set of economics. And it is exciting and interesting because it promises a whole other set of technologies beyond what we already know about.

PS: I am providing two links below that I found really interesting and relevant while I was working on this piece. The Geometry of Intelligence is an interview by Shawn Wang and Alessio Fanelli of Latent Space with Fei-Fei Li and Justin Johnson, co founders of World Labs. I struggled to keep up with them, but it’s worth the struggle!

Then I watched The Thinking Game, a documentary about Demis Hassabis, founder of DeepMind, now Google’s formal research effort around defining and achieving Artificial General Intelligence (AGI).