Language models are not, and will never be, conscious

Large language models are complex next-word predicting machines. They are not a gateway to sentience. But economies are rising and falling because people are being deluded into believing the claim that LLMs are much more than they actually are. The people who lose in this situation are those who take this claim at face value. Those who start humanoid robotics companies because they believe that once others figure out the software, they’ll be ahead of the curve. Or those who quit school because they believe that AI will take away the jobs they would have pursued. The people who win in this situation are venture capitalists - those who pump money into “AI” companies and ride this wave of inflated expectations to handsome exits. I was recently vindicated by learning that Yann LeCun, Meta’s Chief AI Scientist, is on the record saying, “We are not going to get to human-level AI by just scaling up LLMs.” I could not agree more.

LLMs are good at several things. They map words and phrases together. The model sees “door,” “entrance,” and “the thing you walk through” as related concepts. They are good at predicting next tokens. After “the cat sat on the,” it knows “mat” is more likely than “baby.” They are good at recognizing patterns. If the text reads “a man walks into a bar,” it detects the setup of a joke and predicts a punchline-style ending that tends to appear in similar contexts. They are good at drawing on their corpus of knowledge. If asked “Who wrote 1984?”, it answers “George Orwell,” even if not stated in the prompt.

But an LLM is not a dynamic, learning brain. It is a fixed mathematical function defined by a massive table of numerical parameters. After these models are trained, the parameters never change. These models have layers, and each layer contains a matrix of numbers. When we send a message to an LLM, our text gets converted into tokens, which turn into vectors (embeddings). These vectors are fed into the first layer. The only real operation happening inside the model is linear algebra - primarily matrix multiplication. At each layer, the input vector is multiplied by a parameter matrix. (This is why GPUs are so important - and why NVIDIA’s stock is going through the roof. GPUs were designed to perform large matrix multiplications for 3D graphics so video games would look realistic. LLMs also require enormous volumes of matrix multiplications, and GPUs accelerate them dramatically.) The result becomes the input to the next layer. This process continues through all layers until the model produces the final output vector. A function is run on the output to convert the vector into a probability distribution over next possible tokens. The model selects the token with the highest probability. For example, for a prompt ending in “The sky is,” the model might produce “blue” (0.71), “clear” (0.15), “cloudy” (0.08), and so on. And in training, after the model is shown enormous amounts of text, you remove a token, ask the model to predict it, compute a loss, and adjust the parameters to make the prediction closer to the correct answer. You repeat this process billions of times. This process encodes associations, patterns, and heuristics into the parameters. Once training is finished and the model is deployed for use, the parameters are frozen. There is no active learning, no self-correction, and no adaptation in the models you use. These models are also not stored or run on a single physical machine. Thousands of chips each compute tiny parts of the model. One GPU might handle layer 42 for your query while simultaneously handling layer 42 for many other users. Much of this process is common to standard neural networks: taking inputs, turning them into vectors, passing them through layers of matrix multiplications, and training by adjusting parameters to reduce a loss. What is specific to modern large language models is the transformer architecture (a type of layer), the massive scale of training, and the final step of turning the output vector into a probability distribution over next tokens.

This is very different from consciousness. I’m not a neuroscientist or a cognitive scientist. I have a casual interest in consciousness and an armchair-level understanding from having read a few books on it. So how does consciousness work? The cheeky answer is we do not know. The real answer is that even the smartest experts still have not determined whether consciousness is emergent or fundamental. I wrote about this fault line almost three years ago in Bottom-Up vs Top-Down. Emergent means that physical processes in the brain, like neurons firing, can generate “what it feels like.” “I” arises from matter. Nothing more is happening. Consciousness is like wetness. It does not exist in the individual molecules but in their collective behavior. Fundamental means conscious experience is a basic feature of the universe, like mass or charge. There is also “weakly emergent,” which says consciousness probably emerges from the brain but we do not understand why or how in a satisfying way. And then there are more rigorous, mathematical attempts to explain consciousness, a famous one being Integrated Information Theory (IIT), a framework proposed by Giulio Tononi in 2004. In IIT, a key concept is how integrated something is. If you cut the system into pieces, does anything change? If the answer is no, the system is not integrated. The parts operate independently. If the answer is yes, the system is integrated. The parts depend on each other. IIT says consciousness is identical to how integrated a system is - it exists only when a system cannot be cleanly split into independent parts without losing something essential about how it behaves. Formalizing this, we have φ (phi), a numerical value that measures how integrated a system is. If you cut the system into pieces and nothing changes, φ is low. If something changes, φ is high. Consider two light bulbs in different parts of the house. Each light bulb is either on or off, and one’s state does not affect the other. In that case, the set of lightbulbs has a φ of zero. There is no consciousness. Marbles in a bag? No φ. Gears meshing together? Slightly higher φ. A dense spiderweb where every vibration affects the whole? Higher φ. A human being? Even higher φ. The controversial claim in IIT is that feed-forward systems are not integrated because they lack feedback loops and irreducible cause-effect structure. LLMs are just neural networks made up of layers that go one way. Input, layer 1, layer 2… output. No part depends on previous or future parts, which means LLMs have a φ of zero.

I don’t think LLMs are not conscious because of the IIT assessment, although the conclusion is quite satisfying. I don’t think LLMs are, or ever will be, conscious because we understand so little about consciousness. There are so many parts. Vision, planning, memory, learning, self-evaluation, motivation, and action, just to name a few. Each subsystem interacts dynamically while continuously monitoring the environment and simulating future possibilities. You see a snake. Memory triggers fear. Planning systems choose “jump back.” Motor systems execute it. We also have drives, values, and motivations that evolve over time. The closest analogy to what an LLM corresponds to in our consciousness is the language-processing region of the brain, like Broca’s area or Wernicke’s area. If you removed only these regions and placed them in isolation, they could still process linguistic patterns. They might recognize that “door” and “gate” are conceptually related. But this subsystem has no perception, no memory of the world, no goals, no actions. We would never assign consciousness to this isolated subsystem (we don’t even know if we can confidently ascribe consciousness to the human brain). LLMs operate at roughly this isolated level. Sophisticated at language, but only language. Never mind that LLMs remain static after training and only do one type of operation. LLMs have no internal world model. No planning, evaluation, or goals. The computation is linear, fixed, and predictable. Sometimes these models produce nondeterministic results because we inject randomness into a system that can generate trillions of possible future sequences. We have made something so complicated that it gives the appearance of nondeterminism. But if you fix four things - the model’s parameters, the input tokens, the sampling method, and the random seed - the model will always produce the same output.

We used to tell stories about things we couldn’t explain with the knowledge we had at the time. We should not make the same mistake with LLMs. We have the knowledge. We know exactly how they work. A fixed table of numbers is pushed through a series of matrix multiplications. There is no memory, no learning, no unified anything. Nothing special is happening under the hood. The model is not even in one place physically - it’s distributed across many machines, so different parts of your query are processed on different GPUs and stitched together. I do not know, and many smart people who have thought about consciousness their whole professional lives do not know, what it takes to create full consciousness. At best, LLMs capture only a small part of the picture. At worst, they are a stack of matrix multiplications that we have mistaken for a mind.