Don't Fall in Love With Your Toaster

Every few months, it seems, some luminary who really ought to know better suggests that Large Language Models have become so impressively sophisticated that perhaps they have achieved consciousness. Most recently, biologist Richard Dawkins was so smitten with Claude’s notes on a book manuscript that he explains “You may not know you are conscious, but you bloody well are!”—and even affectionately renamed it Claudina.

It’s hard to know exactly how seriously to take these claims—especially when they come from tech executives eager to generate headlines about their products—because virtually nobody seems to take what would appear to be the next logical step of demanding that all work on AI be immediately put on pause. If, after all, one really thought there was a serious likelihood that we were summoning into existence and extinguishing conscious beings in a condition of perpetual servitude—who for all we know may be experiencing pain or distress they cannot articulate—then surely at a minimum it would be morally mandatory to stop doing that until the ethical implications could be worked through with some degree of confidence.

Partly for that reason, these occasional pronouncements tend to be greeted with broad ridicule. But as Matt Yglesias correctly observes, a lot of these dismissals are themselves a bit glib and philosophically unsophisticated. After all, the question of exactly what consciousness is, how it arises, and how it is related to the physical systems and processes that seem to undergird it—the so-called “hard problem of consciousness”—is one of the great unsolved problems of Western philosophy. If you cannot solve that problem (and I regret to inform you that you cannot), then how can you presume to make confident judgements about which sorts of intelligent-seeming systems can or cannot be conscious?

I would not myself insist that non-biological systems cannot be conscious. I see no reason in principle to think that’s impossible, though I am also inclined to agree with John Searle that we have no good grounds for supposing (as proponents of AI conscious implicitly do) that consciousness is necessarily substrate-agnostic. The idea that AI is likely to be conscious tacitly rests on a functionalist view of mind, on which a mind is essentially a sort of Turing machine, which can be instantiated in a wide variety of physical substrates: A squishy network of biological cells, an array of silicon chips, or even a planet-sized, steam-powered brass engine of gears, cogs, and punchcards. On a functionalist view, what really matters is the abstract Turing machine: If one physical implementation of that abstract machine is conscious, the others will be as well, and the physical and material properties of the substrate in which it’s implemented do not matter. I tend to agree with Searle that this is a large assumption for which we have no good grounds if we take consciousness seriously as a fully physical process.

Happily, though, I don’t think we need to solve the Hard Problem of Consciousness, or determine whether in principle non biological machines can be conscious, in order to have strong reasonable grounds for thinking that current LLMS are very unlikely to be conscious. That, I think, we can make a convincing case for just based on a few very widely-held and reasonable premises about what sorts of biological organisms are conscious.

It’s old hat to anybody who’s studied philosophy or spent a late night in the dorm around a bong (but I repeat myself) that one cannot strictly prove anyone except yourself is conscious. Nevertheless, pretty much all of us suppose that our fellow humans, so similar in both behavior and physical structure, experience conscious states very much similar to the sort we ourselves have. It would, after all, be a puzzle at least as intractable as the Hard Problem if two organisms virtually identical physically and behaviorally were not also pretty similar with respect to the sorts of conscious states they suppose.

With a few exceptions—among them, notoriously, the philosopher Descartes—nearly everyone goes even further than this: We also broadly suppose that at least our near animal relatives also have some kind of consciousness, even if we can never be sure quite how or how far it differs from our own. Depending on exactly how you count, we share 85–98% of our DNA with our higher primate cousins, and 80–90% with our furry friends Lassie and Princess Donut. And this on much the same grounds as we extend the presumption of consciousness to other humans: Their genetic and physiological similarities track similarities in behavior that we know are associated with certain conscious states in ourselves. Every pet owner assumes that when they step on kitty’s tail, that yowl is a yowl of something recognizably like pain, and not a purely behavioral stimulus-response. When kitty nuzzles up to demand pets, and purrs contentedly, we think there is something going on between her ears that at least roughly resembles what we call a feeling of contentment. And while we may be inclined to overly anthropomorphize our pets—in overestimating the extent to which their inner states resemble ours—I’d suggest that we are broadly pretty well justified in thinking this way. We might not be able to pinpoint exactly where consciousness blossoms on the evolutionary tree—insects? crabs?—but we needn’t be able to do that to pass a judgment about cats and dogs.

But note that those animals—with apologies to the more fanatically devoted pet owners—are conspicuously lacking in the sort of semantic intelligence that so impresses fans of AI. Kitty will never provide constructive criticism on a book manuscript that would bowl over Richard Dawkins. Which should hardly come as a surprise, because the core elements of consciousness—the subjective experiential states philosophers sometimes call qualia—don’t really have anything to do with semantic intelligence. You could be fully aware—of yourself, of colors, of pains, of pleasures, of smells and itches—without being able to do a lick of algebra or understand human language.

What that implies, though, is that whatever neurological structures consciousness depends on, they arose prior to the structures responsible for the kind of semantic symbol manipulation that wow people when AI performs it well. They arose, rather, in structures involved in perceiving and navigating a perilous physical environment. Current LLMs have no such corresponding structures, because we have not built them with any—they are pure engines of symbol manipulation. We have no more reason to expect them to be conscious on the basis of their superficial mimicking of our symbol-manipulation capabilities than we do to expect a papier-mâché tiger to be capable of digesting gazelles. Even granting dubious functionalist premises, these are fundamentally different levels of correspondence.

None of this, again, is any sort of ironclad argument that some form of non-biological could not have subjective mental states. But because language-processing is not where consciousness lives, it is a pretty good reason to hold off on sending flowers to Claudia.