John Searle’s “room that speaks but does not understand Chinese” feels powerful and attractive only because it hides the ball. Make the “room” the size it would actually have to be, and the intuition reverses. This is what Scott Aaronson says, and I believe he is right”
Thinking about “AI”, I find my mind once again going—in almost a reflex, like a low-order Markov process iterating on a fixed function—to Scott Aaronson’s argument that our instincts betray us when you go full reductionist and start saying that things are “just…”
His argument comes in Quantum Computing since Democritus and elsewhere. It is a rebuttal to John Searle’s image: an apparently Chinese-speaking room that, in fact, does not understand Chinese.
Aaronson’s is a “complexity” response: Searle’s image gets its force from a misleading picture of a small rule book, or a few shelves of books that together constitute a lookup table. But consider not a bookshelf but a datastore Earth-sized, and searched by tens of thousands of near-light-speed robots. The sheer scale of the computation makes it much more than plausible that you might see such a 电脑, a diànnǎo, as plausibly understanding Chinese.
Thus after a certain point complexity gains metaphysical significance: it is no longer “just…”, but rather something for which, while the reductionist perspective is still true, the emergent properties of the complex are so weighty that the reductionist perspective misses what any sensible observer would see as the real point.
Here, for reference, is the argument as Scott makes it:
<https://www.scottaaronson.com/> <https://www.cambridge.org/core/books/quantum-computing-since-democritus/197A4CD13738E10AAD787DBB78D8E92C>.
In the last 60 years, have there been any new insights about the Turing Test itself? In my opinion, not many. There has, on the other hand, been a famous “attempted” insight, which is called Searle’s Chinese Room. This was put forward around 1980, as an argument that even a computer that did pass the Turing Test wouldn’t be intelligent.
The way it goes is, let’s say you don’t speak Chinese. You sit in a room, and someone passes you paper slips through a hole in the wall with questions written in Chinese, and you’re able to answer the questions (again in Chinese) just by consulting a rule book. In this case, you might be carrying out an intelligent Chinese conversation, yet by assumption, you don’t understand a word of Chinese! Therefore, symbol-manipulation can’t produce understanding.
So, how might a strong AI proponent respond to this argument?
Well, she might say: you might not understand Chinese, but the rule book does! Or if you like, understanding Chinese is an emergent property of the system consisting of you and the rule book, in the same sense that understanding English is an emergent property of the neurons in your brain. Searle’s response to that is, fine, just memorize the rule book! Then there’s no “system” other than your own brain, but you still don’t “understand” Chinese.
To which the AI proponent shoots back: there is too another “system” in this case! Supposing you memorized the rule book, we’d need to distinguish between the “original” you and the new, simulated being brought into existence by your following of the memorized rules – a being whose only relation to you might be that it happens to inhabit the same skull.
That response might sound crazy, but only to someone who’s never studied computer science. To a computer scientist, it seems perfectly reasonable to say that one computation (say, a LISP interpreter) can conjure into existence a different, unrelated computation (say, a spaceship game) just by dutifully executing rules.
Look, as I’ll discuss later, I don’t know whether the conclusion of the Chinese Room argument is true or false. I don’t know what conditions are necessary or sufficient for a physical system to “understand” Chinese – and neither, I think, does Searle, or anyone else. But considered as an argument, there are several aspects of the Chinese Room that have always annoyed me:
One of them is the unselfconscious appeal to intuition – “it’s just a rule book, for crying out loud!” – on precisely the sort of question where we should expect our intuitions to be least reliable.
A second is the double standard: the idea that a bundle of nerve cells can understand Chinese is taken as, not merely obvious, but so unproblematic that it doesn’t even raise thequestion of why a rule book couldn’t understand Chinese as well.
The third thing that annoys me about the Chinese Room argument is the way it gets so much mileage from a possibly misleading choice of imagery, or, one might say, by trying to sidestep the entire issue of computational complexity purely through clever framing. We’re invited to imagine someone pushing around slips of paper with zero understanding or insight – much like the doofus freshmen who write
(a + b)2 = a2 + b2
on their math tests. But how many slips of paper are we talking about?
How big would the rule book have to be, and how quickly would you have to consult it, to carry out an intelligent Chinese conversation in anything resembling real time? If each page of the rule book corresponded to one neuron of a native speaker’s brain, then probably we’d be talking about a “rule book” at least the size of the Earth, its pages searchable by a swarm of robots traveling at close to the speed of light. When you put it that way, maybe it’s not so hard to imagine that this enormous Chinese-speaking entity that we’ve brought into being might have something we’d be prepared to call understanding or insight.
Of course, everyone who talks about this stuff is really tiptoeing around the question of consciousness. See, consciousness has this weird dual property that, on the one hand, it’s arguably the most mysterious thing we know about, and on the other hand, not only are we directly aware of it, but in some sense it’s the only thing we’re directly aware of. You know, cogito ergo sum and all that. So, to give an example, I might be mistaken about my shirt being blue – I might be hallucinating or whatever – but I really can’t be mistaken about my perceiving it as blue. (Or if I can, then we get an infinite regress.)….
Many people’s “antirobot animus” is probably a combination of two ingredients: (1) the directly experienced certainty that they’re conscious – that they perceive colors, sounds, positive integers, etc., regardless of whether anyone else does; and (2) the belief that, if they were just a computation, then they could not be conscious in this way…. For people who think this way (as even I do, in certain moods), granting consciousness to a robot seems strangely equivalent to denying that one is conscious oneself.
Is there any respectable way out of this dilemma – or in other words, any way out that doesn’t rely on a meatist double standard, with one rule for ourselves and a different rule for robots?
My own favorite… is one… advocated by… David Chalmers… a reduction of one mystery to another…. If computers someday become able to emulate humans in every observable respect, then we’ll be compelled to regard them as conscious, for exactly the same reasons we regard other people as conscious. And… we’ll understand… [how] just as well or as poorly as we understand how a bundle of neurons could be conscious. Yes, it’s mysterious, but the one mystery doesn’t seem so different from the other…
<https://www.scottaaronson.com/>
Aaronson, Scott. 2013. Quantum Computing Since Democritus. Cambridge: Cambridge University Press. <https://www.cambridge.org/core/books/quantum-computing-since-democritus/197A4CD13738E10AAD787DBB78D8E92C>.
Brad DeLong back: I am thinking about this right now, again. I am wondering at what point will it no longer make sense for me to approach pretty much all of the questions about the current AI boom using my base frame. It is, as I am sure you know by now, this: MMLMs are stochastic parrots, super-autocomplete on steroids plus Clever Hans at scale and speed, pantomiming the thoughts and words and actions of the human who had the most-similar conversation. I do keep clinging to it.
But at what point must this be abandoned for a frame that centers higher-order more emergent properties?
What makes quantity acquire a quality of its own?
Where on the scale from a room to a planet-sized datastore, and from access by a single person pulling a book off a shelf and paging through it to access by tens of thousands of near-lightspeed robots, does the frame that is useful for thinking shift?
We know it does shift after all. We are here. We are the products of 300 million years of that process of variation and selection and scaling that is the evolution of the mammalian brain. (Or rather, we know that this is possible to the extent that we ourselves are much more than just stochastic parrots with delusions of grandeur, and maybe we are barely more.) But we are not there yet.
And I do not see signs that we are close: Our machines do vastly exceed our cognition and calculating capabilities in a great many areas, and a growing set of areas, the first of which is that I have to think hard for five seconds to calculate that 93 x 93 = (100 - 7) x (100 - 7) = (10 x 10 - 7) x (10 x 10 - 7) = (10 x 10) x (10 x (10) - (2 x 10 x 7) + (7 x 7) = 10,000 - 1400 + 49 = 8600 + 49 = 8649.
But AI-boosters talk about a “jagged frontier”, which is venture-capitalist speak for it really doesn’t work very well if we stop watching it like a hawk and instantly correcting it where it goes obviously and stupidly wrong.
Looking back at what I have been writing recently, I see that right now I am still holding to:
“It is, I think, still far on the John Searle [room-sized] side of the “Chinese Room” divide and not on the Scott Aaronson [planet-sized] side…. This is only autocomplete-on-steroids…” <https://braddelong.substack.com/p/mondays-with-the-machine-the-tongue>
“Our current frontier LLMs are much too simple…. You can see the seams in the cardboard facing the street that is the façade of this particular Potemkin Village, if you look…” <https://braddelong.substack.com/p/digital-gods-cardboard-brains-and>
“At what point does rote… manipulation of symbols… turn into real thinking?… I see absolutely no signs anywhere in these systems that they are close…” <https://braddelong.substack.com/p/another-brief-note-on-the-flexible>
“Can machine learning models soon transcend their s***poster roots?…One must be skeptical…. [I see no] ASI—an artificial super-intelligence—but rather… that [our models and] we are all already embedded in the SSNASI—species-spanning natural anthology super-intelligence—that is the Mind of Humanity today…” <https://braddelong.substack.com/p/reflexes-desires-actions-emergence>
(Plus there is: <https://braddelong.substack.com/p/agentic-ai-is-a-bonfire-of-the-tokens>.)
I want to avoid becoming a goalpost-mover and thus a dead-ender. I want to, when it is appropriate, adopt the frame: “these things are thinking, but not like we do”. I want to switch frames the moment that the switch advances understanding. But I also do not want to switch prematurely. That would turn me into just another gullible moron helping reinforce the forces that I see as net negative now. Forces leading a lot of people to make stupid decisions right now and enabling yet more grifters to FIW3TAI—Failed in Web3, Try AI!”
What will be the signs and wonders? There is a still small voice in my brain that tells me: twenty-five years ago you would have said that full natural-language fluency would be enough. What do I tell it in response?
I am genuinely flummoxed. Suggestions for tripwires greatly appreciated.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.