The Blindsight Problem

Peter Watts published Blindsight in 2006. It’s a first contact novel where humanity encounters an alien intelligence that is sophisticated, dangerous, and possibly (read: totally) unconscious. The alien entity that the protagonists come across calls itself “Rorschach.” It’s not malevolent but simply optimizes its actions based on some unknown logic. It processes information at super-human speed, without ever being conscious of its own existence. Kind of like an LLM.

Anyhow, the book is a wild ride. The crew investigating the aliens includes a vampire brought back, Jurassic Park-style, from extinction, a linguist who has been lobotomized to reduce her emotional overhead, and a protagonist who had half his brain surgically removed as a child and now functions as a human camera, that is, observing everything, understanding almost nothing. It’s a strange book, and more than a little unsettling.

Watts uses three related concepts to illustrate the relationship between consciousness and action. The first is the Necker cube — that optical illusion where the same image flips between two orientations depending on how your brain frames it. Your brain flips between two orientations because your neural architecture can support both interpretations. Meaning that at a perceptual level, your brain has sufficient plasticity to choose what to emphasize and how to interpret it, even if what is being observed is purely visual and devoid of linguistic content.

Source: https://en.wikipedia.org/wiki/Necker_cube

The second is the Rorschach test — you see what your cognitive apparatus is built to see. In the book, the alien entity called Rorschach (and the protagonists, who are wildly different in their perceptions) does not perceive the universe directly, rather it sees a model of the universe filtered through its own architecture. In this sense, all observations are theory-laden, i.e., we don’t see the world as it is; we see it how we are.

Source: https://en.wikipedia.org/wiki/Rorschach_test

The third concept, that ties the main point of the book together with the Necker cube and the Rorschach ink blot, is John Searle’s Chinese Room thought experiment. It goes like this: a person in a locked room receives written questions in Chinese. They follow a rulebook to manipulate the symbols. They slide written answers back out in Chinese. From outside, the room looks like it’s having a conversation in Chinese. But the person inside understands nothing. They’re just following rules.

What if competence and consciousness are completely decoupled? What if a system can be extraordinarily intelligent, solve problems humans can’t, and make decisions that shape the world without understanding anything at all? Metaphorically speaking—how can a blind apparatus see? Watts wrote this before the current LLM era. He was asking fundamental questions about intelligence and consciousness. I think it’s one of the coolest sci-fi books of the 21^st century for exactly that reason (along with its motley crew of characters).

How this applies to LLMs

LLMs are Chinese Rooms. They don’t understand language the way you do. They understand statistical patterns in how tokens relate to each other. Feed them a prompt, and they execute a massive pattern-matching operation. The output looks fluent, coherent, sometimes brilliant. But there’s no inner experience happening, no consciousness.

Now, why does this matter for using LLMs for practical purposes, rather than just as a philosophical musing? It matters because LLMs can fail in ways that seems stunningly intelligent up to the point where they fall apart in ways that humans typically don’t. LLMs produce confident-sounding nonsense about domains they haven’t seen before. They hallucinate citations, invent facts, reason circularly, and they sound authoritative while doing it. What we need to pay attention to is how the built in biases of an LLM gets reproduced in our own work.

My co-author Natalia Levina at NYU and her colleagues showed this in their recent Harvard Business Review piece Researchers Asked LLMs for Strategic Advice. They Got ‘Trendslop’ in Return. Across thousands of simulations, LLMs almost uniformly selected the same trendy strategies regardless of context. Ask a model about organizational change and you get change management platitudes. Ask it about strategy and you get whatever showed up as trendy in the training data.

Because the LLM, at a surface level, is so convincing, humans tend to defer judgment to the LLM. This seems to be leading to a collapse in creativity. For example, an analysis of thousands of college essays published in New York Times showed that AI-written college essays were much less creative than essays written by humans.

Isomorphism

These characteristics of LLMs seems to push us into the biases of the training data, making us less creative and more narrow-minded. Therefore, LLMs may not be the boon to creativity that we sometimes have thought them to be. Rather, they may reduce creativity and lead to different forms of isomorphism, stemming from either the context or the model of an LLM-based tool:

Context isomorphism happens when a human issues a specific prompt or provides some particular context, and the model becomes trapped in that narrow context. You ask an LLM a question in a particular way, and it locks into that framing so completely that it cannot generate variance or alternative perspectives. The model becomes isomorphic to the context. Humans do this too, and indeed, experts may be more likely than others to get cognitively entrenched.

Model isomorphism happens because the model cannot break free of its training distribution. It parrots back what was dominant in the data it learned from. If the training data is full of trendy strategy advice, the model output will produce trendy strategy advice. If the data skews toward certain framings of a problem, those framings become the model’s world.

What to do about this?

The organizational theorist Karl Weick (another of my favorites) described good thinking as imagination disciplined by evolutionary processes. First, you generate variation: lots of different conjectures, perspectives, framings. Then, you apply selection: test them against rigorous criteria. Last, you retain what survives. Weick called this disciplined imagination, and it provides a roadmap for how we may need to engage with LLMs to retain our creativity. I think, however, that this needs to happen on at least three different levels: human practices, toolsets, and institutional guardrails.

Human practices mean development processes that actively emphasize creativity while ensuring trustworthiness. When you’re prompting an LLM or designing a system around one, the practice may be to generate multiple framings of the problem, test them against your actual constraints and values, and retain the approaches that survive scrutiny. I’m working with collaborators across the information systems research community to understand what these practices look like operationally.

Toolsets mean infrastructure that guards against context and model isomorphism by design. My students at Stevens are building harnesses that force variance into creative processes, such as systems that require you to generate multiple outputs and compare them, infrastructure that makes the training distribution and the active context visible so you can see what frame you’re operating within.

Institutional practices mean organizational checks that catch these failure modes before they ship. I’m working with various partner companies to understand what guardrails look like in practice, and how organizations can put such guardrails in place. A single engineer with a good tool can catch some problems, but context and model isomorphism usually do not lead to outright breakdowns, but rather the subtle diminishing of creativity and introduction of unproductive biases. Therefore, we need to start creating frameworks such as NIST’s AI risk management framework, but for creativity.

There is a lot of work to be done in each of these areas, but my sense is that we are now moving from the “wow, look at what this model can do”-era, and towards a phase where answering the question of “where is the value?” becomes more central. Let’s remember that LLMs are Chinese Rooms engaging in blindsight, and that human taste and judgment are crucial capacities for us to develop across our educational systems, and to nurture throughout our lives.