Information experience

This article details how both humans and AI systems turn information into interpretations and actions. It is a summary of a series of articles I wrote a few years ago, updated for the age of generative AI.

Let’s begin at the beginning:

The big bang at the beginning of time consisted of huge numbers of elementary particles, colliding at temperatures of billions of degrees. Each of these particles carried with it bits of information, and every time two particles bounced off each other, those bits were transformed and processed. The big bang was a bit bang. Starting from its very earliest moments, every piece of the universe was processing information. The universe computes.

This is our starting point, courtesy of quantum physicist Seth Lloyd. Everything computes, and information is everywhere, from the tiniest particles to the large-scale structure of the universe.

Around the same time that was written, across the Atlantic, Walter Van de Velde was approaching computational thinking from the other direction:

We explore the idea that the world that we live in can be viewed as a kind of computer that continually computes its future…. We want to apply the idea of computation to the inhabited world of the natural, biological, cultural, technical social system that we live in.

This is computer science without computers. It’s the universe as computer but at human scale.

How is the world-as-computer programmed? This need not be complex. Stephen Wolfram, after years of designing software, came to believe that simple programs can create great complexity, and proposed that something like such programs might underlie many natural and social systems. And now, led by scientists like Lloyd, information in physics is seen as being more fundamental than matter.

Specifically how this works, according to Lloyd, is as follows: “Every physical system registers information, and just by evolving in time, by doing its thing, it changes that information, transforms that information, or, if you like, processes that information.” Even something seemingly static like a rock, a coin, or a house is registering information simply by existing; because everything exists in time, it evolves (however slowly), and as it transforms, this creates information.

Out there, there is no light and no colour, there are only electromagnetic waves…there is no sound and no music, there are only periodic variations of the air pressure…there is no heat and no cold, there are only moving molecules with more or less mean kinetic energy.
— Heinz Von Foerster

What does it mean to talk about waves? Look to the sea, of course — waves roll in and out of the shore, oscillating in cycles. The rate at which they rise and fall, come and go varies, depending on location, the topography of the land beneath them, and environmental conditions — big waves are generally driven by high winds and weather conditions that may be some distance away. The waves tell us something about those distant conditions — they carry information.

We glean such information from the size and rate of the waves. The height of the waves (their amplitude) is a measure from the top of a cresting wave to the bottom of a trough, and the rate at which they come in (their frequency) is measured from when the top of one wave passes a certain point until the next one does.

At some level, then, information is related to oscillations — to change over time, difference and sameness. A sea of sameness, of flatness and consistency, contains little information. It’s therefore easy to describe. More information comes with change, and difference. It’s a measure of how difficult something is to describe. As the anthropologist Gregory Bateson said, a single unit of information is “a difference that makes a difference".

This is similar to what Claude Shannon said, in formulating his mathematical model of communication, known as information theory. His example was quite different, however — the alphabet. He looked at the frequency of each letter as it appears in the English language, so ‘E’ appearing most frequently, for example. Then, in a striking series of diagrammatic examples (I redrew one above), he reconstructed the language based on this frequency of occurrence — the letter ‘U,’ for example, appears most often after ‘Q’ and so on. Groups of two, then three letters, to predict words and sentences.

But when it comes to decoding information — finding the signal in the noise — we come up against individual and cultural differences in perception and interpretation. Something as simple as changing your rate of breathing can change your perceptual experience. So an important corollary to Bateson’s dictum that information is a difference that makes a difference is in differences between individual perceivers. What I perceive as difference, you might not. Or you might see something different — the colour red that you see is probably different from the one I do.

Going further, then, information is patterns of organisation, where meaning resides in the connections and diffractions between things. The physicist and philosopher Karen Barad eloquently appends Bateson’s dictum: information is in patterns of difference that make a difference. When you throw a stone in a pond, causing waves to ripple outward in all directions, you can look for information where and how these waves interact — some waves meet each other at the crest, some at the trough and everywhere in between, building on each other or cancelling each other out.

In a computer, bits of information are assigned to specific physical locations in memory. Von Foerster, quoted above, defines cognition as the computation of reality. The difference is that in a computer, a device based on numbers and built for calculation, these spaces of memory are numeric addresses, like apartments in a skyscraper. Humans, on the other hand, remember in images and emotions — just think of your most memorable moments.

There’s an even bigger difference. When a computer recalls something from memory, accuracy is important, grounded in Shannon’s approach to transmitting data from one place to another without loss: communication within and between computers relies on sending and receiving an accurate signal amidst any noise. Constructing bytes from bits, like Shannon’s growing sequences of letters, means recognizing patterns and checking for errors.

Humans also rely on a kind of pattern, which acts as a kind of carrier wave for knowledge, according to the computer scientist Philip Armour:

The ’strongest’ of these patterns are our most conscious and intentional thoughts — those that are strong enough to be accessible to and recognised by the ‘consciousness’ pattern. Our habits might also be strong patterns, though we may be quite unaware of them. Some patterns resemble other patterns and these similarities are themselves signals. Some signals are so weak they are almost gone. When they weaken further or are completely buried in other patterns they will be gone and we will have ‘forgotten.’ Patterns can be made stronger by continually revisiting them as happens when we practice playing a musical instrument. Patterns that are very similar to others may become conflated over time and memories merge.

But images, or any other pattern of information, are not stored in specific locations in the brain, as in computer memory. Neuroscientists and psychologists know that the act of recall always changes a memory. In Barad’s analogy of diffraction patterns, we can think of a single memory as a pebble in a pond, but we always read its ripples through others. Have you ever tried to retrieve a stone thrown into a pond? Even to see a stone thrown in a shallow pond, you look through rippling waters and among countless other stones.

What is a single memory anyway? To recall a single number or a name, we might fairly easily emulate a computer and recall without error. What about an image? We can think of it as a particular pattern, but the mind isn’t a topological landscape. In this sense it is closer to an AI model in which a memory might be encoded in a whole pattern of neural activity.

What is a “bit” or single element of information? We know the specific definition of this when it comes to computers: binary on/off states of logic gates or switches create bits, combine into bytes that contain eight bits, and so on. Mihaly Csikszentmihalyi (the psychologist best known for the concept of “flow”) applies this computational perspective to humans, estimating our attentional capacity at 126 bits per second. When we attend to one thing over another, this constitutes one bit of information, with an “attentional unit” equal to 1/18th of a second.

But we’re not computers. Human cognition is not the mere accumulation and processing of information in memory, and not everything can be broken down to individual “information elements.” Think of learning to ride a bicycle or drive a car — millions of tiny movements are unconsciously processed, recalled and recombined. The body plays a role here, and an important corollary is that some processing seems to take place outside the brain, in our arms and legs, and indeed in every cell of our bodies.

Evidence for this comes from an unlikely source: bugs. Scientists attached small rockets to cockroaches (!) and found that the insects can correct their movements very quickly — faster than it would take for a signal to go from leg to brain and back. That indicates a feedback mechanism located in the legs themselves. There is some evidence for something similar in humans, and for what is commonly called muscle memory.

These findings are now applied to computers — particularly for robots to quickly correct their movements to maintain stability. “Smart” objects have embedded sensors and processors to recognise and communicate with people, surroundings and other objects. This has been thought of as giving things “perceptual intelligence,” and there are both benefits and dangers of doing this.

With ever faster and smaller processors, sophisticated computation (even machine learning, which is closer to higher cognitive processing in the brain) can now take place at the edges of a network, not only in centralised servers—just as in cockroaches. I am doing machine learning, for example, on a microcontroller the size of a fingernail, in small robots.

With more and more robots, sensors, cameras and other devices connected to the internet, there’s a lot of information around. AI adds a huge amount more.

But contemporary AI systems are built to direct their attention to the most relevant bits, at each step in parsing a sequence of informational “tokens” attending to all the other tokens in parallel. Attention is all you need, goes the mantra that has now become foundational in Silicon Valley. Chatbots apply this directly to language in constructing responses to your queries, and this can be traced right back to Shannon’s information theory. That is, after all, where Claude got its name.

Humans, meanwhile, selectively perceive, process and store information, and just as in nature, selection implies some sort of competition between bits of information. Herbert Simon goes further to say that not only do we consume information, but it too consumes:

In an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.

Enter the online “attention economy”, where attention translates directly into monetary profit. A financial approach is premised on attracting “eyeballs” to content. Like many metaphors, this one has a basis in reality. Jonathan Crary links such commercial competition directly to biology:

Even as a contemporary colloquialism, the term “eyeballs” for the site of control repositions human vision as a motor activity that can be subjected to external direction or stimuli. The goal is to refine the capacity to localize the eye’s movement on or within highly targeted sites or points of interest. The eye is dislodged from the realm of optics and made into an intermediary element of a circuit whose end result is always a motor response of the body to electronic solicitation.

The eyeball is, after all, the only part of our brain that’s directly exposed to the world. So we could say that when we view information, more than when we hear, touch or taste it, we’re making a direct connection between mind and world.

We can apply this to the following insight from Van de Velde, as part of his world-as-computer thought experiment: “Attention is the oxygen of information. Without attention information is dead. With attention it influences action.”

Specifically, Van de Velde was interested in how “smart objects” could influence our behavior — by simply giving us advice: “Our idea is to view a piece of advice as entailing an expectation on future behavior, and to track user behavior in order to evaluate the effectiveness of the advice.” A computer can assign statistical probabilities to how you might act on such advice, and by tracking what you do in response, it could refine its predictive models.

At this point you will realize that systems like this are in place, in the form of Amazon recommendations, Google search results, location tracking on smartphones.

What does it mean to be “interested” in something? For Van de Velde, an interest is simply a resource for behavior, different from a goal or a task. Goals, according to Van de Velde, “are just means to an end, the end being behavior.” Interests, then, “do not need to be measurable except through the successfulness of the behaviors which they enable.” He continues:

we represent an interest as a data structure with a measurable satisfaction function defined on it. An interest represents the pursuit of a resource for behavior, but it leaves in the middle what that behavior is. Thus, an interest is neither a goal nor a task. It is not limited in time, neither is it something reachable.

Turning from interests to behaviors, Van de Velde then maps out possible behaviours in a topology if branching decisions (I redrew his diagram above). Recall Csikszentmihalyi’s definition of a decision as one bit of information. Applying such an idea to a whole “behavior landscape,” Van de Velde thereby sees the whole world as a computer — one that we program by influencing each individual, attentional decision. In so doing, it computes individual futures, and thereby our collective future.

When information is transmitted instantaneously, copied endlessly, and held everywhere at once but in no one’s memory, it inhabits a timeless space which distinguishes it from insight, says the philosopher Byung-Chul Han. There is thus a problem with information as a concept that characterises our age. Just as information can prompt action in people, from a philosophical perspective, we create reality every second through our very perception, as Von Foerster wrote.

We have to also accept that the very notion of information itself relies on its own shadow: the absence of information. Above, I discussed a lack of information where there is a lack of variation, but that is a difference in quantity, not quality. Here I am referring to the perception of something and the creation of meaning where no perceivable information exists. Call it apophenia if you like, but we have an innate drive to make sense of things, even where sensory information is absent.

Digital media explode this notion – we have more information than ever but we perceive it in even more deprived, disembodied ways. More information, but paradoxically, even though it can be transmitted with less loss – thanks to Shannon – the sheer quantity of information concentrated into the constrained space of the screen simply creates even more noise.

More and more systems creating more and more information, on top of the universe-as-computer of colliding particles continuously generating information, and the world-as-computer constantly creating branching futures. Attention (along with ever-sharper tools) is all you need to need to perceive it, but human insight is still needed to spot the differences – or the patterns of difference – that make a difference.

Want more? You can read my four-part series of articles here.