The information content of sensory experience

In my previous post, I explained how fundamental scientific discoveries are often based on intuitions derived from our rich stream of sensory experience. Senses like touch, smell, and hearing. And since current AI models are mostly trained on images and text only, they are at an inherent disadvantage when forming novel explanations of the universe. For example, Einstein’s famous elevator thought experiment which eventually led him to the theory of general relativity, was rooted in what gravity felt like to him.

But what exactly is our advantage here? If it’s really a matter of more/better data, perhaps we can quantify it. So in this post I’ll estimate the information content of one small slice of sensory experience and then explore how it relates to our ability to make fundamental discoveries about our universe.

For simplicity, I’m going to focus only on the senses alluded to in Einstein’s elevator thought experiment. As a reminder, here it is:

Consider a scientist inside an elevator with no windows, floating in deep space. If we place the elevator on a planet, like Earth, the scientist will feel the pull of gravity and see any floating objects around her fall to the ground. If instead we strap rockets onto the elevator and accelerate it, from the scientist’s point of view, the effect will be identical. She and any floating objects will be pulled toward the elevator’s floor.

The scientist in the experiment felt gravity/acceleration using the following senses:

• Touch. The ground pushing up against the scientist’s feet.

• Vision. The ball falling to the ground.

• Proprioception. Forces in muscles, tendons, and joints as the scientist stabilizes herself against the acceleration.

• Interoception. Internal body sensations like organs stretching or moving. You can feel this if you take a deep belly breath, or on a rollercoaster ride.

• Vestibular perception. The sense of acceleration and rotation from the inner ear.

To simplify the problem even more, I’m going to ignore everything except touch and focus on the feeling of the elevator floor pushing up into the scientist’s feet. This feeling starts when tactile receptors in the bottom of her feet are deformed by the floor, causing sensory neurons to fire electrical impulses. These impulses travel along nerves to the spinal cord and then into the brain. There are many kinds of tactile receptors, each tuned to respond to the different ways skin deforms, e.g., pressure, stretching, light touches, sustained touches, etc. For simplicity I will not distinguish between them.

According to this paper, there are around 4000 tactile sensory neurons per foot. So that’s 8000 per person. Then, we can model the sensory data produced by the feet as a time-varying vector where each element corresponds to the voltage of a sensory neuron. That is,

This signal is continuous in time, which means the vector exists at every instant. In practice, we discretize the signal by recording it at specific times. When we discretize the signal, we represent each element as either 1 or 0, indicating whether that neuron fired during the corresponding time interval. For instance, we might choose to record the neuron values once every millisecond, or 1000 times a second. This corresponds to a sampling rate of 1 kHz, and according to this paper, is sufficient to capture tactile information.

Next I’ll estimate the amount of data we would obtain by running Einstein’s elevator thought experiment. Each element is 1 bit, implying 8000 bits per sample. If we sample at 1 kHz we record 8 million bits per second. And if we assume the elevator experiment takes 10 seconds, the full experiment is 80 million bits, which is around 10 megabytes (MB). That’s about one song worth of data.

Ok, great, now we know how much data we have, but we still need to quantify how informative it is. 10 MB of meaningless noise doesn’t tell us nearly as much as 10 MB of tactile data from the scientist’s feet in an accelerating elevator. One way of answering this question is to leverage information theory and calculate the information content of every element, which is given by1

where P(x_i) is the probability of observing x_i.

Without getting too technical, you can think of this notion of information as capturing surprisal: an informative observation is one that surprises us and challenges our prior understanding of the system that generated it. For example, if we were expecting a neuron to spike (i.e., we assigned a high probability to that event), and then it doesn’t fire, then we have a very surprising and informative observation. And we can use it to form a new theory for how the system works.

But what Einstein did was slightly different. He did not take a look at sensory data and conclude that it was surprising or inconsistent. In fact, the reason his thought experiment was so useful was precisely because in both elevators the sensory experience was exactly the same. His genius was in coming up with the right experiment, one that exposed an inconsistency in the existing paradigm, i.e., Newtonian physics would not have predicted acceleration and gravity to be the same.

This creativity does not seem to be a function of simply ingesting more data. Yes, if models have access to richer data streams they might get better at predicting more physical phenomena. But I don’t see how that leads to designing experiments that are better at teasing out inconsistencies in our current understanding of the universe. I sense that this might be especially true of nebulous research topics such as consciousness or AI alignment.

One could argue that this creativity is implicit in the text that AIs are currently trained on, and that with enough examples they may learn how to generate creative experiments of their own. I’m not sure! This is all very mysterious to me. Maybe in a future post I’ll dig deeper into Einstein’s creative leap, and its relationship to the rich sensory data it was based on. As always, thanks for reading!

Post 40 out of 100