Intelligence as Compression

I’ve been thinking about revising my piece on intelligence for a long while, but I’ve decided finally to opt not to replace it wholesale and instead write about things I wish to update as they come. This particular piece is focused on intelligence, as prior defined, and is a stronger and more versatile construction of the idea as I’ve been using it.

Introduction

Intelligence can be construed as compression. This view is not novel, and I’ve been seeing bits and pieces of this in the context of LLMs and how those systems work, but I set that topic aside and will speak to intelligence as it occurs in people, and as I’ve already written about some years ago. The definition I had placed on intelligence then is repeated here:

Intelligence is the ability to collate information: to be intelligent is to be able to learn.

and another pertinent quote is:

A key part of intelligence is synthesis—the conversion of raw, abstract knowledge into something usable by the self.

whence we get: Intelligent people (de)compress information efficiently.

Compression

Raw information is hard to work with. One typically does not work with the full, detailed accounts of what they know, and instead deal with abstractions—models, in a sense—which sufficiently capture the key bits of a piece of information, and are “smaller” in some sense than the raw data. Compression is the reduction of these details into heuristics, of fitting a model to these data. Thus, compression can be constructed as effectively and efficiently collapsing data into a simpler representation.

It is important to recognize the utility and importance of compression. It is hard to coalesce opinions and make connections and conclusions across vast swaths of information (e.g. information from papers or books) due to noisy disparities, such as presentation format, language, contexts, etc. One’s mental model serves as an abstraction, and by way of abstraction that baggage (information) is shed, thus allowing connections to be more easily be made in this latent space, where information is highly (lossily) compressed. This latent space is very useful for you, but it is predicated on its innateness—i.e., it is not portable. I’ve written about a very similar idea in On Minimality. You use this for drawing conclusions, ascertaining relationships, finding patterns, etc., but you can’t share these latent representations.

Decompression

The natural parallel to compression then is decompression. Decompression in the modelling setting is like a posterior predictive distributions—you’re using your model to generated data, sending your low rank approximation back up to the full size data. Decompression is important because this is how you communicate results; how you use your latent knowledge. We interact with decompressed data (as seen in the compression section), so correctly decompressing data is important. In a less technical sense, decompression is akin to implementing ideas (latent information).

Intelligence

So how exactly does intelligence fit in? If we say that we can measure information in bits (abusing concepts from information theory), we can get a clearer idea. Let us say that you read something which has 100 bits of information. A more intelligent person could compress this to a smaller size, i.e. to something which encodes for most of those 100 bits but takes far less space than those 100 bits, say, 10 bits. We can measure the “accuracy” so to speak of these 10 bits by seeing how much of the original information we recover when we throw the 10 bits out of the latent space back into the true space (e.g., thinking about what we’ve read, then writing what we think). Let us say we get 80 bits of true information back. For the sake of the example, let us assume that we don’t make mistakes and instead simply drop information—i.e. we have 80 bits of veritable information without any noise reducing the effective size to something lesser.

Intelligence then is a measure of how efficient our compression is (size of latent representation, here 10 bits—as well as time taken to compress, though one may argue that is more about speed), and how effective it is (what proportion of original information have we recovered?) Here, . There is no absolute measure of intelligence, of course, and this serves solely as a framework to view intelligence as.

Conclusion

I find this view of intelligence very satisfying because it begins to unify some ideas I’ve been having, and neatly explains what synthesis really means—I never liked sweeping that definition under the rug as I felt that it held the heart of what I mean when I said “intelligence”, so I think this piece serves as a worthwhile extension.

There is a far more formal definition which relies more on modelling and statistics, but I removed it due to lack of appeal and general explanatory power.