AI for Human Insight (Part 1)

In my previous post I asked the questions: how can we measure scientific insight? Can we train an AI agent to be a research assistant in the sense that it leads the human to more insight about complex systems? So in this post I’ll elaborate on what I mean by insight and slowly shape these questions into something we can attack with existing tools.

The first step is to define “insight.” A reasonable definition of insight is that it’s a thought or embodied understanding that is useful for explaining observations. When you have insight into something you have a model for how and why it works the way it does. You have a sense of the ingredients that go into a process and how they interact to form an observable output. Here are some other words that orbit around my definition of insight: theory, hypothesis, explanation or conjecture.

You may acquire insight about anything you can observe. Insight can be on everyday things like what kind of cooking techniques result in tastier food, e.g., when to add salt to your scrambled eggs. Or it may be about more complex topics like the fundamental relationship between time and space, e.g., Einstein’s theory of relativity.

Insight may be explicit or implicit. When it’s explicit, it can be described by a set of instructions. A recipe book has a lot of explicit insight about cooking. When it’s implicit, it’s harder to pin down with language and is often referred to as “feel”, “talent”, “taste”, or tacit knowledge. A grandmother has a lot of implicit insight about cooking.

Acquiring insight is the process of observing the world, forming an explanation for what you see, testing the explanation with experiments, and adjusting your explanation. This process is often called “science”, although in practice most “scientists” are doing something else. Here’s an example of science in progress:

1. Observation: my eggs tasted better today than yesterday. I wonder why.

2. Model: salt added early breaks down egg proteins, so curds set looser and stay tender. Salt added late tightens proteins, so eggs turn rubbery. Tender eggs taste better.

3. Experiment: blind taste test with friends. Salt before, during, and after cooking. See what they prefer.

The academic study of the methods and implications of science is known as philosophy of science. It has a vast and incredibly interesting body of literature which I will not be getting into in this post. Instead, I’ll dig into a narrow application of it: can we design AI agents to explicitly help humans develop insight about systems the humans interact with? By “system” I mean a group of things working together as a whole to produce observable outputs. There are relatively simple systems (e.g., the taste of your breakfast scrambled eggs as a function of when you apply salt to them) and more complex systems (e.g., the fundamental relationship between time and space). Note that AI agents often only have access to the systems through humans. In fact, these systems often only exist because of humans, e.g., psychology, geopolitics, etc.

Now, you could say: “Fernando, isn’t this exactly how AI agents are trained now?” Well, yes and no. Techniques like reinforcement learning from human feedback result in agents that are decent research assistants for humans. This is often measured by datasets of what “good” assistance looks like or by human-generated grading rubrics. However, the training procedure never explicitly optimizes for helping humans make sense of nebulous phenomena that are often harder to communicate compared to simple instructions like “teach me calculus.”

For example, in 1948 Claude Shannon published “A Mathematical Theory of Communication”, in which he laid the foundations for the field of information theory. Information theory is built on existing formal tools such as probability and calculus. However, the need to formalize “communication” and “information” could only have come from a human: indeed, it is humans who are doing the “communicating” and it is humans who are processing “information.” Without us, the concept of a “mathematical theory of communication” does not make sense. In fact, neither does “mathematical theory”!

If Shannon had access to an AI agent when he wrote his paper, he could’ve saved himself much of the work found in his paper. However, he would not have saved himself the work of making sense of the world and deciding that “information” was something worth modeling. This reminds me of this post:

kache@yacineMTB

do not think that you can avoid doing this. you need to understand. you can outsource thinking, writing code, but you can't outsource understanding. to know where you can go, you must know where you are

1:53 AM · Mar 25, 2026 · 13.3K Views

9 Replies · 13 Reposts · 434 Likes

So even though it is true that existing agents often indirectly help us make sense of complex systems by, for example, writing/reading/summarizing complex code, they are not trained for the explicit goal of eliciting this understanding in us. This is what I’m interested in.

So a natural next question is: “why hasn’t this been done before?” One reason is that concepts like “human insight” are very nebulous, and measuring and optimizing for nebulous objectives is not easy. It is much easier to measure how well an AI agent can reproduce text on the internet, than it is to measure whether a human has gained any real understanding of a complex system. This is especially true when the agent doesn’t understand or even have access to the system in question, which is likely to be the case when a human is working on a cutting edge topic.

Another reason why we currently don’t train our agents for human insight is that we haven’t really tried to do it before. This is not meant as a diss on AI researchers: we had to start somewhere, and that happened to be methods that modeled how humans wrote on the internet1. However, now that we have all this infrastructure and knowledge on AI, I think it’s worth exploring whether we can build such agents. As Richard Hamming put it, “It’s not the consequence that makes a problem important, it is that you have a reasonable attack.” So in Part 2, I’ll outline the existing AI scaling paradigm and propose a first “attack” on building AI for human insight.

Post 33/100