The Lost in the Middle Problem in LLMs

Five new blogs this week, covering two ways to shape how AI agents work, a silent failure that breaks long-context systems, the hidden signal a model leaves in its text, and the 2017 paper that started RLHF.

Let’s get started.

Graph Engineering is the practice of designing an AI system as a graph, where every step of the work is a node and every path from one step to another step is an edge.

In simple words, instead of writing one huge instruction and hoping that the model does everything correctly in one go, we break the work into small steps, and then we clearly decide which step runs after which step. Every graph has only three parts: a node is one step of work, an edge is the path to the next step, and the state is the shared notebook that travels along.

This is the heart of it. We let the model do the thinking, and we keep the control of the flow with ourselves. The model still does the thinking. The graph decides where that thinking goes.

Read here: https://outcomeschool.com/blog/what-is-graph-engineering

Loop Engineering is the practice of designing the repeating cycle that an AI agent runs, so that the agent keeps making real progress on a task and stops at the right moment with the right result.

A modern AI agent does not answer in one shot. It thinks, it does something, it looks at what happened, and then it thinks again. One turn of the loop is think, act, observe, and it repeats until the work is done. The model itself does not loop. The loop is code that we write around the model, and that is why we can engineer it.

A strong model inside a badly designed loop will still fail. A modest model inside a well designed loop will often finish the job. As agents take on longer work, the quality of the loop, and not the cleverness of the prompt, decides whether the work actually gets finished.

Read here: https://outcomeschool.com/blog/what-is-loop-engineering

The Lost in the Middle problem is the behaviour where an LLM pays strong attention to the information placed at the beginning and at the end of a long input, and pays very less attention to the information placed in the middle.

In simple words, the model reads the start, the model reads the end, and the middle just slips away. If we move one small fact from the beginning to the middle to the end and measure the accuracy at each spot, the line goes down and then comes back up, like the letter U.

What makes this dangerous is that it fails silently. There is no error and no warning. The model answers confidently with a wrong or incomplete answer, and we have no idea that the correct answer was already sitting right in front of it, just in the middle.

Read here: https://outcomeschool.com/blog/lost-in-the-middle-problem-in-llms

LLM watermarking means putting a hidden mark inside the text that an LLM writes. Text has no paper and no light, so the mark is hidden inside the choice of words.

An LLM writes one token at a time, and at most steps many different tokens would give perfectly good text. This freedom is where the watermark lives. A secret key acts as a seed: at every step, the key plus the current context splits the vocabulary into preferred tokens and other tokens, and the preferred tokens get a small boost in their probability.

One token gets a tiny change, so nothing looks unusual. But across thousands of tokens, far more preferred tokens appear than chance allows, and that statistical pattern is the watermark. A detector with the same secret key can rebuild the preferred set at every step and check for it later.

Read here: https://outcomeschool.com/blog/how-does-llm-watermarking-work

The paper showed how to teach a machine a goal without ever writing down a reward. Instead of hand-coding what “good” means, we show a human two short clips of the agent’s behavior and let them pick the better one. From thousands of these simple choices, the machine learns what we want.

For many real tasks we cannot write a good reward function. How do we write a number for “do a backflip”? So instead, a reward model learns from the human comparisons, and ordinary reinforcement learning trains the agent to chase that learned reward. A simulated robot learned to do a backflip from only about 900 human comparisons.

This is the origin of RLHF. The recipe never changed: collect human comparisons, train a reward model, optimize a policy with reinforcement learning. The only difference later was that game clips became pairs of text answers, and that recipe gave us InstructGPT and ChatGPT.

Read here: https://outcomeschool.com/blog/decoding-deep-rl-from-human-preferences

That’s it for now.