All we need is a *Modular* LLM Agent

LLMs are trained to be large monolithic systems traditionally. Trillions of tokens, billions of parameters, layered across thousands of GPUs, engineering at scale. In spite of their mammoth scale, LLMs are surprisingly non-modular.

• The trillion-token data is compressed, memorized into the numeric parameters. All of these memories with knowledge and instructions are entangled together.

• The core model is enveloped by a dynamic context layer. It holds the system instructions, interaction memory, tool definitions, contextual data and so on. This enables defining practical, task-executing, Agents via descriptive natural language instructions and contextual data.

• A nested loop takes care of the responses at runtime.

  • The outer execution loop (the harness)combines the system instructions and dynamic context into a sequence of tokens and presents them to the LLM to output the model response.

  • The inner loop iterates over the presented tokens in an auto-regressive manner and predicts the next token continuously, until the END/STOP token is generated.

System design demands modularity. It is hard to build large, reliable systems without introducing modularity. Apparently, LLMs defy this principle!

• All the world knowledge compressed implicitly into billions of parameters. No indices. Updates for one task can tamper with other learned task parameters arbitrarily, and so on.

• Behavioral instructions or system prompts can extend to several pages in natural language, with overlapping / conflicting instructions spread all across the prompt.

• Tools help interact with the structured world. All (hundreds of) tools definitions (related or otherwise) are mashed together in the context presented to the LLM.

In spite of their huge scale, the best LLMs are surprisingly non-modular.

There are several negative side effects of this — including reduced flexibility in handling knowledge, unpredictability (such as incorrect tool usage or ignored instructions), and, more broadly, the difficulty of steering LLMs in a fully deterministic way.

Contrast this non-modular LLM computation model with the time-tested, highly modular and efficient von Neumann architecture, which forms the basis of modern computing. The LLM appears to be, at best, a medieval era computer.

It is natural to ask: how do we transform these non-modular monoliths to modular, flexible engines?

This classical architecture abstracts a computer as a system composed of a processing unit and a shared memory, linked by a control mechanism (an orchestrator) that executes instructions one step at a time.

• Both data and instructions reside in the same memory

• The CPU accesses them (data and instructions) through a structured sequence of operations known as the fetch-decode-execute cycle.

• Fetch brings in a new instruction (from a special memory address), which is decoded and then executed (by loading memory, arithmetic and writing to memory).

Modern hardware refines this classical setup with faster processors, multi-level caches, and parallel pipelines, but the underlying principles remain largely the same.

Note how the von Neumann (vN) and the LLM computation models contrast:

• In vN, there is a distinct separation between instruction and data types, even though both are stored in memory. In contrast, instructions and memory in the LLM computer are all entangled together in both the parameters and the context. Makes it very hard to control or steer deterministically.

• fetch-decode-execute is the only so-called system prompt for the vN computer. The execution control flow is loaded on-demand from memory and executed dynamically. In contrast, the LLM computer contains all sorts of meta- and micro-instructions for main and sub-tasks entangled together, along with contextual data.

Because the non-modular LLM architecture, we pay deeply in terms of adaptiveness, performance and costs. Consider each component of the architecture one by one.

• The Memory is not modular.

  • Both common and less frequent facts are intertwined in the parameters.

  • Not distinguishing what is working vs long-term memory makes it hard to weigh them properly.

  • To specialize on a task/domain or business-specific facts, we must fine-tune the entire monolith on new facts or accumulate adapters for different scenarios.

  • The other options to inject new knowledge, is to inject it into the overlaid context window. The latter is a raw mishmash of instructions and data. This causes context rot, context lookups become unreliable.

• No separation of memory and reasoning. Both memory and reasoning as modeled as next-word generation, based on the transformation of inputs by parameters. No in-built mechanism to control when to generate facts vs when to reason over them.

• The System prompt is not modular. Instructions for dozens of tasks are all intertwined into a bloated stream of tokens, making it hard to distinguish the meta-instructions vs those which are specific to sub-tasks.

• Some instruction receive more attention, others are ignored without any warning. (note how this contrasts with the fully transparent fetch and execute in the von Neumann model)

• There is no good way to ensure that the correct instructions are loaded or activated at runtime, depending on the current context.

• Potentially infinite context layer ensures the same fate for data. We don’t know which data parts are attended to vs ignored, whether correct parts are attended to.

Altogether, non-modularity leads to lack of

🤖 Controllability: Can’t reliably constrain how models solve problems
💡 Accuracy: Runtime response don’t reflect user intent
🦺 Safety: Opens doors to shortcuts and hallucinations

The only part which supports modularity is the model training phase. Updating a large monolith model to add-and-compress new data is very expensive. Instead of updating the full model, we attach adapter parameters and fine-tune only adapters with techniques like LoRA. This avoids touching the older memories, at the cost of small additional memory. Parallel/pipeline optimizations help scale up the update algorithm on a distributed cluster. LoRA-based training is now a community-wide standard: except the GPU-rich, most resort to modular fine-tuning.

We observe that non-modularity pervades both the parameter and the context layer of the LLMs. Get rid of it is tough. We need several changes of the naive architecture to make it more modular.

In a Modular LLM, we expect the Instructions, Memory, Reasoning and Runtime components to be clearly (or controllably) separated.

How do we make the design modular? Again, let’s go part by part.

Let’s start with the Memory. There are many ways to partition memory. A simple taxonomy proposed here: Implicit memory (model parameters), Working memory (context key-values / layer), Explicit memory. Working memory can be further partitioned into episodic, recent and scratchpad memories.

Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining “abstract knowledge”.

Separating memories by term (short-term or long-term) or knowledge frequency (common vs rare/long-tail) is useful.

Explicit memory can be co-trained with the LLM as shown by the hierarchical embedded memory work.

• small language models that accesses large hierarchical parametric memory banks encoding world knowledge

• co-trained with LLM: selected memory banks fetched and fine-tuned

• allows local/sparse updates to selected memories, without touching the rest of the tree

Our pretraining learns to store long-tail world knowledge in the memory parameters, while the small language model acts as an anchor capturing common knowledge and general reasoning abilities.

We find that our proposed hierarchical feed-forward memories work robustly across transformer architectures, whether added during pretraining or post-hoc.

What criteria do we use to separate memory that stays implicit with the LM and what goes into external? That is an open problem.

Cognitive Core or Reasoning-Memory Separation. Andrey Karpathy mentioned the term cognitive core to decouple reasoning from knowledge in LLMs. Cognitive core is a lean reasoning module that stores only the common facts and primarily takes care of reasoning. The key aspect is that this module is not engineered to store knowledge - facts are separately stored and the core is responsible for retrieving knowledge on demand.

Making memory modular implicitly turns the LLM into a lean cognitive core.

Similarity with the classic von Neumann architecture begins to emerge.

One of the deep sources of non-modularity is the multipage system prompt. How do you split down a multi-page system prompt into modular parts?

We can separate instructions into markdown sections or tag boundaries to get basic modularity. Now we can edit each section separately, track changes better. However, the full prompt must be loaded upfront.

Can we split the system prompt into parts that are loaded dynamically depending on context?

One reason for bloat is that the system prompt combines together instructions for many specific sub-tasks. These are not central to the Agent, not invoked in every run, but only in specific scenarios when the user’s task needs them.So, why include those in the core-system instructions and bloat it up?

Instead, dynamically load this sub-task package when needed, and remove it when task is done. This is what Skills from Anthropic do.

Think of Skills as custom onboarding materials that let you package expertise, making Claude a specialist on what matters most to you.

• Skills are folders that include instructions, scripts, and resources that an LLM (Claude) can load when needed. LLMs access skills only when relevant to the context and Skills make LLM better at handling specialized, e.g., brand-specific tasks

• Skills are more than Tools. They include orchestration instructions over Tools at the Text layer. Skills = Toolgroups + Orchestration. (read more about Toolgroups here)

• Skills are composable, portable. Only load what’s needed, when needed.

• Skill (packages) can be created interactively in natural language. No manual structured programming needed.

Decomposing them as Skills makes it possible to make System instructions modular. Moreover, Skills swap-in and swap-out ensures that irrelevant instruction conflicts are avoided.

LLMs can generate Structured Response that follow a schema. This is absolutely essential for connecting the Text layer to the structured world and ensuring reliability of execution.

However, we require additional modularity in responses for Plan generation:

• Estimate Task complexity

• Identify Sub-Tasks

For complex tasks, we prefer that the Agent plans first and then generates the response. This separation (between planning and response generation) could be implemented implicitly or explicitly.

Implicit: via <scratchpad> .. </scratchpad> or <thinking> sections in output. Which are generated before the actual response. Ensured by training models on such data with reasoning tokens.

Explicit: Have distinct planner, executor modules in the LLM Agent.

The default chain-of-thought planning and execution by a monolithic agent may not be sufficient to carry out complex tasks. We need a modular Planner (see above), where different sub-modules take care of task decomposition and performing a tree-search to find executions that satisfy goals.

Modular separation enables having more powerful planners and cheaper executors. We can do more to modularize reasoning by first generating reasoning abstractions and then generate responses (see RLAD).

Traditional LLMs are monolithic engines, where

• Instructions and Memory are deeply entangled together in the model description

• Planning, Reasoning and Response generation occur in a tightly coupled manner at runtime

This design limits flexibility, reliability, interpretability, and the ability to scale capabilities efficiently.

Introducing modularity at the levels of memory, instructions, and runtime execution, offers a path forward. By

• separating implicit, contextual and explicit memories

• decomposing system instructions and loading on-demand

• disentangling planning from response generation, and

• enabling modular coordination at runtime

we can build more transparent, efficient, and extensible LLM Agents.

Ultimately, modularity transforms LLMs from static monoliths into dynamic system of interoperable components, laying the foundation for more steerable, adaptive, and reliable AI. By infusing modularity, I believe LLM Agent design will match the simplicity and universality that the traditional von Neumann architecture offers.