The LLMs get the publicity. The data layer does the work.

I’m gonna let you in on a secret. No one building agents has a clue how to build them. The people who look like they know what they’re doing are just throwing spaghetti at the wall faster than everyone else. The technology changes day-to-day, best practices have a half-life of about six hours, and the conference talk you saw last month is already obsolete. We’re all just holding on for dear life.

Over the past year, I’ve had the luxury of seeing how some of the most sophisticated organizations are building agents, and one pattern keeps emerging: every AI application, at its core, is becoming a data application. Not a model application. Not a prompt engineering application. A data application. The LLMs get all the publicity, but the data layer does all the work.

I should disclose upfront that I work for ClickHouse, so I’m about as unbiased on the topic as a fox writing restaurant reviews for a henhouse, but I’ll try to be honest about where my bias shows up. You should always form your own opinions.

Most agents in the wild follow a pattern called ReAct – reason, then act. If you know what that term means already, skip to the next section. For everybody else who’s still here, a ReAct agent takes an input, thinks about it, does something, and produces an output. If the output isn’t good enough, it loops back and tries again. That’s it. That’s the magic.

The “does something” part is where it gets interesting. LLMs, at their core, take in text, and spit text out. They can’t inherently do things. So we gave them tools – bits of application logic that let an agent search the web, query a database, send an email, write code, invest your life savings into Gamestop options, whatever you need. It’s a little bit of a crude analogy, but your hands are kind of like a tool for your brain. It’s what your brain uses to interact with the rest of the world. Every major model provider supports tool use now, and it’s the primary mechanism through which agents interact with the world.

What makes agents actually work, though, is the interplay between three things: the model (which reasons), the tools (which act), and the context (which informs both). To take a human analogy, the model is your brain, a tool might be a hammer (inspired, I know), and the context is the knowledge you have about how to use that hammer to put a nail into a board. These three things form a triumvirate, and understanding their relative importance is key to understanding where the value lives in the agent stack.

The thing most people get wrong is that they obsess over the model, spend time on the tools, and treat context as an afterthought. It should be the other way around.

A few years ago, ChatGPT could barely add two numbers together. Now Claude is one of the most prolific software engineers in the world. (Claude has also been my primary editor for my last couple posts, and really hated a few of my early joke iterations, so I’m choosing to view that as the technology working).

The pace of model improvement is staggering. But what’s happening alongside that improvement is commoditization. GPT5, Claude Opus, and Gemini 3 are all incredibly capable all-rounder reasoning models, and they’re increasingly competing on cost and speed rather than capability. I’ve talked to companies that test different models weekly – swapping models in and out like you’d change a config value – because for a well-abstracted agent, that’s literally all a model change is. A config change.

When a technology layer commoditizes, the value doesn’t disappear. It migrates. When compute commoditized (AWS, GCP, Azure all offer basically the same VMs), the value moved up to the services layer – databases, queues, orchestration – the stuff you run on the compute. The same thing is happening with models. The model is becoming the commodity substrate, and the value is migrating to the layers around it, which, conveniently for me and my stock options, means the data layer. To put a finer point on it, my belief is that, in the AI world, the proprietary data that a company amasses to help their agents better complete their tasks is the organization’s moat. Cursor has a dataset that every other coding agent tool would kill for, precisely because it enables them to better inform their agents on how to write high-quality, bug-free code.

Tools tell a similar story. A year ago, building tool integrations was a custom job for every agent. Then MCP happened, and suddenly every company and their mother has an MCP server. Tool integration is essentially a solved protocol problem. That doesn’t mean tools aren’t important – choosing the right tools for an agent is a genuine design problem. But the mechanisms are standardized and the plumbing is commoditized. And none of it matters if the agent doesn’t have the right context to know which tools to use, and the right data to work with when they get called.

Here’s the thesis in one sentence: for any given task, the model and the tools are only as good as the context that informs them. Context is where a chatbot becomes an agent. And context, at scale, is a data problem.

If you’ve ever watched Claude confidently reference something from a week ago while having zero memory of what you told it thirty seconds prior – congratulations, you’ve experienced the context problem firsthand. That’s the consequence of the right context not getting to the agent.

Context in modern agents is typically architected as a memory system, and it works a lot like your own memory does. Short-term memory is the stuff an agent needs on the tip of its tongue – the current conversation, user preferences, the plan it’s executing, the corrections you just gave it. For example, I like my chatbots to have a bit of snark and a healthy mix of wisecracks. That preference lives in short-term memory, and is the last line of defense against millennial gray AI.

The implementation depends on the workload. Structured lookups, such as “what are this user’s preferences?” want something like Redis: extremely fast, exact retrieval, purpose-built for caching. But not everything in short-term memory is a structured lookup. Sometimes the agent needs to find related context, not exact matches, and that’s where vector search enters the picture.

Vector search systems take a piece of text and find semantically similar content in a corpus. Before you go haranguing me for mentioning vector search without also talking about RAG – keep reading, because RAG and memory are more intertwined than most people realize, and I’ll get there.

The critical thing about short-term memory is that it’s mutable. It changes constantly as the agent learns, receives corrections, and updates its understanding. This makes OLTP systems (think Postgres – databases built for lots of small, fast reads and writes) a natural fit. Postgres with pgvector gives you both relational storage and vector search in a system designed for frequent updates. OLAP systems, like ClickHouse (the database, not the company) are really not built for this type of workload, so this is a great spot for Postgres. Conveniently for you, the reader, ClickHouse (the company) offers a managed Postgres offering, as well.

Long-term memory is a different beast. This is the kind of stuff that is going to be valuable at some undetermined future point – product documentation, historical interactions, skill libraries, reference material. There’s a lot of it, most of it isn’t relevant to any given conversation, and the challenge is finding the right memories when you need them.

When a user raises a specific topic, information gets retrieved from long-term memory and promoted to short-term memory for faster access. This is fundamentally a search problem over a large corpus, and it’s where ClickHouse can genuinely shine – fast vector search across massive datasets, on data that changes slowly. An OLAP-based vector search system is actually the right choice here.

If this all sounds kind of like a RAG (retrieval-augmented generation) process, that’s not a coincidence. RAG is just the technique of retrieving information and then injecting it as additional context into a reasoning step. Memory is the retrieval system. The two are the same problem in different clothes.

For those unfamiliar with how vector search systems work: text (or images, or video, or audio) gets converted into numerical vectors using embedding models. Those vectors plot content into a multi-dimensional space where semantically similar concepts end up near each other. Search means finding nearby points. That’s the whole idea.

When you zoom out, what you’re looking at is a data orchestration problem. Conversations generate memories. Memories get categorized by relevance and recency. They get loaded into different storage tiers. They move between tiers as conversations evolve. The AI future everyone was promised was sentient robots and digital consciousness. The AI future we got is glorified ETL pipelines arguing with vector databases about cosine similarity. I’ve never felt more at home.

Here’s something people don’t realize about agents that query data systems: they don’t work the way humans do (and Jeff Dean agrees). A human analyst sits down, thinks about a problem, and crafts a single query. An agent fires off a dozen queries in rapid succession, reasoning between each one, iterating toward an answer at machine speed. A single user question doesn’t translate to a single database query. It translates to a chain of them, and each one needs to come back fast because the agent is waiting to think about the result before issuing the next.

At ClickHouse, we have an internal chatbot called DWAINE, built on top of LibreChat and ClickHouse (which we use as our internal data warehouse). Sales reps use it constantly – checking consumption trends, predicting when a customer will run out of credits, analyzing usage growth. DWAINE goes straight to the data warehouse, queries, reasons about the results, queries again, and produces an answer. Marketing teams ask about channel ROI and seasonality patterns. Same loop. All of it is analytical in nature, and all of it requires fast, iterative access to data.

Then there’s the definition problem. I heard an anecdote recently about a company with a dozen different ways of defining “daily active user.” If you ask an agent for DAUs by geography, the agent first has to figure out which definition to use, which might require its own chain of queries against a semantic layer or data catalog. This is, under the covers, yet another data problem – the kind that semantic layers and tools like Atlan and Collibra exist to solve.

Agents also make mistakes. They start down a reasoning chain, realize the results don’t make sense, back up, and try a different approach – the same way a human would, but faster. An SRE agent investigating a production outage might query logs, form a hypothesis, query metrics, pivot, query traces, and progressively narrow down a root cause across multiple systems. Every one of those queries needs to return fast because the agent isn’t going to sit there and contemplate. It already knows the next question.

When a sales rep asks DWAINE how a customer’s consumption is trending, they don’t want to wait two minutes while the agent chains through a half-dozen queries. They want an answer in the time it takes to sip their coffee. This is where specialized OLAP systems like ClickHouse – built from the ground up for fast analytical queries over massive datasets – earns its keep.

Building an agent is easy. Building an agent that consistently does the right thing is brutally hard. You need to observe what the agent is doing in production, and you need to evaluate whether what it did was actually good. Both of these are – you guessed it – data problems.

The first step is tracing – capturing every code path, function call, LLM response, and tool call in an interaction. Anybody can vibe-code their way to a v0 with Claude Code or Replit or Lovable. Whether the v0 actually works is a separate question entirely. At some point, you graduate from vibes to measurement. Traces get you there, and tracing is the gateway drug to evaluations.

Tracing is a data problem from day one, because agent traces are nothing like traditional application traces. In a software application, a span (one step in a trace) is about 1 KB. In an AI application, spans start around 50 KB because you’re dealing with long LLM responses, and that’s just for text. Add in images, video, or audio, and you’re in a different universe. Scaled agents can generate gigabytes of trace data per second. If you thought your Datadog bill already made your CFO cry, just wait until your agents start writing five-paragraph essays about why they chose to call the wrong API. Every trace is a short novel, and you’re paying per chapter.

The quality problem is even harder. You can’t write deterministic tests for stochastic systems. There’s no specific input that always produces a specific bad output. LLMs vary every time, even on the same prompt. Improving an agent means evaluating its responses across large datasets of examples, and building those datasets means finding the right traces – the ones that illustrate specific categories of failure. That requires sifting through massive volumes of LLM reasoning text, which demands full-text search at a scale that most systems aren’t built for. Finding traces that illustrate specific failure categories is a devastatingly hard data problem.

And here’s where context circles back. Even evaluations depend on getting context right. If I ask an agent to identify a movie from the description “Harry Potter goes to Hogwarts for the first time, meets new friends, and has to defeat Professor Quirrel,” a good, non-hallucinatory response is either “Harry Potter and the Sorcerer’s Stone” or “Harry Potter and the Philosopher’s Stone.” Both are technically correct. But which one is right depends on whether the user is American or British – information that would come from short-term memory in a production system. Evals have to test for this, which means eval datasets need contextual variation, which means you need robust data systems underneath your evaluation pipeline.

ClickHouse shines here, which is why many of the major AI observability platforms chose it as their underlying datastore. Langfuse (recently acquired by ClickHouse), Helicone, Laminar, and Weave from Weights and Biases all run on ClickHouse. LangSmith from LangChain offers a managed ClickHouse underpinning its Enterprise offering. Not everyone uses ClickHouse – Braintrust initially started out on top of ClickHouse, before building their own custom Brainstore database. Arize built their own datastore, as well, and their open-source Phoenix project runs on Postgres. It makes sense, for the same reason Datadog built Husky – they built a database that was hyperspecific to their use case. Building your own database is also an extremely hard thing to get right, so it’s no surprise that the pattern emerged. And now that Langfuse has been acquired by ClickHouse, it’s likely that ClickHouse will get even better for this class of application.

When you map out everything an AI application needs from a data layer, one pattern keeps emerging: these are all the same class of problem. Memory retrieval, analytical queries inside a ReAct loop, trace ingestion, and search for observability – they’re all variations of storing and querying large volumes of data at low latency. It’s one problem wearing four different hats.

ClickHouse isn’t going to play in every part of the stack. It doesn’t provide models. It’s not your agent framework or orchestration layer. And it remains an analytical system, not a transactional one – short-term memory is not its arena. Here’s what the full picture looks like:

I’ve written before about how I think about career decisions as investment decisions. Models will continue to be worth trillions. Frameworks will rise and fall. But every agent and every AI application needs to store, retrieve, search, and analyze data at speeds that keep up with the reasoning loop.

The reason I’m at ClickHouse is this thesis – that the data layer is the most underappreciated part of the AI application stack, and that a company assembling the right pieces to own that layer has an enormous amount of upside. ClickHouse – the business – clearly recognizes that owning the AI data layer means covering workloads that ClickHouse – the database – wasn’t originally designed for. They’ve made a number of strategic acquisitions (Langfuse) and introduced new products (a managed Postgres offering). That’s a company explicitly acknowledging that some workloads need a different database – and doing something about it. The bet I’m making is on ClickHouse, the company, continuing to execute on precisely that strategy.

If you’re building agents, or thinking about building agents, I’d challenge you to map out where data systems show up in your architecture. You might be surprised how much of what you’re building is really a data problem wearing an AI costume.