Context Engineering in Practice

Most agents don’t fail because the model is weak. They fail because the context is wrong.

When you first start building a writing agent—something that can draft blog posts, edit them across several turns, store them, schedule them—you assume the hard part will be the model.

It’s never the model.

Most people assume that building a powerful AI agent requires a model with a gigantic context window. “If I can just feed the model 100,000 tokens every time, it will always know everything it needs.”

But here’s the quiet truth:

For most real tasks, you don’t need a huge context window at all. You need smart context engineering.

The real challenge is getting the agent to behave coherently across multi-turn workflows without drowning the LLM in thousands of tokens of irrelevant junk.

In this post, we’ll walk through how context actually works in a real agent system, why tools make context bigger (not smaller), and how to design prompts and tool results that stay efficient even as your users write long-form content.

This applies to every agent that needs long-term awareness—writing assistants, planning agents, organization tools, content managers, or anything with multi-step state.

Let’s start at the beginning.

Most agent projects start with something like:

llm.chat_completion(messages=[{”role”: “user”, “content”: user_input}])

Single turn. No state. No memory.

This works until the user says:

• “Let’s start a draft.”

• “Expand section two.”

• “Rewrite the intro.”

Suddenly the model is supposed to know what “the draft” is.
Of course it doesn’t. You never gave it any memory.

So you do what everyone does:
you dump the entire conversation history into every prompt.

And it works… briefly.

But as soon as the user writes real content—500-word drafts, repeated revisions, long feedback threads—the context blows up. Costs rise. Latency spikes. And the model begins to behave unpredictably.

This is your first encounter with context bloat.

To solve the state problem, you add tools:

• create_post

• get_post

• update_post

• schedule_post

• list_posts

Now the model can manipulate actual stored data.
It feels like progress, and it is.

But there’s a detail many developers underestimate:

Tool results are inserted directly into the model’s next prompt.

If the agent calls get_post(”draft1”), and your tool returns:

{
  “id”: “draft1”,
  “title”: “AI Safety and Tradeoffs”,
  “body”: “Here comes 2,500 words...”
}

…the entire response becomes part of the model’s next input.

And this is absolutely intentional.
It’s how the model “sees” the content.

But it creates a new kind of explosion.

This is a key point, and it deserves to be explicit:

You put the full draft into the tool result because otherwise the model can’t work on it.

If the model is supposed to rewrite, analyze, or edit a post, it needs to have the post in front of it. The LLM is not telepathic; it must receive text in its input.

This is why early agents often shove entire documents, databases, and lists into every tool response: it feels necessary.

The problem is not that this is wrong.

The problem is that if you do this naïvely, your context window becomes a black hole.

Imagine this workflow:

1. User: “Summarize my draft.”
   → Model: fetches the post using get_post
   → 2,500-word draft added to the context

2. User: “Rewrite section two.”
   → Model: fetches again
   → the same 2,500 words added again

3. User: “Change the tone of the intro.”
   → Another fetch
   → Another 2,500 words

Even if you aggressively trim history, the current tool result still includes the entire draft.

This is how even simple editing workflows balloon into 20k, 30k, 50k token prompts.

This is what your agent runtime is actually passing in (simplified):

[
    {”role”: “system”, ...},
    {”role”: “user”, “content”: “Rewrite section two.”},
    {
        “role”: “tool”,
        “tool_name”: “get_post”,
        “content”: {
            “id”: “draft1”,
            “title”: “My Draft”,
            “body”: “2,500 words of content...”
        }
    }
]

Every tool call injects another slab of text.

This is how most agents break.

Developers try:

• Keeping only the last 3–5 turns

• Moving previous turns into a summary

• Dropping anything “unimportant”

This helps, but only partially.

Because while you can shrink history, the tool result must stay visible if the model needs to edit the content.

You can’t hide the content. The model needs it.

And so trimming helps, but does not solve the core problem:
your working set is still enormous.

Time for actual engineering.

Good agents follow a simple rule:

The model should see exactly what it needs, exactly when it needs it, and nothing else.

This means restructuring the prompt into several layers.

A short, rolling description of what has happened so far.

Key facts stored as small JSON structures:

{”current_draft”: “9e3d”, “mode”: “editing”}

“What should the model do right now?”

If the user is editing section 2, only that section should be returned.
Not the entire document.

This gives the model enough information to work while keeping the prompt lean and predictable.

Here’s the kind of prompt a real agent sends the model:

messages = [
    {
        “role”: “system”,
        “content”: SYSTEM_INSTRUCTIONS
    },
    {
        “role”: “assistant”,
        “content”: (
            “Summary: The user is editing draft #9e3d. “
            “The draft has four sections. Last action: summarization.”
        )
    },
    {
        “role”: “assistant”,
        “content”: “State: {’current_draft’: ‘9e3d’, ‘mode’: ‘editing’}”
    },
    {
        “role”: “user”,
        “content”: “Please refine section 2.”
    }
]

Only if needed will the model fetch content via a tool.

The agent remains responsive, but the context stays compact.

This is the core idea of context engineering.

Since tool results go straight into the LLM, they should be as small as possible.

Here are the patterns production agents use:

Perfect for lists, indexes, or dashboards.

If the user edits section 2, return only section 2.

Especially when working with versioned documents.

Let the model request the rest only if it needs it.

Let tools return summaries rather than repeating entire documents.

The best agents treat the LLM like a senior editor:
only put the relevant part of the document on their desk.

When context is engineered well:

• Token usage drops dramatically

• Agents stop hallucinating state

• Latency decreases

• Multi-step editing becomes reliable

• The agent feels cohesive and “aware”

• Debugging becomes possible (finally)

When context is not engineered:

• Agents re-edit the wrong section

• They forget what they were doing

• They hallucinate previous steps

• Tool calls loop

• The prompt balloons

• Costs skyrocket

Most agent failures are context failures wearing a different mask.

The misconception is that agents need memory.
They don’t. They need context.

But not “dump everything into the prompt” context.

They need curated, structured, purposeful context.

A well-designed agent feels like it remembers everything.
In reality, we are selectively feeding it the minimum information required to reconstruct the right mental state at the right time.

The magic is not in the LLM.
The magic is in the engineering.