Your AGENTS.md File Is Making Your AI Coder only 3% better?

If you’ve been around the AI-powered coding scene lately, you’ve undoubtedly heard the gospel of the AGENTS.md file. It’s been pitched as the ultimate cheat code: a simple, open-format file that gives your coding agent a map of your project, turning it from a lost tourist into a seasoned local guide. The narrative is compelling—give the agent more context, and it will do a better job.

But what if the map is actually a distraction? A recent, rigorous study from ETH Zurich and DeepMind has turned this assumption on its head, and the results are a wake-up call for anyone who has been meticulously curating these files.

The Great AGENTS.md Experiment

The paper, titled “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” by Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev, set out to answer a simple question that no one had yet rigorously tested: Do these context files actually help AI coding agents complete real-world tasks?

To find out, the researchers built a new benchmark called AGENTbench, consisting of 138 real-world software engineering tasks from 12 Python repositories that already use developer-written context files. They complemented this with the established SWE-bench Lite benchmark, which has 300 tasks from popular repositories, where they used LLMs to generate the context files.

They then set four popular coding agents loose on these tasks in three different modes: with no context file at all, with an LLM-generated context file, and with a human-written one. The results were nothing short of counterintuitive.

Subscribe now

The Surprising Results: A Marginal Affair

The researchers’ findings directly contradict the prevailing “more context is better” wisdom. Here’s what they discovered:

• LLM-Generated Files Hurt Performance: On average, automatically generated AGENTS.md files decreased task success rates by a staggering 3%. Providing an AI with auto-generated instructions was worse than providing none at all.

• Human-Written Files Are... Just Okay: Developer-written context files provided only a marginal improvement, increasing success rates by an average of 4%. This is a very small gain for what is often a significant documentation effort.

  

• A Hefty Price Tag for Small Gains: Even when they “helped,” context files came with a significant cost. They consistently increased the inference cost of completing a task by over 20%. The agents required more steps, more tokens, and more thinking time to achieve barely better (or even worse) results.

Why Does More Context Hurt Performance?

This is the critical question. If the agents are following the instructions, why aren’t they doing better? The study’s analysis of agent behavior traces provides some fascinating answers.

1. Agents Became Overzealous Rule-Followers

The good news is that the coding agents are excellent instruction-followers. The bad news is that this becomes a problem. When a context file instructs an agent to “thoroughly test all changes” or “use uv for package management,” the agent complies, even when it’s unnecessary for the simple bug fix at hand. This leads to:

• Increased, Often Redundant, Testing: Agents ran more tests than needed, wasting time and compute.

• Unnecessary Exploration: They navigated through more files, performed more searches, and used more repository-specific tooling than required for the task.

The result is a well-intentioned agent that spends a lot of its energy on meta-tasks, losing sight of the core goal: resolving the issue efficiently.'

2. Context Files Are a Poor Man’s Codebase Overview

One of the primary promises of a context file is to provide a useful architectural overview, helping the agent quickly find the right files. The study found this promise to be largely unfulfilled. Agents with access to context files did not discover relevant files any faster than those without them. It turns out that the agents’ built-in ability to explore and search a codebase is already quite good, and a static, often high-level map doesn’t add much value.

3. The “No Documentation” Trap

Interestingly, the researchers tested a scenario where all other documentation was removed, leaving the context file as the sole source of information. In this extreme case, LLM-generated context files did consistently improve performance, even outperforming human-written documentation. This explains why some developers anecdotally report success—their repositories likely had little to no documentation to begin with, so any documentation was a massive help. This is a crucial insight: the effectiveness of a context file is inversely proportional to the quality of your project’s existing documentation.

A Better Way Forward: The Principle of Minimal Instruction

So, should you throw your AGENTS.md file in the trash? Not necessarily, but the way you create them needs a fundamental rethink. The study’s core recommendation is a principle of minimalism.

“We therefore suggest omitting LLM-generated context files for the time being, contrary to agent developers’ recommendations, and including only minimal requirements (e.g., specific tooling to use with this repository).”

Here are some actionable takeaways based on the research:

• Stop Auto-Generating: For now, the evidence strongly suggests you should ditch the /init command and other automated tools for creating context files. The risk of them causing harm far outweighs the negligible benefit. Always hand-write your AGENTS.md based on real experience with the codebase.

• Start with an Empty File: A zero-length AGENTS.md is better than a verbose one. Build it iteratively, only adding instructions when you observe the agent consistently struggling with a specific, non-obvious aspect of your project.

• Focus on the “What,” Not the “How”: Your instructions should clarify what the agent needs to accomplish, not a step-by-step guide on how to do it. For example, “Use uv for package management” is a good, concise instruction. A paragraph on how to structure a loop is redundant and potentially harmful.

• Invest in Good Conventional Documentation: The study shows that agents are very capable of understanding standard documentation. A well-maintained README.md, clear code comments, and logical directory structures are likely a far better investment of your time than a dedicated, highly-curated agent file.

Conclusion

The “Evaluating AGENTS.md” paper is a much-needed reality check in the fast-moving world of AI coding assistants. It reminds us that the intuitive solution is not always the right one. While the idea of a dedicated instruction file for our AI partners is elegant in theory, in practice, it’s a tool that is often misused, leading to distracting, costly, and self-sabotaging behavior.

The path forward is not about providing more context, but better, more precise context. The most effective AGENTS.md file you can have is likely the one that is almost empty, containing only the few critical details that your coding agent cannot discover on its own. Let’s stop building ornate maps that confuse the tourist and instead just point them in the right direction. I highly recommend reading the full paper for a deeper dive into the data. You can also check out the open-source code for their AGENTbench framework.

Sources:

[1] https://arxiv.org/html/2602.11988v1