Notes on evaluating agentic tools

Over the past few months, I’ve worked on a three-part series on using coding agents for investigative journalism — replicating a published data investigation, cleaning and transcribing messy document collections, and generating tipsheets from raw datasets behind Pulitzer-winning stories. Across all three uses cases, it was clear that with the right scaffolding, coding agents are capable reporting tools.

However, underneath each of those headline results is a lot of material—transcripts, intermediate analyses, schemas, inferences, dropped threads—that I couldn’t evaluate in any rigorous way. Coding agents create a specific kind of evaluation challenge that will only become more pressing as researchers, practitioners, and model developers try to make claims about “performance” in complex agentic workflows. This post is an attempt to articulate a few of those challenges, both in general and in the specific context of the three case studies.

Three features of these workflows make them difficult to grade.

First, the objectives are multifaceted, and the priorities aren’t given. Is a more expensive, more comprehensive solution the right one? Should we prioritize interactivity, accuracy, brevity, cost, cautious handling of uncertainty, or something else entirely? These are editorial and methodological questions as much as technical ones, and they don’t resolve to a single quantifiable dimension.

Second, the sheer volume of output overpowers any ability to review it line by line. A single session can produce hundreds of tool calls, dozens of intermediate files, and a transcript stretching thousands of lines. Multiply that across runs and a few test cases and you quickly have a review surface that no one person can cover exhaustively.

Third, workflows are highly individualized. There are many valid paths to a given outcome; one journalist’s preferred approach is another’s detour. “Technically correct” and “not the way I would do it” are not mutually exclusive, and a good evaluation needs some way to distinguish the two. Otherwise you end up penalizing valid differences, or rewarding workflows just because they look familiar.

In my own series, I leaned on known endpoints to structure evaluation. The MuckRock replication had published top-line numbers to match against. The OCR cases had ground-truth text on the page. The tipsheet evaluation used 30 leads pulled from seven Pulitzer-winning investigations as a rubric. Anchoring to a known outcome lets you ask a concrete question—did the agent get to the same place?—that doesn’t dissolve into taste.

But endpoint anchoring essentially blinds you to the intermediate process. You can eyeball the transcripts for vibes around priorities like transparency and robustness, but what is “enough” documentation? What is “correct” provenance? Those are the properties we most care about in journalistic contexts, and endpoint matching often can’t speak to them directly.

Researchers try to close that gap through benchmark design and automated grading procedures. SWE-bench turns real GitHub issues into a closed-form evaluation: Agents produce patches, and unit tests decide whether they pass. τ-bench extends that idea to tool-using dialogue. The agent interacts with a simulated user and a programmatic API, and the benchmark grades it by comparing the final database state to an expected one. This side-steps the need to judge the conversation itself. In more open-ended domains, rubric-based approaches pair each prompt with 20–40 prompt-specific criteria that a judge (human or LLM) scores independently, trading the sharpness of a single verdict for fine-grained coverage of what “good” looks like on a particular task. Each design makes a different bet about how to operationalize correctness. None of them generalize cleanly to an overarching, domain-specific question like, “Did the agent do good investigative work?”

The other available move is field deployment. Putting tools in front of practitioners and watching what they actually pick up is, arguably, the only surefire way to discover what they value and don’t—a kind of reach test for prompts and skills. If a skill doesn’t become habitual for a working journalist, it doesn’t really matter what its benchmark score looks like. The reverse is also true: Workflows that benchmark indifferently might become indispensable once they’re in someone’s hands.

At some point, this challenge rolls up to a more philosophical problem. Evaluation requires precisely articulating how a value or priority gets operationalized in a specific domain. What does it mean to be “transparent” in journalistic data analysis, versus not? What does it mean to be “efficient,” or “cautious,” or “defensible”? If we don’t have solid answers, we risk repeating what we’ve seen with past technologies: Evaluations tilt toward what we can measure, whether or not that’s what we actually want to optimize for. The metric becomes the thing.

To avoid this outcome, we need to treat articulation—of the values a system should embody, and of how those values show up in observable behavior—as part of the evaluation work, not a preamble to it.

Each case study ran into a different slice of the problem.

The investigation replication. The final goal—matching MuckRock’s top-line counts—provided a useful anchor. The numbers either did or didn’t line up. But replicating a known outcome in an interactive environment raises validity problems I couldn’t fully neutralize. Consciously or not, an operator can steer a system toward an outcome they already know. I tried to guard against this by interjecting minimally, publishing full transcripts, and documenting every intervention, but there are no guarantees. This is a tough contradiction: Known outcomes allow validation, but an interactive system will always require a fallible operator.

The OCR and document wrangling cases. Here, too, there’s a known end state: Does the processed text map onto what’s on the page? This is a straightforward question in principle, but it’s nuanced in practice. The Wired primate-research documents, for example, mixed typed forms, partly-handwritten tables, marginalia, and stickers applied to the scanned sheets. There’s an information hierarchy baked into the task—some things matter more to the reporter than others—and an error hierarchy to match. A one-letter misspelling in a handwritten margin note isn’t the same as the wrong number in a drug dosage table. Both hierarchies are extremely contextual to the document and the reporter, which makes a broadly applicable rubric hard to build. And in an interactive environment, operator expectations bend the evaluation: Is the agent supposed to clean everything autonomously, or stop for review per-page, or per-document? Complicating this further, there are many valid approaches to OCR: Direct multimodal transcription, tesseract, commercial OCR pipelines, bespoke scripts. Each is a legitimate approach with different cost, accuracy, and auditability profiles. A rubric has to account for many valid tradeoffs, which multiplies its caveats.

The tipsheet evaluation. Here I built the most structured rubric of the three: 30 leads, drawn from seven Pulitzer-winning investigations, with consistent units and criteria across disparate datasets. That took real setup work but produced a consistent measure across contexts. It also came with its own problems. The interactivity concern is smaller, because the agent mostly ran without intervention, but all of the chosen test cases fall within the model’s training-data cutoff. I can’t cleanly separate what the agent derived from the data itself from what it may have pulled from parametric memory, however unlikely it seems that a model would shortcut to memorized news coverage. And even in a well-defined rubric, the outcomes aren’t cleanly quantifiable. Sorting leads into “strong,” “weak,” and “missing” buckets is partly an art. If an agent gets the right abstract idea but illustrates it with a different anecdote than the published story, is that really much weaker? What about the original leads—ones not in the rubric—that look compelling but aren’t scorable under the design?

These aren’t failings of any specific case study, so much as evidence that every reasonable evaluation design trades off something. Precision costs generality, and vice versa.

One approach worth highlighting in closing, which I’ve found increasingly useful, is simply making transcripts from agentic systems easier to read. Simon Willison built a viewer for Claude Code transcripts that turns raw JSONL logs into a browsable interface. It’s a small tool with outsized utility for evaluation work: Being able to step through an agent’s actual actions, inspect tool calls, and flag surprising moves is a precondition for any kind of qualitative coding or post-hoc review.

That points to a broader direction evaluation will need to embrace. Alongside benchmark-style summaries, we’re going to need something closer to the social-science tradition of qualitative coding—reviewing many instances of agent behavior, annotated against a shared scheme, with enough tooling to make review tractable. Kapoor et al. make a related case in their recent paper on "open-world evaluations", arguing that long-horizon, real-world tasks assessed through qualitative log analysis are a necessary complement to benchmarks, precisely because benchmarks privilege what can be precisely specified and automatically graded. This kind of intensive review is time-consuming, and it's more difficult to distill down than quantitative evaluation. But it's also a reasonable way to keep the measurable from crowding out the meaningful.