Prompts Are Not Workflows: rewriting Deep Research

I woke up, checked my terminal, and the research agent was still running. It had been running for sixteen hours.

Not working. Running. A single URL on a HuggingFace blog had hung during a fetch. The agent, instructed to “batch fetches 2-3 at a time,” had fired ten in parallel. One hung. The other nine completed and returned their data. Then the whole process sat there, waiting, because the tool has no concept of a timeout apparently. The progress file said “READ phase.” The dispatcher saw “READ phase” and concluded: still reading.

Technically true. Functionally useless.

My deep research skill is the most valuable skill I’ve ever written and a foundational block of my path to autonomous agents, but it was driving me crazy with its erratic behaviors. I had been running the latest version for a few months, a Claude Code skill that orchestrated an 8-stage research pipeline: PLAN, DIVERGE, CHALLENGE, SEARCH, TRIAGE, READ, REASSESS, SYNTHESIZE. Each stage had its own rules, parameters, quality checks, and output formats. The entire methodology lived in a 1,099-line prompt file called research-prompt.md, paired with a 948-line dispatcher. Nearly 2,000 lines of prose instructions telling the model how to conduct web research.

When it worked, the output was genuinely impressive. Thorough, well-sourced, multi-perspective research reports with structured data appendices. The model followed the methodology because the methodology was well-designed and the model was smart enough to understand it.

The problem wasn’t the model’s intelligence. The problem was that I’d written a workflow in the wrong language.

A prompt describes intent. It says what you want and how you’d like it done. That’s exactly right for tasks requiring judgment: reasoning about ambiguous inputs, extracting meaning from text, synthesising information into narrative.

A workflow is a process. Stages that must execute in order. Gates that must pass before proceeding. Budgets that must be respected. Timeouts that must be enforced. State that must survive failures. These aren’t judgment calls. They’re mechanical requirements. And when you express mechanical requirements in natural language, you’re writing pseudocode.

Pseudocode is useful for thinking through logic. It’s terrible for running in production. The difference between max_total_fetches: 40 in a config file and “stop all link-following immediately once the budget of 40 fetches is reached” in a prompt is the difference between a constraint and a wish. One is enforced by the runtime. The other is enforced by the model’s willingness to remember and comply, which is a polite way of saying it isn’t enforced at all.

The budget nobody respected. The prompt specified hard caps: 15 fetches for quick runs, 40 for standard, 60 for thorough. It said, in bold, “once reached, stop all link-following immediately.” The model routinely fetched 80 pages on a standard run. The sources looked interesting, so it kept going. No counter, no enforcement. It burned through context window and time, and extraction quality degraded as the context filled up. The prompt also said “exponential backoff (1s, 3s)” for API rate limits. The model fired requests as fast as it could.

The gates that weren’t. The prompt said “check query diversity” and “require at least 1 authoritative source” and “if reassessment fails, trigger a refinement cycle.” These were suggestions. The REASSESS stage was supposed to catch bad framings and trigger re-search. In practice, the model always said “framing looks good, proceeding to SYNTHESIZE.” Path of least resistance, a deep desire to please.

The rogue re-searches. This was the most insidious one. When a search returned poor results, the model would sometimes decide to redo the entire search phase on its own. Different queries, ignoring the approved research plan, skipping the DIVERGE and CHALLENGE stages, not running results through TRIAGE. It was trying to be helpful, but it bypassed every quality gate: no diversity enforcement, no contrarian angles, no deduplication.

The results looked fine. That’s the dangerous part. If you read the output without knowing the methodology, you’d think: solid research, good sources, reasonable conclusions. You’d never notice the contrarian angles were missing, that the sources hadn’t been deduplicated, that the diversity requirements were ignored. It’s the same as a student who skips the lab protocol and eyeballs the measurements. The report looks right. The numbers pass a sanity check. But the process that makes them trustworthy was never followed.

No visibility. A research run takes 15-25 minutes. The only window into what was happening was a progress file the model was supposed to update at each phase transition. It often forgot. When it did write timestamps, they were fabricated: rounded to the nearest five minutes, clearly made up. How many sources were fetched? How many failed? Which queries returned nothing? Where is the agent right now? No way to know.

You can’t trust what you can’t monitor/eval.

So I rewrote it. The old skill was ~2,047 lines of pure prompt text. The new pipeline, which also does a lot more than its predecessor prompt, is ~6,035 lines of Python, plus 711 lines of focused prompt templates split across eight files, plus 2,369 lines of tests. The total system grew by 4.5x. The amount of text the LLM actually sees shrank by 35%.

The principle: about 55% of the work moved to Python, 45% stayed as LLM calls.

Code now handles stage sequencing and gate enforcement; search routing across multiple APIs (Tavily, Exa, Serper, You.com) with quota tracking and circuit breakers; content fetching with hard timeouts and per-request failure isolation; state checkpointing to JSON after every stage, with per-source granularity; progress logging with real timestamps and fetch counts; quality gates as code assertions that cannot be skipped; and map-reduce synthesis that splits large finding sets across parallel chunks when they exceed the model’s context budget.

The LLM still handles query generation (PLAN, DIVERGE, CHALLENGE), relevance scoring (TRIAGE), content extraction (READ), framing reassessment (REASSESS), and final synthesis (SYNTHESIZE). Each of these is a genuine thinking task: given a research question, generate contrarian queries. Given a fetched page, extract findings relevant to the question. Given all findings, write a coherent report.

Each LLM call is now a focused prompt in its own file, typically 40-60 lines, sometimes up to 293 for synthesis. Stateless, one-shot, called via claude_agent_sdk.query(). The model doesn’t need to remember what stage it’s in. The code tracks that. The model just does the one thing it’s good at: think about a well-scoped problem and return a structured answer.

Timeouts that work. A single hung URL used to stall the entire agent for sixteen hours. Now asyncio.wait_for(timeout=60) is a hard kill. If a URL doesn’t respond in sixty seconds, it’s marked failed, logged with the URL, error type, and elapsed time, and the pipeline moves on. Not a suggestion in prose. An asyncio cancellation that the code enforces regardless of what the model thinks.

Gates that enforce. “Check query diversity” used to be a suggestion the model ignored. Now assert len(unique_domains) >= config.min_source_diversity stops the pipeline and triggers a recovery action. When REASSESS triggers a re-search, that re-search goes through the full DIVERGE, CHALLENGE, and TRIAGE stages. No more freelance redos with improvised queries.

Budgets that stick. “Max 40 fetches” in prose, 80 in reality. Now it’s if state.fetch_count >= config.max_total_fetches: break. The SearchRouter tracks calls per provider, enforces monthly quotas, and implements actual backoff with real asyncio.sleep() delays.

State you can recover. A crash at SYNTHESIZE used to mean total loss of all collected findings. Now ResearchState checkpoints to JSON after every stage, with per-source granularity. A crash at source 19 out of 20 means reprocessing one source, not twenty.

Different models for different jobs. The old version needed Opus for everything because the 1,099-line methodology required the strongest model just to follow the instructions. Sonnet would lose track of where it was. Now extraction runs on Sonnet, which is cheaper, faster, and good enough when the prompt is 60 lines instead of 1,100. Planning and synthesis run on Opus, where reasoning quality matters. A model resolver reads a config and assigns per-node.

Tool integration you control. Want to swap Tavily for Exa? In the old version: rewrite the prompt and hope the model calls the right tool. Now: the SearchRouter manages five providers with failover. Adding a provider is a Python class. The model never touches search infrastructure.

Transparency about failure. Failures used to silently disappear. Now the pipeline appends a “Failed Sources” table to every report: URL, error type, context. In one run, this table showed four blocked sources. I manually fetched those pages and discovered a finding the pipeline missed because it couldn’t reach the content. What didn’t work is itself a finding.

This might be the most important part, and it’s the one that’s hardest to see until you’ve lived it.

Once you have code, you can add logs. Obvious. But what you can do with logs may be less obvious to many who haven’t been there before.

Once you have logs, you can record operational details of every run: which providers were called, how long each stage took, how many tokens each LLM call consumed, which sources failed and why. Once you have operational details, you have data. And once you have data, you can build things that were impossible when the orchestration lived inside a prompt.

Token tracking. Because the pipeline calls claude_agent_sdk.query() you get full access to token usage per call. You know exactly how many input and output tokens each node consumed.

Cost budgets. From token tracking, you derive cost per run. Set a limit: “this research should not cost more than $5.” The pipeline checks after each LLM call and stops if the budget is exhausted. Try that with a prompt. Write “stay within budget” in bold and underlined, and the model will nod politely and spend whatever it wants.

Performance profiling. Which stage is the bottleneck? How long does TRIAGE take vs SYNTHESIZE? Which search provider returns fastest? In one early run, I found that government PDF pages were taking 30-50 seconds per fetch, which is why my initial 15-second timeout was killing everything. I raised it to 60. That kind of targeted fix is only possible when you can see the data.

A/B testing. I compared three fetch-and-extract strategies. Strategy A (fetch, then extract in separate calls) used 939 tokens at 20.2 seconds. Strategy C (combined fetch-and-extract in a single SDK call) used 594 tokens at 16.9 seconds. A 37% token reduction and 16% latency improvement. Code gives you instrumentation points. A prompt can’t reliably A/B test itself.

Quality metrics over time. Track source diversity across runs. Track gate pass rates. Track how often REASSESS actually triggers a refinement cycle now that it’s enforced. Build a picture of whether your methodology is improving, or whether you’re running the same process and hoping for better results.

Each layer enables the next, and you can’t skip ahead. Without code, no logs. Without logs, no metrics. Without metrics, no budgets. Without budgets, you can’t run a pipeline unattended and trust that it won’t burn through $50 on a $3 task. Without A/B testing, you don’t know whether Sonnet at a third of the cost produces 90% of the quality of Opus for extraction. It does.

None of this is possible when the orchestration lives inside a prompt. The LLM is a black box: what went in, what came out, nothing in between. Code makes the journey visible, measurable, and controllable. That’s what takes you from “I think my pipeline works” to “I know exactly how well it works, what it costs, and where to improve it.”

I’m not arguing that prompts are bad. I’m arguing they’re the wrong abstraction for a specific class of problem.

Use LLM calls when the task requires judgment, reasoning, or NLP. When the input is ambiguous or unstructured. When you’re asking “what should we do?” or “what does this mean?” Query generation, relevance scoring, content extraction, synthesis: genuine thinking tasks. The model is excellent at them.

Use code when the task has a deterministic sequence. When you need enforcement: gates, timeouts, retries, budgets. When you need state that survives crashes. When you need observability. When you need to integrate external APIs with proper error handling. When you need different models for different subtasks. When you need cost control.

The tell is simple. If you’re writing “must,” “always,” or “never” in a prompt, you’re describing a constraint. Constraints belong in code. If you’re writing “consider,” “evaluate,” or “determine,” you’re describing a judgment call. Judgment calls belong in prompts.

The temptation is that the models are so good it feels like you can prompt your way through anything. Claude Opus can follow a 1,099-line methodology. Most of the time. But “most of the time” is about 80%, and the 20% failure mode is silent, unrecoverable, and invisible. The model doesn’t crash. It doesn’t throw an error. It quietly takes the path of least resistance: skips a gate, blows past a budget, fabricates a timestamp, freelances a re-search. The output reads well. You’d never know unless you had the instrumentation to check.

Code gives you 100% reliability on the deterministic parts. That frees the model to focus on the 45% where it genuinely shines, with focused prompts a fraction of the size, and gives you full control over everything else.

Here’s what I don’t want to lose: the prompt version was the right starting point. I wouldn’t have gotten to the code pipeline without it.

Having the methodology in plain English helped me think through the phases. I could iterate by editing prose. I could change the TRIAGE criteria in five minutes by rewriting a paragraph. I could add a new stage, CHALLENGE was a late addition, by describing it in a few sentences and seeing if the model could execute it. I could involve the model in the design itself, asking it to review the methodology and flag gaps.

Prose is the right medium for design. You think in sentences. You catch logical gaps when you try to explain a process in plain language. The prompt forced me to articulate what I wanted at each stage, what “good” looked like, and what should happen when things went wrong. That clarity is what made the code version possible.

But there’s a moment when the prototype works and you need it to work reliably. When you need observability, not just output. When you need to trust the process, not just the result. That’s the graduation from prompt to code. A graduation, not a rejection. The prompt did its job. Now it’s a specification, not a runtime.

The first end-to-end run of the code pipeline produced 154 results from 39 out of 40 queries, extracted 233 findings from 16 sources, ran map-reduce synthesis across parallel chunks, and wrote a 47,000-character report. When four sources were unreachable, it told me which ones and why. I know exactly what that run consumed, how long each stage took, and where to improve it next time.

The old version could have produced something similar. On a good day, with a following wind, if no URLs hung and the model felt like following all the instructions. And I would have known nothing about how it got there.

That’s the difference between a prompt and a workflow. One describes what you want. The other makes it happen.