GitHub

Provider-agnostic autonomous agent loop for JavaScript. Built on the Vercel AI SDK, it drives any LanguageModel through a tool-use loop until the task is complete.

⚠️ No sandbox

agent-do is not sandboxed within its working directory: when file tools are enabled, the agent can read, write, edit, and delete files under that directory.

By default, file tools operate on files reachable from the working directory (--cwd, defaulting to the current directory). Path-traversal guards prevent the agent from escaping that root, but within that scope a misbehaving prompt or unintended tool call can cause permanent data loss.

--read-only blocks writes, deletes, and edits — but the agent can still read, list, and grep every file in the working directory and send its contents to the model provider. If a directory contains secrets you don't want exposed, use --no-tools instead.

The CLI prints a one-line warning to stderr on every run that has file tools enabled, so the blast radius is visible up front. The warning text adapts to the resolved configuration (read-only vs full read/write), and a saved agent's noTools / readOnly settings win over CLI flags so you don't get a misleading "no tools" warning when a saved config silently re-enables them.

Before using agent-do, especially the CLI:

  • Understand what the agent will do before it runs — review the task and system prompt carefully.
  • Run with --read-only to prevent writes, deletes, and edits while still letting the agent reason about files.
  • Run with --no-tools to disable all file access entirely (including reads).
  • Always work in a directory you are comfortable giving the agent full access to.
  • Keep important files backed up or under version control.

There is no undo. Proceed with caution.

Features

  • Provider-agnostic -- works with any Vercel AI SDK LanguageModel (OpenAI, Anthropic, Google, Mistral, Ollama, etc.)
  • Autonomous loop -- calls tools, reads results, and continues until the model responds without tool calls
  • Streaming and non-streaming -- stream() yields ProgressEvents as an AsyncIterable; run() returns the final text
  • Built-in tool factories -- createMemoryTools (private scratchpad), createWorkspaceTools (project files + deny-list, optionally sandboxed), createShellTool (sandbox-mediated bash)
  • Skills system -- install, search, and manage skill definitions that extend the agent's system prompt
  • Lifecycle hooks -- intercept tool calls, track steps, modify arguments, or halt execution
  • Permission system -- accept-all, deny-all, or ask mode with per-tool overrides
  • Usage tracking -- built-in cost estimation for 50+ models with per-run and per-day spending limits
  • Testable -- createMockModel() returns a mock LanguageModel with predetermined responses
  • Eval framework -- defineEval() + runEvals() to measure agent quality with 13 assertion types, LLM-as-judge, and multi-provider comparison

Install

npm install agent-do

Requires Node.js 22 or newer — the floor set by AI SDK 7.

Peer dependency: ai (Vercel AI SDK v7+).

The CLI ships with @ai-sdk/anthropic, @ai-sdk/google, and @ai-sdk/openai bundled so npx agent-do works out of the box. These are declared as optional peers for library consumers — if you only use one provider, npm won't complain about the others being missing, but the CLI covers them all.

Using a different provider

The CLI knows about anthropic, google, openai, ollama, and zai. For any other provider (Mistral, Groq, Cohere, OpenRouter, Bedrock, xAI, etc.), install the SDK and use agent-do as a library:

npm install agent-do @ai-sdk/mistral
import { createAgent } from 'agent-do';
import { createMistral } from '@ai-sdk/mistral';
const agent = createAgent({
  model: createMistral()('mistral-large-latest'),
});
await agent.run('your task');

Any Vercel AI SDK LanguageModel works — see sdk.vercel.ai/providers for the full list.

Z.ai (GLM)

The CLI supports Z.ai (Zhipu AI's GLM models) out of the box via its OpenAI-compatible endpoint — no extra install:

export ZAI_API_KEY=...           # get one at https://z.ai/
npx agent-do --provider zai --model glm-4.6 "Explain CRDTs"

zai reuses the bundled @ai-sdk/openai against https://api.z.ai/api/paas/v4, so it adds no dependency. To target the GLM Coding Plan endpoint instead, set ZAI_BASE_URL=https://api.z.ai/api/coding/paas/v4.

As a library, the same thing with any OpenAI-compatible client:

import { createAgent } from 'agent-do';
import { createOpenAI } from '@ai-sdk/openai';
const zai = createOpenAI({
  baseURL: 'https://api.z.ai/api/paas/v4',
  apiKey: process.env.ZAI_API_KEY!,
  name: 'zai',
});
const agent = createAgent({ model: zai('glm-4.6') });
await agent.run('your task');

See examples/19-zai.ts for a runnable example.

CLI

Run agents from the command line with zero config:

# One-shot task
npx agent-do "What is TypeScript?"
# Pipe content as context
cat README.md | npx agent-do "Summarize this"
# Pipe + prompt merged
echo "function add(a, b) { return a + b }" | npx agent-do "Review this code"
# Interactive chat
npx agent-do
# Choose provider and model
npx agent-do --provider google --model gemini-3.6-flash "Hello"
# Create a reusable agent
npx agent-do create code-reviewer --provider anthropic --system "Review code for bugs"
# Run a saved agent by name
npx agent-do run code-reviewer "Review this function"
# List saved agents
npx agent-do list
# Run a custom agent script (.js/.mjs/.cjs/.ts files).
# --script is required to import local JavaScript/TypeScript; see
# "Script mode" below for the security reasoning.
npx agent-do run ./my-agent.ts --script "Do something"
# Run eval cases
npx agent-do eval evals/basic.ts
# Compare providers
npx agent-do eval evals/ --compare anthropic,google,openai --output json

CLI options

npx agent-do [options] [prompt]          One-shot or interactive
npx agent-do create <name> [options]     Create a reusable agent config
npx agent-do list                        List saved agents
npx agent-do run <name|file> [task]      Run a saved agent OR a script file
npx agent-do eval <file|dir> [options]   Run evals
npx agent-do scheduled-tasks <verb>      Cron-driven runs (run|list|status|install|start)
Options:
  --provider <name>      anthropic | google | openai | ollama | zai (default: anthropic)
  --model <id>           Model ID. Defaults: anthropic claude-sonnet-5,
                         google gemini-3.6-flash, openai gpt-5.6-terra,
                         ollama llama3.2, zai glm-4.6
  --system <prompt>      System prompt
  --cwd <dir>            Working directory for workspace tools (default: cwd)
  --memory <dir>         Memory directory for --with-memory (default: .agent-do/)
  --with-memory          Enable memory tools (agent scratchpad)
  --read-only            Block all writes (workspace + memory)
  --exclude <globs>      Extra deny-list patterns (comma-separated, gitignore-style)
  --include-sensitive    Bypass built-in sensitive-file deny list (.env, .ssh, etc.)
  --max-iterations <n>   Max loop iterations (default: 20)
  --no-tools             Disable all file tools
  --verbose              Show per-step thinking + tool summaries (stderr)
  --show-content         With --verbose: also include each tool's full result
  --log-level <level>    silent | info | verbose | debug | trace (default: info).
                         Overrides --verbose / --show-content. See
                         "Debugging & Observability" below.
  --script               Required for `run <path>` to import local JS/TS files
  -y, --yes              Skip the interactive confirmation for --script
  --provider-tool <name> Enable a provider-native tool (repeatable). See below.
  --provider-options <json>
                         Provider-specific options forwarded to every model
                         call, e.g. '{"google":{"useSearchGrounding":true}}'.
  --json                 JSON output
  --output <fmt>         console | json | csv (eval only)
  --compare <providers>  Compare providers (eval only, comma-separated)
  --concurrency <n>      Parallel eval cases (default: 1)

Provider-native tools and options

Provider packages (@ai-sdk/google, @ai-sdk/anthropic, @ai-sdk/openai) ship server-side tools — Google search-with-grounding, Anthropic web search, OpenAI web search, code execution, URL context, etc. — and provider-specific call options like Google's useSearchGrounding, Anthropic's thinking, or OpenAI's reasoningEffort. Both are exposed on the CLI:

# Google search grounding via the provider tool
npx agent-do "what won the F1 race last weekend?" \
  --provider google \
  --provider-tool googleSearch
# Plus provider options on the model call itself
npx agent-do "summarize https://example.com" \
  --provider google \
  --provider-tool urlContext \
  --provider-options '{"google":{"useSearchGrounding":true}}'
# Anthropic web search (alias `webSearch` resolves to the latest dated tool)
npx agent-do "find recent papers on diffusion models" \
  --provider anthropic --provider-tool webSearch
# OpenAI web search
npx agent-do "latest research on graph neural networks" \
  --provider openai --provider-tool webSearch

--provider-tool is repeatable and can take comma-separated values (--provider-tool googleSearch,urlContext). A small alias table maps short names to the latest dated versions: webSearch → webSearch_20260209, bash → bash_20250124, etc.

The CLI only accepts tools that work with empty config — currently googleSearch, urlContext, codeExecution, webSearch, webFetch, bash, textEditor, computer, memory, webSearchPreview, codeInterpreter, imageGeneration, applyPatch. Tools that need per-tool args (fileSearchvectorStoreIds, mcp → server URL, customTool → name/description, …) must be configured from a script export so you can pass real args. Trying to enable one from the CLI fails fast with a copy-pasteable script-mode snippet.

The same fields are first-class on saved agents and AgentConfig:

npx agent-do create researcher \
  --provider google --model gemini-3.6-flash \
  --system 'Research topics' \
  --provider-tool googleSearch --provider-tool urlContext \
  --provider-options '{"google":{"useSearchGrounding":true}}'
npx agent-do researcher "summarize the latest Mars helicopter mission"

In a script (Format 2), set them on the exported AgentConfig:

import { google } from '@ai-sdk/google';
export default {
  id: 'researcher', name: 'researcher',
  model: google('gemini-3.6-flash'),
  tools: {
    google_search: google.tools.googleSearch({}),
    url_context: google.tools.urlContext({}),
  },
  providerOptions: { google: { useSearchGrounding: true } },
};

Script mode: run <path> --script

agent-do run <arg> resolves saved-agent names by default. To run a local JavaScript or TypeScript file as an agent, pass --script explicitly:

npx agent-do run ./my-agent.ts --script "Do something"

Importing an arbitrary JS/TS file runs its top-level code with your user privileges — the same trust model as running any local script. The --script flag is a deliberate speed bump so that:

  • A missed saved-agent lookup (typo, stale name) fails with a clear error instead of silently import()-ing a stray file that happens to match the name.
  • Social-engineering vectors like "download this helper script, then run agent-do run helper.js" require an explicit opt-in.

When --script is passed:

  • The path must point inside --cwd (symlinks and .. escapes are rejected after canonicalisation).
  • Only .js/.mjs/.cjs/.ts/.mts/.cts extensions are allowed.
  • The file must be a regular file (no directories, no special files).
  • A banner prints the path, size, and SHA-256 prefix, then asks Continue? [y/N]. Pass -y/--yes to skip the prompt (required in non-TTY contexts — CI, piped input — since there's nowhere to type the answer).

Tools: workspace vs memory

agent-do splits file access into two distinct concepts so the agent knows whether it's touching your project or its own notes:

  • Workspace tools (read_file, write_file, list_directory, grep_file, find_files, edit_file, delete_file) are enabled by default and rooted at --cwd (defaults to the current directory). This is what most CLI users want — the agent reads and modifies real project files.
  • Memory tools (memory_read, memory_write, memory_list, memory_delete, memory_search) are opt-in via --with-memory. They give the agent a private, per-agent scratchpad under --memory (default .agent-do/). Use memory when you want the agent to remember notes or plans across runs without scribbling on the project.

Both respect --read-only. To disable all file access, use --no-tools.

Sensitive-file deny list

Workspace tools ship with a gitignore-style deny list that blocks access to credential material by default:

  • Reads blocked: .env*, *.pem, *.key, id_rsa*, id_ed25519*, .ssh/**, .aws/**, .gcloud/**, .kube/**, .git/objects/**, .git/hooks/**.
  • Writes blocked (above plus): .git/**, node_modules/**.

Reads of node_modules/** and .git/HEAD are allowed — the agent can inspect dependencies and branch state but cannot silently rewrite git hooks or clobber installed modules.

Layer your own policy on top:

  • --exclude 'secrets/**,*.cred' — per-invocation patterns.
  • .agent-doignore at the workspace root — project-scoped, gitignore- style file. Merged with the defaults.
  • --include-sensitive — opt out of the built-in defaults when you explicitly want the old fully-open behaviour. .agent-doignore and --exclude still apply.

Blocked operations surface in --verbose logs as [blocked] entries with the matched rule; the model sees only that it was blocked (not the rule name) to avoid letting it probe the policy.

Tool result layering: model vs user vs programmatic

Every built-in tool returns a structured ToolResult with three views:

  • modelContent — the string the LLM sees. File contents are wrapped in <tool_output tool="…" path="…">…</tool_output> markers, capped at 256 KB (maxReadBytes), and common prompt-injection markers (ignore previous instructions, <system> tags) are replaced with a visible [redacted prompt-injection marker].
  • userSummary — a one-liner for operator logs. Includes real paths, byte counts, line counts, match counts, block reasons, errno codes.
  • data — structured fields (path, bytes, lines, truncated, redactedMarkerCount, matchCount, hiddenByDenyList, rule, …) for programmatic consumers.

In --verbose CLI mode you see the userSummary + a compact data line on stderr. Full raw tool output is withheld by default so secrets and large file contents don't leak into CI logs; pass --show-content to include it.

Library consumers who need the full raw payload on tool-result progress events pass emitFullResult: true in AgentConfig:

const agent = createAgent({ /* ... */, emitFullResult: true });
for await (const event of agent.stream('review the code')) {
  if (event.type === 'tool-result') {
    console.log(event.summary);            // always present
    console.log(event.data);                // structured, always present
    console.log(event.toolResult);          // only when emitFullResult: true
  }
}

Custom tools can return a ToolResult directly for full control:

import { tool } from 'ai';
import type { ToolResult } from 'agent-do';
const myTool = tool({
  description: 'Do a thing',
  inputSchema: z.object({ path: z.string() }),
  execute: async ({ path }): Promise<ToolResult> => ({
    modelContent: 'Short sanitised view for the model',
    userSummary: `[my_tool] ${path} — did a thing`,
    data: { path, widgetCount: 42 },
  }),
});

String returns still work and are normalised automatically.

Piping

Piped stdin is merged with the command-line prompt:

stdin prompt result
no "Hello" Task: "Hello"
"context" no Task: "context"
"context" "Summarize" Task: "Summarize\n\n---\n\ncontext"
no no Interactive mode

Quick Start

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const model = createMockModel({
  responses: [
    { text: 'The capital of France is Paris.' },
  ],
});
const agent = createAgent({
  id: 'geography',
  name: 'Geography Agent',
  model,
});
const result = await agent.run('What is the capital of France?');
console.log(result); // "The capital of France is Paris."

Streaming

stream() returns an AsyncIterable<ProgressEvent> that yields events as the agent works:

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const model = createMockModel({
  responses: [
    { toolCalls: [{ toolName: 'lookup', args: { query: 'Paris' } }] },
    { text: 'Paris is the capital of France.' },
  ],
});
const agent = createAgent({
  id: 'geo',
  name: 'Geo',
  model,
  tools: {
    // ... your tools here
  },
});
for await (const event of agent.stream('Tell me about Paris')) {
  switch (event.type) {
    case 'thinking':
      process.stdout.write(event.content);
      break;
    case 'tool-call':
      console.log(`Calling ${event.toolName}`, event.toolArgs);
      break;
    case 'tool-result':
      console.log(`Result from ${event.toolName}:`, event.toolResult);
      break;
    case 'text':
      console.log('Agent says:', event.content);
      break;
    case 'step-complete':
      console.log(`Step ${event.step! + 1} complete`);
      break;
    case 'done':
      console.log('Final answer:', event.content);
      break;
    case 'error':
      console.error('Error:', event.content);
      break;
  }
}

ProgressEvent types

Type Description
thinking Partial text streaming from the model
tool-call The model is calling a tool (toolName, toolArgs)
tool-result A tool returned a result (toolName, toolResult)
text Final text output for a step
step-complete An iteration of the loop finished
done The agent completed its task
error Something went wrong or limits were exceeded

Multiple Agents

Create multiple agents with different models, tools, and system prompts:

import { createAgent } from 'agent-do';
import { createAnthropic } from '@ai-sdk/anthropic';
const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const assistant = createAgent({
  id: 'assistant',
  name: 'Assistant',
  model: anthropic('claude-sonnet-5'),
  systemPrompt: 'You are a helpful assistant.',
});
const researcher = createAgent({
  id: 'researcher',
  name: 'Researcher',
  model: anthropic('claude-haiku-4-5'), // cheaper model for research
  systemPrompt: 'You are a research assistant. Be thorough.',
});
// Each agent has its own conversation context
const answer = await assistant.run('Hello!');
const research = await researcher.run('Find info about TypeScript');

Slash Commands

Deterministic pre-model dispatch keyed on the first token of the task. When a user's input starts with /<name>, the loop routes the remainder to a configured sub-agent before any model call on the parent — zero LLM tokens, zero tool round-trips. Contrast with the orchestrator's delegate_task, where the model decides to delegate.

import { createAgent } from 'agent-do';
const parent = createAgent({
  id: 'parent',
  name: 'Parent',
  model,
  slashCommands: {
    research: createAgent({ id: 'research', name: 'Researcher', model: cheapModel, /* own tools/skills */ }),
    review:   createAgent({ id: 'review',   name: 'Reviewer',   model }),
  },
});
await parent.run('/research quantum cryptography');
//   → runs the `research` sub-agent with task 'quantum cryptography'
await parent.run('/review');
//   → runs `review` with empty args
await parent.run('/ship');
//   → Unknown slash command "/ship". Available commands: /research, /review.
//     (no model called — the listing is returned as the final text)
await parent.run('what is the weather?');
//   → parent handles it as normal (non-slash input bypasses the router)

Keys must match /^[a-zA-Z0-9_-]+$/; values must be Agent instances. Sub-agents are full agents, so each carries its own model, tools, skills, routines, permissions, and hooks. Nested slash commands (/a/b) are not supported — dispatch is a single deterministic hop, validated at createAgent() time.

A path-shaped task like /etc/hosts is not treated as a command (the name must be command-shaped), so it falls through to the parent.

The CLI passes input through unchanged, so npx agent-do run <agent> "/research quantum cryptography" routes correctly. Parent conversation history is not forwarded to the sub-agent by default — the sub-agent starts a fresh turn with just the remainder (and the same context, if any).

parseSlashCommand(input) and unknownSlashCommandMessage(name, commands) are exported for callers that want to inspect or build routing themselves.

Shellm Scripts

A shellm file is a plain-text prompt you run like a shell script — "shell" + "LLM". Drop a prompt (optionally with a shebang and YAML frontmatter) into a file, make it executable, and run it directly:

$ cat > weekly-summary.shellm <<'EOF'
#!/usr/bin/env agent-do
Summarize the git log from the last week and list any breaking changes.
EOF
$ chmod +x weekly-summary.shellm
$ ./weekly-summary.shellm

The kernel turns the shebang into agent-do /abs/path/weekly-summary.shellm, so the file path arrives as the CLI's first positional. Prompt mode detects it and uses the file body as the prompt — zero boilerplate, no imports, no config objects.

Opt-in detection

A positional is treated as a shellm file only when it opts in via one of:

  • a .shellm extension (agent-do ./analyze.shellm), or
  • an agent-do shebang on the file's first line (#!/usr/bin/env agent-do).

This dual gate keeps agent-do readme.md meaning "the literal prompt readme.md" (today's behaviour) instead of silently reading the file. Both signals work alone — a .shellm file needs no shebang, and a shebanged file with any extension works too.

Frontmatter config

Optional YAML frontmatter sets the run config:

$ cat > review.shellm <<'EOF'
#!/usr/bin/env agent-do
---
provider: google
model: gemini-3.6-flash
system: You are a code reviewer. Be terse.
---
Review the staged git diff for bugs.
EOF
$ ./review.shellm

Frontmatter values override the CLI defaults for provider, model, and system. Flags like --read-only, --verbose, and --no-tools still apply (they have no frontmatter twin).

Piping

Stdin is merged with the file prompt exactly as in echo ctx | agent-do "prompt" — the file is the instruction, piped data is the context:

$ cat data.csv | ./analyze.shellm          # analyze the piped CSV
$ git diff --staged | ./review.shellm > review.md   # in a pipeline

Trust model

A shellm file is data (a prompt), not code — parsing it never import()s anything, so it's strictly safer than agent-do run x.ts --script. But its contents reach the model, and with the default workspace tools the agent can read/write/edit files in --cwd. Don't run shellm files from sources you don't trust — the same rule as shell scripts. --read-only and --no-tools narrow the blast radius.

See examples/22-shellm.shellm for a runnable sample with frontmatter.

Scheduled Tasks

Many useful agents don't run on-demand — they run on a schedule. Inbox sweep every 15 min, daily brief at 8am, weekly digest on Sunday. agent-do scheduled-tasks turns a cron entry into a safe, overlap-proof agent run. The production wiring is system cron: install emits the crontab lines, and the system cron invokes run <id> on schedule.

Define tasks

Tasks live in a JSON file (<memoryDir>/scheduled-tasks.json by default, overridable with --tasks <path>):

[
  {
    "id": "inbox-triage",
    "cron": "*/15 8-21 * * *",
    "payload": "Read inbox.md and move anything actionable into tasks.md. If there's nothing new, reply with just OK.",
    "description": "inbox sweep every 15 min during work hours"
  },
  {
    "id": "daily-brief",
    "cron": "0 8 * * 1-5",
    "payload": "Summarise this week's tracker.md into a three-line daily brief.",
    "description": "weekday 8am brief"
  }
]

cron is standard 5-field (minute hour dom month dow; 0 and 7 both Sunday). The portable numeric subset is supported — */n, ranges, comma lists, and Vixie dom/dow OR-semantics. Month/weekday names and L/W/# are not (support varies by platform cron).

CLI

npx agent-do scheduled-tasks list                          # show configured tasks
npx agent-do scheduled-tasks status                        # last-run times + outcomes
npx agent-do scheduled-tasks install                       # emit crontab lines to stdout
npx agent-do scheduled-tasks run inbox-triage              # run one task now (lock-protected)
npx agent-do scheduled-tasks start                         # foreground loop (dev/test)

install prints one crontab line per task that calls agent-do scheduled-tasks run <id>. Append the output to your crontab (crontab -e) or a systemd timer — the system cron does the timing, agent-do does the safe execution:

*/15 8-21 * * * /usr/bin/env agent-do scheduled-tasks run inbox-triage --tasks "/abs/tasks.json" --memory "/abs/.data"

Overlap safety

Every run acquires the #15 file lock before executing, so overlapping firings skip (cron fired before the previous run finished, or two hosts share the same --memory dir). A run killed mid-execution leaves a lock that the next firing reclaims after staleMs (default 15 min) — exactly the cross-process guarantee a scheduler needs and an in-process mutex can't provide. Last-run times, outcomes, and run counts are recorded in <memoryDir>/scheduler-status.json.

Library API

import { createAgent } from 'agent-do';
import { matchesCron, runScheduledTask } from 'agent-do';
const agent = createAgent({
  /* ... */
  scheduledTasks: [
    { id: 'sweep', cron: '*/15 * * * *', payload: 'triage inbox' },
  ],
});
// Drive a run yourself (the lock + status recording are reusable):
await runScheduledTask('sweep', {
  memoryDir: './.data',
  statusFile: './.data/scheduler-status.json',
  payload: 'triage inbox',
  run: (text) => agent.run(text),
});
matchesCron('*/15 * * * *', new Date()); // → boolean, for your own scheduler

sessionTarget: 'main' (persistent conversation) and wakeMode: 'systemEvent' (structured wake event) are accepted in the schema but reserved in v1 — every run is isolated, and the payload is treated as the task text. They're noted follow-ups once agent-do ships a history store and an event channel.

See examples/scheduled-tasks/tasks.json for a runnable task file.

Tools

Define tools using the Vercel AI SDK's tool() function:

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
import { tool } from 'ai';
import { z } from 'zod';
const agent = createAgent({
  id: 'math',
  name: 'Math Agent',
  model: createMockModel({
    responses: [
      { toolCalls: [{ toolName: 'add', args: { a: 2, b: 3 } }] },
      { text: 'The sum of 2 and 3 is 5.' },
    ],
  }),
  tools: {
    add: tool({
      description: 'Add two numbers',
      inputSchema: z.object({
        a: z.number(),
        b: z.number(),
      }),
      execute: async ({ a, b }) => `${a + b}`,
    }),
  },
});
const result = await agent.run('What is 2 + 3?');

The agent loops automatically: it calls tools, feeds results back to the model, and continues until the model responds with text only (no tool calls) or hits maxIterations (default: 20).

History hygiene

Between iterations, the loop replaces older <tool_output>...</tool_output> blocks in the conversation history with self-closing redacted="stale" markers. This keeps the model's view of what happened (which tool ran, on what path) but drops the body, so injected content from a poisoned file can't keep influencing the model on every subsequent step. It also keeps token spend bounded as iterations accumulate.

Tune the window with AgentConfig.historyKeepWindow (default 1 — only the most recent iteration's tool outputs flow in full to the next call). Set to Infinity to restore the historical "everything-stays-in-context" behaviour.

Tool factories

Three consumer-facing factories cover the common cases. See docs/sandbox.md for how each interacts with a sandbox.

Workspace tools

createWorkspaceTools(workingDir, opts?) gives the agent file tools (read_file, write_file, edit_file, list_directory, delete_file, grep_file, find_files) rooted at workingDir, with a deny-list (.env, .ssh/**, ...) applied at the tool layer. Pass { sandbox } to swap the internal store for a SandboxBackedMemoryStore and route every file op through the sandbox.

import { createAgent, createWorkspaceTools } from 'agent-do';
const agent = createAgent({
  id: 'coder', name: 'Coder', model,
  tools: createWorkspaceTools(process.cwd(), { readOnly: true }),
});

Memory tools

createMemoryTools(store, agentId, opts?) gives the agent a private scratchpad (memory_read, memory_write, memory_list, memory_delete, memory_search) backed by any MemoryStore.

import {
  createAgent, createMemoryTools, InMemoryMemoryStore,
} from 'agent-do';
const store = new InMemoryMemoryStore();
const agent = createAgent({
  id: 'writer', name: 'Writer', model,
  tools: createMemoryTools(store, 'writer'),
});

Shell tool

createShellTool(sandbox?, opts?) gives the agent a single shell tool (default name bash) wired to a SandboxApi. Defaults to createHostSandbox() when no sandbox is supplied; for real isolation, pass createJustBashSandbox() or your own connector.

import { createAgent, createShellTool, createJustBashSandbox } from 'agent-do';
const sandbox = await createJustBashSandbox();
const agent = createAgent({
  id: 'runner', name: 'Runner', model,
  tools: createShellTool(sandbox),
});

MemoryStore

The MemoryStore interface abstracts file storage for agents. Three implementations are included:

  • InMemoryMemoryStore — for testing and prototyping (data lost on exit)
  • FilesystemMemoryStore — persists to the local filesystem (survives restarts)
  • SandboxBackedMemoryStore — adapts a SandboxApi connector into a MemoryStore (see docs/sandbox.md)
import { FilesystemMemoryStore, createMemoryTools, createAgent } from 'agent-do';
const store = new FilesystemMemoryStore('./agent-data');
const agent = createAgent({
  id: 'my-agent',
  name: 'My Agent',
  model: model as any,
  tools: createMemoryTools(store, 'my-agent'),
});
// Files persist at ./agent-data/my-agent/

Security: FilesystemMemoryStore

Warning: FilesystemMemoryStore gives the agent read/write access to the specified directory. The agent decides what files to create and modify. Use readOnly: true to restrict to read-only access, or onBeforeWrite to approve each write operation.

// Read-only mode — agent can read but not create/modify/delete
const readOnlyStore = new FilesystemMemoryStore('./data', { readOnly: true });
// Write confirmation — approve each operation (sync or async)
const guardedStore = new FilesystemMemoryStore('./data', {
  onBeforeWrite: (agentId, canonicalPath, operation) => {
    console.log(`Agent ${agentId} wants to ${operation}: ${canonicalPath}`);
    // Return true to allow, false to block
    // The path is canonicalized — ../traversal is resolved before this callback
    return true;
  },
});

Concurrent access (locking, #15)

By default write() is a naive overwrite — fine for single-process use, but concurrent runs lose data: two processes that read-modify-write the same file clobber each other. Set the opt-in lock option to serialise mutating ops across processes:

// Orchestrator workers, overlapping run() calls, or cron-driven runs (#79)
// — all safely serialised on a per-file basis.
const store = new FilesystemMemoryStore('./data', {
  lock: {
    staleMs: 60_000,                              // reclaim locks older than 1 min
    retry: { count: 20, minDelayMs: 25, maxDelayMs: 1000 },
  },
});

The lock is a zero-dependency sidecar file (O_CREAT|O_EXCL + mtime stale reclaim + retry/backoff), visible to every process sharing the same baseDir — so it protects across orchestrator workers and fresh cron-spawned processes, not just within one Node process. Lock files live in <baseDir>/.locks/<agentId>/ and are invisible to the agent's own list() / search() / read(). Writes also become atomic (temp-file + rename), so unlocked readers never see a half-flushed file.

POSIX locks are advisory: a process that writes without the lock option (say, an editor, or a store instance created without it) can still clobber a locked file. undefined preserves today's behaviour byte-for-byte.

Need stronger guarantees on a network filesystem? Plug in a battle-tested lock without agent-do taking a dependency:

import properLockfile from 'proper-lockfile'; // your dep, not agent-do's
const store = new FilesystemMemoryStore('./data', {
  lock: {
    adapter: {
      async acquire(lockFile, opts) {
        // wrap proper-lockfile here; return { async release() {…} }
      },
    },
  },
});

acquireFileLock, withFileLock, and the FileLockOptions / LockAdapter / LockHandle types are exported for direct use. For genuinely-concurrent merge semantics (multiple writers whose edits should combine rather than serialise), a lazily-imported CRDT-backed store (agent-do/stores/crdt, Yjs) is planned as a Tier-2 follow-up.

For other backends, implement the interface:

interface MemoryStore {
  read(agentId: string, path: string): Promise<string>;
  write(agentId: string, path: string, content: string): Promise<void>;
  append(agentId: string, path: string, content: string): Promise<void>;
  delete(agentId: string, path: string): Promise<void>;
  list(agentId: string, path?: string): Promise<FileEntry[]>;
  mkdir(agentId: string, path: string): Promise<void>;
  exists(agentId: string, path: string): Promise<boolean>;
  search(agentId: string, pattern: string, path?: string): Promise<Array<{ path: string; line: string }>>;
}

Custom implementations

See examples/08-custom-memory-store.ts for complete patterns for:

  • Node.js filesystem (fs)
  • AWS S3 (@aws-sdk/client-s3)
  • Google Firestore (@google-cloud/firestore)
  • SQLite (better-sqlite3)

Conversation History

Pass previous conversation turns to maintain context:

import { createAgent, type ConversationMessage } from 'agent-do';
const history: ConversationMessage[] = [];
// First turn
const r1 = await agent.run('My name is Alice', undefined, history);
history.push({ role: 'user', content: 'My name is Alice' });
history.push({ role: 'assistant', content: r1 });
// Second turn — agent remembers the name
const r2 = await agent.run('What is my name?', undefined, history);
// r2 = "Your name is Alice."

Skills

Skills extend an agent's system prompt with additional instructions. They can be installed, removed, searched, and managed through a SkillStore.

Defining a skill

import type { Skill } from 'agent-do';
const skill: Skill = {
  id: 'code-review',
  name: 'Code Review',
  description: 'Reviews code for quality and best practices',
  content: `When reviewing code:
- Check for error handling
- Look for security issues
- Suggest performance improvements`,
};

Using InMemorySkillStore

import { createAgent, InMemorySkillStore } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const skills = new InMemorySkillStore();
await skills.install({
  id: 'code-review',
  name: 'Code Review',
  description: 'Reviews code for quality',
  content: 'When reviewing code, check for errors and suggest improvements.',
});
const agent = createAgent({
  id: 'reviewer',
  name: 'Reviewer',
  model: createMockModel({ responses: [{ text: 'LGTM' }] }),
  skills,
});

When a SkillStore is provided, the agent gets:

  • Installed skill content injected into the system prompt, wrapped in <skill>…</skill> markers with a preamble instructing the model to treat the body as reference data rather than overriding instructions.
  • Auto-generated tools: search_skills, list_skills, remove_skill. The install_skill tool is not exposed by default — see below.

allowSkillInstall (privileged)

The LLM-facing install_skill tool lets the model write skills into the backing SkillStore. Because installed skills get injected into every subsequent run's system prompt, a prompt-injected agent with install access could plant a persistent jailbreak across sessions.

Set allowSkillInstall: true on the agent config to expose install_skill to the model. Default is false — library callers install skills themselves (via skills.install(...)) and the agent only searches / lists / removes them.

const agent = createAgent({
  // ...
  skills,
  allowSkillInstall: true, // opt-in: model can persist new skills
});

Inputs to install_skill are validated by a strict schema (id matches /^[a-zA-Z0-9_-]+$/, content ≤ 8 KB, name ≤ 64 chars, description ≤ 256 chars) regardless of who calls the tool, and any <skill> or </skill> sequences inside the skill body are neutralised before the prompt is rendered so the structural isolation can't be broken from inside.

Parsing SKILL.md files

import { parseSkillMd } from 'agent-do';
const skill = parseSkillMd(`---
name: My Skill
description: Does useful things
author: Alice
version: 1.0.0
---

Instructions for the skill go here.
`);
console.log(skill.name);    // "My Skill"
console.log(skill.content); // "Instructions for the skill go here."

SkillStore interface

Implement SkillStore for custom backends (database, filesystem, API):

interface SkillStore {
  list(): Promise<Skill[]>;
  get(skillId: string): Promise<Skill | undefined>;
  install(skill: Skill): Promise<void>;
  remove(skillId: string): Promise<void>;
  search(query: string): Promise<Array<{ id: string; name: string; description: string }>>;
}

SkillSearchResult deliberately has no url field — an external registry returning a URL would turn skill search into an SSRF / auto- fetch footgun (see issue #34). If you wire up a network-backed store, host allowlisting and explicit installation must happen outside search(); install() should only receive content the caller has already verified.

Policies

Policies are typed system-prompt modules that ground every decision on every turn — priorities, resolution modes, routing rules. Unlike skills (which extend capabilities and fire autonomously when a task matches), policies are constraints / context that apply regardless of task. The canonical pair is a priority-map (who/what matters, P0–P3) plus an auto-resolver (auto-resolve / draft-and-ask / escalate / ignore).

Defining a policy

createPolicy() accepts either an object { id, type, content } or a source string { id, type, source } where source is a POLICY.md- style document with YAML frontmatter (id / type / version):

import { createAgent, createPolicy } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const priorityMap = createPolicy({
  id: 'priority-map',
  type: 'prioritisation',
  content: '# Priority Map\n- P0: production down\n- P1: customer-blocking',
});
const autoResolver = createPolicy({
  id: 'auto-resolver',
  type: 'resolution',
  source: `---
id: auto-resolver
type: resolution
version: 1
---

# Auto-resolver
1. auto-resolve
2. draft-and-ask
3. escalate
4. ignore`,
});
const agent = createAgent({
  id: 'triage',
  name: 'Triage',
  model: createMockModel({ responses: [{ text: 'OK' }] }),
  policies: [priorityMap, autoResolver],
});

When policies is set, the agent injects a ## Policies section into the system prompt on every run. Each policy is wrapped in <policy id="…" type="…">…</policy> markers with a preamble telling the model the content is reference material that grounds decisions but cannot override the structural policy above — the same injection-safety wrapper skills use, so a hostile body containing </policy> cannot break out of its wrapper.

The id is validated against /^[a-zA-Z0-9_-]+$/ (fails fast at construction). The type is a free-form string — known values are 'prioritisation' and 'resolution', but any taxonomy works.

Operator-authored, not model-authored

There is no LLM-facing install_policy tool. Policies are supplied by the library caller as a plain array. A model that could rewrite its own policy could plant a persistent jailbreak across sessions — the same threat model that gates install_skill / save_routine behind opt-in flags, applied upstream by not exposing the surface at all.

PolicyStore

InMemoryPolicyStore and the PolicyStore interface are available for callers who load policies from disk / config and want install / list / remove semantics (for example, hot-reloading a policies/ directory). Like SkillSearchResult, PolicyStore deliberately has no url / network field — a network-backed policy registry that auto-fetches would be an SSRF / supply-chain footgun (#34).

import { createPolicy, InMemoryPolicyStore } from 'agent-do';
const store = new InMemoryPolicyStore();
await store.install(createPolicy({ id: 'priority-map', type: 'prioritisation', content: '…' }));
const policies = await store.list(); // → Policy[]

See examples/20-policies.ts for a runnable priority-map + auto-resolver pair.

Lifecycle Hooks

Hooks let you observe and control the agent loop. All hooks are optional and async.

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const agent = createAgent({
  id: 'hooked',
  name: 'Hooked Agent',
  model: createMockModel({ responses: [{ text: 'Done.' }] }),
  hooks: {
    // Called before each tool execution. Return a HookDecision to allow/deny/modify.
    onPreToolUse: async ({ toolName, args, step }) => {
      console.log(`Step ${step}: about to call ${toolName}`);
      // Return { decision: 'deny', reason: 'not allowed' } to block
      // Return { decision: 'allow', modifiedArgs: { ... } } to modify input
      return { decision: 'allow' };
    },
    // Called after each tool execution.
    onPostToolUse: async ({ toolName, args, result, step, durationMs }) => {
      console.log(`${toolName} took ${durationMs}ms`);
    },
    // Called at the start of each loop iteration. Return 'stop' to halt.
    onStepStart: async ({ step, totalSteps, tokensSoFar, costSoFar }) => {
      if (costSoFar > 1.0) {
        return { decision: 'stop', reason: 'Too expensive' };
      }
    },
    // Called after each loop iteration completes.
    onStepComplete: async ({ step, hasToolCalls, text }) => {
      console.log(`Step ${step} done, has tools: ${hasToolCalls}`);
    },
    // Called when the entire run finishes.
    onComplete: async ({ result, totalSteps, usage, aborted }) => {
      console.log(`Finished in ${totalSteps} steps, cost: $${usage.totalCost.toFixed(4)}`);
    },
    // Called after each step's usage is recorded.
    onUsage: async (record) => {
      console.log(`Step ${record.step}: ${record.inputTokens}in/${record.outputTokens}out, $${record.estimatedCost.toFixed(4)}`);
    },
  },
});

HookDecision

Returned from onPreToolUse and onStepStart:

interface HookDecision {
  decision: 'allow' | 'deny' | 'ask' | 'stop' | 'continue';
  reason?: string;
  modifiedArgs?: unknown; // Only for onPreToolUse: replace the tool's input
}

Permissions

Control which tools the agent can call.

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const agent = createAgent({
  id: 'safe',
  name: 'Safe Agent',
  model: createMockModel({ responses: [{ text: 'Done.' }] }),
  permissions: {
    // Base mode: 'accept-all' | 'deny-all' | 'ask'
    mode: 'ask',
    // Per-tool overrides: 'always' | 'ask' | 'never'
    tools: {
      read_file: 'always',   // Always allowed, even in deny-all mode
      delete_file: 'never',  // Always blocked, even in accept-all mode
      write_file: 'ask',     // Falls through to onPermissionRequest
    },
    // Called when mode is 'ask' or a tool's level is 'ask'
    onPermissionRequest: async ({ toolName, args }) => {
      console.log(`Allow ${toolName}?`, args);
      return true; // or false to deny
    },
  },
});

Permission evaluation order

  1. If mode is accept-all, allow (but still check per-tool never overrides)
  2. If mode is deny-all, deny (but still check per-tool always overrides)
  3. Check per-tool override: always -> allow, never -> deny
  4. If ask or no override: call onPermissionRequest (defaults to allow if no callback)

Usage Tracking

Track token usage and costs across agent runs with built-in pricing for 50+ models.

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
const agent = createAgent({
  id: 'tracked',
  name: 'Tracked',
  model: createMockModel({ responses: [{ text: 'Hi' }] }),
  usage: {
    enabled: true,
    limits: {
      perRun: 0.50,  // $0.50 max per run
      perDay: 5.00,  // $5.00 max per day
    },
    // Called when a limit is exceeded. Return true to continue anyway.
    onLimitExceeded: async ({ type, spent, limit }) => {
      console.warn(`${type} limit exceeded: $${spent.toFixed(2)} / $${limit.toFixed(2)}`);
      return false; // stop the run
    },
    // Optional: override built-in pricing
    pricing: {
      'my-custom-model': { input: 1.0, output: 3.0 }, // per 1M tokens
    },
  },
});

UsageTracker class

For standalone usage tracking outside of createAgent:

import { UsageTracker, estimateCost, DEFAULT_PRICING } from 'agent-do';
const tracker = new UsageTracker({
  perRunLimit: 1.0,
});
// Record a step
const record = tracker.record(0, 'claude-sonnet-5', 1000, 500);
console.log(record.estimatedCost); // cost based on built-in pricing
// Get summary
const summary = tracker.getSummary();
console.log(summary.totalCost, summary.totalInputTokens, summary.totalOutputTokens);
// Check limits
const ok = await tracker.checkLimits(); // false if limit exceeded
// Standalone cost estimation
const cost = estimateCost('gpt-4o', 10000, 5000);

How model IDs resolve to prices

DEFAULT_PRICING is keyed by model ID. A lookup tries an exact match first, then falls back to the longest key that the model ID starts with — so gpt-4.1-mini-2025-04-14 prices as gpt-4.1-mini, not as gpt-4.1. OpenRouter-style IDs (anthropic/claude-sonnet-5) are normalised by stripping the provider prefix.

A model with no matching entry estimates $0 and logs a one-time warning to stderr. That is not a free run — it means cost tracking and both spending limits are disabled for that model. If you use a model agent-do doesn't know, pass your own rates:

usage: { enabled: true, pricing: { 'my-model': { input: 1.0, output: 3.0 } } }

Debugging & Observability

--verbose / --show-content expose what the agent loop sees — text deltas, tool call summaries, streaming events. One layer lower is what's actually crossing the wire to the model provider: the resolved system prompt, the full message list per step, the cache metrics coming back, and every raw stream part. That's the debug surface.

CLI log levels

# Default: final answer + errors
npx agent-do "hello"
# Thinking + tool summaries (what --verbose has always been)
npx agent-do --log-level verbose "hello"
# Legacy: --verbose still works and implies --log-level verbose.
# + system prompt, messages per step, cache metrics, request metadata
npx agent-do --log-level debug "hello"
# + every raw stream part (text-delta, tool-call, finish, …)
npx agent-do --log-level trace "hello"

At debug the CLI emits compact, labelled lines to stderr:

[debug:request] step=0 model=claude-sonnet-5 tools=[read_file,write_file,grep_file]
[debug:messages] step=0 count=2 bytes=1247
[debug:cache] step=0 read=0 write=1198 no-cache=49 out=112 hit=0%
…second iteration benefits from the cache write on the first:
[debug:cache] step=1 read=1198 write=0 no-cache=14 out=87 hit=98%

That last pair is what you want when you're checking whether Anthropic prompt caching is actually firing — read/(read+no-cache) is the hit rate.

API: AgentConfig.debug

Library callers opt in explicitly and control fan-out:

import { createAgent, type DebugEvent } from 'agent-do';
const agent = createAgent({
  id: 'debug-me',
  name: 'Debug Me',
  model,
  systemPrompt: 'You are helpful.',
  debug: {
    systemPrompt: true,  // log the resolved prompt once per run
    messages: true,      // log the message list going into each step
    request: true,       // model id + tool names + providerOptions
    cache: true,         // per-step cache read/write/no-cache tokens
    response: false,     // opt in to raw stream parts
    // Optional: sink to a separate destination in addition to the
    // progress stream. The sink fires *in addition to* the
    // `type: 'debug'` progress events, not instead of them.
    sink: (event: DebugEvent) => {
      myLogger.info(event.channel, event);
    },
    // Cap body size in system-prompt / messages events. Default 16 KB.
    maxBodyBytes: 8 * 1024,
  },
});
for await (const event of agent.stream('task')) {
  if (event.type === 'debug' && event.debug?.channel === 'cache') {
    const hit = event.debug.cacheReadTokens /
      (event.debug.cacheReadTokens + event.debug.noCacheTokens);
    console.log(`cache hit rate: ${(hit * 100).toFixed(1)}%`);
  }
}

Channels

Channel Fires when Payload highlights
system-prompt once per run, before the first model call content, bytes, truncated
messages before each streamText call full messages[], bytes, truncated
request before each streamText call model, toolCount, toolNames, providerOptions
response-part each raw stream part partType always; full part when traceResponseParts: true
cache after each inner model step cacheReadTokens, cacheWriteTokens, noCacheTokens, outputTokens, providerMetadata

Notes

  • response-part at trace only: the full stream is noisy (text-delta fires per token). At debug level the channel emits just the partType for aggregate counting; bump to trace when you need the contents.
  • String model IDs skip middleware: if you passed model: 'gpt-4o' as a string, the AI SDK resolves the provider lazily and the middleware can't attach. The other channels (system-prompt, cache) still fire — they don't depend on middleware. Pass a structured model (createAnthropic()('claude-sonnet-5')) for full coverage.
  • Sink errors are swallowed. A broken sink can't wedge the run. Sync and async throws are both caught.
  • No secret redaction. If you opt into debug and point it at a log file, it's your job to keep that file private. The maxBodyBytes cap exists only to prevent accidental megabyte dumps.

Testing

createMockModel() returns a mock LanguageModel compatible with the Vercel AI SDK. It uses predetermined responses so you can test agent behavior without API keys.

import { createAgent } from 'agent-do';
import { createMockModel } from 'agent-do/testing';
import { tool } from 'ai';
import { z } from 'zod';
// Simulate a multi-step agent run: tool call -> final answer
const model = createMockModel({
  responses: [
    // Step 1: model calls a tool
    { toolCalls: [{ toolName: 'get_weather', args: { city: 'London' } }] },
    // Step 2: model responds with text (ends the loop)
    { text: 'The weather in London is rainy.' },
  ],
  modelId: 'test-model',
  inputTokensPerCall: 100,
  outputTokensPerCall: 50,
});
const agent = createAgent({
  id: 'test',
  name: 'Test',
  model,
  tools: {
    get_weather: tool({
      description: 'Get weather for a city',
      inputSchema: z.object({ city: z.string() }),
      execute: async ({ city }) => `${city}: rainy, 12C`,
    }),
  },
});
const result = await agent.run('Weather in London?');
// result === 'The weather in London is rainy.'

MockModelOptions

Option Default Description
responses (required) Array of MockResponse objects, used in order
modelId 'mock-model' Model ID for logging
provider 'mock-provider' Provider name for logging
inputTokensPerCall 10 Simulated input tokens per call
outputTokensPerCall 20 Simulated output tokens per call

Eval Framework

Define eval cases to measure agent quality, compare providers, and catch regressions.

import { defineEval, runEvals } from 'agent-do/eval';
import { createAnthropic } from '@ai-sdk/anthropic';
const anthropic = createAnthropic();
const suite = defineEval({
  name: 'my-assistant-eval',
  model: anthropic('claude-haiku-4-5'),
  systemPrompt: 'You are a helpful assistant.',
  cases: [
    {
      name: 'knows capitals',
      input: 'What is the capital of France?',
      assert: [
        { type: 'contains', value: 'Paris' },
        { type: 'not-contains', value: 'London' },
      ],
    },
    {
      name: 'saves notes correctly',
      input: 'Save a note that my name is Alice.',
      assert: [
        { type: 'tool-called', tool: 'write_file' },
        { type: 'file-contains', path: 'memories/user.md', value: 'Alice' },
        { type: 'max-steps', max: 5 },
        { type: 'max-cost', maxUsd: 0.05 },
      ],
    },
  ],
});
const result = await runEvals(suite);
// Console output:  ✓ PASS  knows capitals  ($0.0012, 800ms, 1 steps)
//                  ✓ PASS  saves notes correctly  ($0.0035, 2100ms, 2 steps)

Assertion types

Type Description
contains Response text contains a string
not-contains Response does NOT contain a string
regex Response matches a regex pattern
json-schema Response is valid JSON matching a schema
tool-called A specific tool was called during execution
tool-not-called A specific tool was NOT called
tool-args Tool was called with specific arguments (partial match)
file-exists A file was created in the memory store
file-contains A file in the store contains a string
max-steps Agent completed in N or fewer steps
max-cost Agent completed within a cost budget (USD)
llm-rubric Another LLM scores the response against a rubric
custom Custom function receives the full result

Multi-provider comparison

const result = await runEvals(suite, {
  providers: [
    { name: 'anthropic', model: anthropic('claude-sonnet-5') },
    { name: 'google', model: google('gemini-3.6-flash') },
    { name: 'openai', model: openai('gpt-5.6-terra') },
  ],
});
// Prints a comparison table with pass rate, cost, and latency per provider

LLM-as-judge

{
  name: 'explains clearly',
  input: 'Explain quantum computing to a 10 year old',
  assert: [
    {
      type: 'llm-rubric',
      rubric: 'The explanation should be simple, use analogies, avoid jargon.',
      score: 'pass-fail', // or '1-5'
    },
  ],
}

Output formats

// Console output (default)
await runEvals(suite);
// JSON (for CI/dashboards)
await runEvals(suite, { output: 'json' });
// CSV (for spreadsheets)
await runEvals(suite, { output: 'csv' });
// Silent (programmatic use)
const result = await runEvals(suite, { output: 'silent' });

API Reference

Main exports (agent-do)

Export Type Description
createAgent (config: AgentConfig) => Agent Create an agent with run(), stream(), and abort()
runAgentLoop (config, task, context?) => Promise<RunResult> Run the loop directly (lower-level)
streamAgentLoop (config, task, context?) => AsyncGenerator<ProgressEvent> Stream the loop directly (lower-level)
createShellTool (sandbox?, opts?) => ToolSet A single shell tool (default name bash) whose execute calls sandbox.exec. Defaults to host when no sandbox is supplied. (docs/sandbox.md)
SandboxBackedMemoryStore class Adapt a SandboxApi into a MemoryStore
createHostSandbox (opts?) => SandboxApi Direct passthrough to the host — not a security boundary
createJustBashSandbox (opts?) => Promise<SandboxApi> Wrap a vercel-labs/just-bash Sandbox
wrapJustBashSandbox (instance) => SandboxApi Wrap an externally-constructed just-bash instance
createSkillTools (store: SkillStore) => ToolSet Create skill management tools
buildSkillsPrompt (skills: Skill[]) => string Build a system prompt section from skills
parseSkillMd (content, id?) => Skill Parse a SKILL.md with YAML frontmatter
InMemorySkillStore class In-memory reference implementation of SkillStore
InMemoryMemoryStore class In-memory store (testing/prototyping)
FilesystemMemoryStore class Node.js filesystem store (persistent)
createOrchestrator (config) => Orchestrator Create a multi-agent orchestrator
parseSlashCommand (input) => { name, rest } | null Parse a /<name> slash command from a task string
unknownSlashCommandMessage (name, commands) => string Build the "unknown command" listing
validateSlashCommands (commands) => string | null Validate an AgentConfig.slashCommands map
evaluatePermission (toolName, args, config) => Promise<boolean> Evaluate a permission check
UsageTracker class Track usage and costs within a run
estimateCost (model, input, output, pricing?) => number Estimate cost in USD
DEFAULT_PRICING PricingTable Built-in pricing for 50+ models

Test exports (agent-do/testing)

Export Type Description
createMockModel (options: MockModelOptions) => LanguageModel Create a mock model for testing

Key types

Type Description
AgentConfig Full agent configuration (model, tools, hooks, permissions, usage)
Agent Agent instance with id, name, run(), stream(), abort()
ProgressEvent Event emitted during streaming
RunResult Result of run() with text, usage, steps, aborted flag
AgentHooks Lifecycle hook callbacks
PermissionConfig Permission mode, per-tool overrides, callback
Skill / SkillStore Skill definition and storage interface
RunUsage / UsageRecord Usage summary and per-step records
HookDecision Return value from hooks to control execution
PricingTable Model pricing lookup (per 1M tokens)
MemoryStore Storage interface for agent file operations
SandboxApi Pluggable sandbox contract (Flue-shaped) — see docs/sandbox.md
FileStat / ExecOptions / ExecResult Shapes returned by SandboxApi methods
FileEntry File/directory entry from list()
ConversationMessage User/assistant message for conversation history
Orchestrator / OrchestratorConfig Multi-agent orchestration types
BuildSystemPromptOptions Options for the prompt builder
SectionFn Function that returns a prompt section string
PromptTemplate Named template with ordered section list

Eval exports (agent-do/eval)

Export Type Description
defineEval (config: EvalSuiteConfig) => EvalSuiteConfig Define an eval suite (type-safe helper)
runEvals (suite, options?) => Promise<EvalResult> Run an eval suite and return results
evaluateAssertion (assertion, result, judgeModel?) => Promise<AssertionResult> Evaluate a single assertion
EvalSuiteConfig type Eval suite definition (name, model, cases)
EvalCase type Single eval test case (input, assertions)
Assertion type Union of all 13 assertion types
EvalResult type Full eval result with provider breakdowns
CaseResult type Result of a single eval case

Prompt exports (agent-do/prompts)

Export Type Description
buildSystemPrompt function Compose a system prompt from templates, sections, and variables
interpolate function Simple {{variable}} replacement
builtinTemplates object Preconfigured templates: assistant, coder, researcher, reviewer, writer, planner
builtinSections object Reusable sections: identity, memoryManagement, fileTools, efficiency, etc.
roleSections object Role-specific sections: codingApproach, researchApproach, etc.

Store exports (agent-do/stores)

Export Description
MemoryStore Storage interface (type)
FileEntry File entry type

Store implementations

Export Import path Description
InMemoryMemoryStore agent-do In-memory store for testing/prototyping (data lost on exit)
FilesystemMemoryStore agent-do Node.js filesystem store (persistent, path-traversal safe)

Examples

The examples/ directory contains runnable examples:

# File Description
1 01-basic-agent.ts Simplest possible agent
2 02-agent-with-tools.ts Custom tools (weather, calculator)
3 03-agent-with-memory.ts File tools with InMemoryMemoryStore
4 04-lifecycle-hooks.ts Hooks for monitoring and control
5 05-multi-provider.ts Anthropic, Google, OpenAI, Ollama
6 06-conversation-history.ts Multi-turn conversations
7 07-multi-agent-orchestration.ts Master + worker agents
8 08-custom-memory-store.ts Patterns for S3, Firestore, SQLite, filesystem
9 09-skills.ts Skills system
10 10-testing.ts Testing with createMockModel
11 11-filesystem-store.ts Persistent filesystem storage — explore the created files
12 12-prompt-builder.ts Composable system prompts from templates + sections + variables
13 13-eval-framework.ts Eval framework — define cases, assert quality, compare providers
14 14-mcp.ts Mount external MCP servers and expose their tools to the agent
15 15-routines.ts Saved routines — reusable named procedures with {{arg}} interpolation
16 16-sandbox-bash.ts Pluggable sandbox + bash tool (host connector)
17 17-sandbox-with-memory.ts Sandbox alongside InMemoryMemoryStore (different substrates)
18 18-sandbox-with-filesystem.ts Sandbox alongside FilesystemMemoryStore (soft policy + sandboxed bash, plus a strong-isolation pattern)
19 19-zai.ts Z.ai (GLM) via the bundled OpenAI-compatible provider — no extra install
20 20-policies.ts Typed system-prompt modules — priority-map + auto-resolver policy pair
21 21-slash-commands.ts Slash-command router — deterministic /<name> dispatch to sub-agents
22 22-shellm.shellm Shellm — a prompt file you chmod +x and run directly (shebang + frontmatter)
23 23-concurrent-store.ts Concurrent MemoryStore access — opt-in cross-process file locking (#15)
24 scheduled-tasks/tasks.json Scheduled tasks — cron-driven agent runs with lock-file concurrency (#79)

Run any example: npx tsx examples/01-basic-agent.ts

Releasing

Version management uses Changesets. Releases are manual by design — publishing requires short-lived npm credentials that the maintainer mints ahead of time, so nothing lives in CI as a long-lived token.

Recording a change

When you make a change that should appear in the next release, record a changeset describing it:

npm run changeset

The CLI asks whether the change is a major / minor / patch bump and writes a markdown file in .changeset/. Commit that file alongside your code change. The changeset body becomes an entry in CHANGELOG.md at release time.

Rules of thumb for pre-1.0:

  • patch — bug fixes, internal refactors, doc updates
  • minor — new features, non-breaking API additions, security fixes that don't change the public API
  • major — save for the 1.0 cut; until then, breaking changes can ride in minor

Not every PR needs a changeset. If a change doesn't affect what gets shipped to npm consumers (CI config, internal tests, comment tweaks), skip it.

Cutting a release

Once one or more changesets are sitting in .changeset/ on the branch you want to cut from (usually main):

# Mint a short-lived npm token, then either:
npm login                            # interactive, stays in ~/.npmrc for this session
# or:
export NPM_TOKEN=npm_…               # paste the token
npm run release

npm run release executes scripts/release.sh, which walks the whole manual flow in one go:

  1. Preconditions — dirty tree aborts, zero pending changesets aborts.
  2. Quality gatetypecheck + test + build.
  3. Apply changesetschangeset version bumps package.json and prepends entries to CHANGELOG.md. The script prints the diff so you can eyeball it.
  4. Commitchore: release vX.Y.Z on the current branch.
  5. Publishchangeset publish pushes to npm (with provenance) and creates the git tag.
  6. Push — commit + tag to origin/<current-branch>.

Any failure stops the script. Steps are idempotent enough that you can usually re-run after fixing the cause.

After publish, open https://github.com/PaulKinlan/agent-do/releases and (optionally) create a GitHub release from the fresh tag with the CHANGELOG body pasted in.

Commit message format

Not required. Changesets determines the semver impact from the .changeset/*.md files, not from commit messages, so commits can be whatever shape fits the change. Conventional-commit prefixes (feat:, fix:, chore:) are fine but not enforced.

The one discipline to keep is: any PR that changes user-facing behaviour needs a changeset. If that ever drifts, the cheapest enforcement is a CI check that fails PRs which touch src/ without adding a file in .changeset/ — happy to wire that up if it becomes a problem.

License

Apache 2.0

Read the original on github.com ↗