Overview

An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.

Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the TypeScript client protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.

defineEval

eve discovers evals under the app-root evals/ directory, in .eval.ts files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see Cases). The file path is the eval's identity, so you don't author an id or name. Directories group related evals (evals/weather/brooklyn-forecast.eval.ts becomes id weather/brooklyn-forecast).

my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json

An eval is a single async test(t) function. You drive the agent with t and assert on the run with the same t:

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(t.reply, includes("Sunny"));
  },
});

test is the only required field. The rest are optional: description, judge, tags, metadata, timeoutMs, and reporters. The init template adds evals/**/*.ts to tsconfig.json, so your eval code type-checks alongside the app.

evals.config.ts

Every evals/ directory needs exactly one evals.config.ts at its root. It declares the defaults every eval shares:

import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "openai/gpt-5.4-mini" },
  reporters: [Braintrust({ projectName: "my-agent" })],
});

Everything is optional. judge sets the default model for LLM-as-judge assertions (t.judge.*); a tree of fully deterministic evals can omit it. reporters, maxConcurrency, and timeoutMs round out the defaults. Config reporters observe every eval in the run, so set one Braintrust() here instead of adding it to each eval. CLI flags (--max-concurrency, --timeout) and per-eval values take precedence over the config defaults.

Deterministic fixture models

Use mockModel when an eval fixture needs to exercise eve's runtime without calling a model provider. A static fixture can be one line:

import { defineAgent } from "eve";
import { mockModel } from "eve/evals";

export default defineAgent({
  model: mockModel("A deterministic reply"),
});

Pass a callback when the reply depends on the conversation. The callback receives an eve-owned view of the prompt, including lastUserMessage, userMessages, userMessageCount, available tools, and prior toolResults:

export default defineAgent({
  model: mockModel(
    ({ lastUserMessage, userMessageCount }) => `Turn ${userMessageCount}: ${lastUserMessage}`,
  ),
});

The callback may return { text, toolCalls, usage } for deterministic tool loops or explicit token counts. Use the options form only when a fixture also needs a custom model identity:

model: mockModel({
  modelId: "weather-script",
  provider: "my-fixtures",
  respond: ({ toolResults }) =>
    toolResults.length === 0
      ? { toolCalls: [{ name: "get_weather", input: { city: "Brooklyn" } }] }
      : `Weather: ${JSON.stringify(toolResults[0]?.output)}`,
});

mockModel() uses "Mock response" when no response is supplied. It handles both generated and streamed responses, derives deterministic response metadata, and estimates token usage. Because the model is part of the agent definition, use it for a dedicated fixture agent; it remains mocked whether that fixture runs locally or as a deployed eval target.

The t context

t is both the driver and the assertion surface. There are no separate input, run, checks, or scores fields. You write ordinary control flow, sending turns and asserting inline.

• Drive the agent: t.send(...), t.start(...), t.cancel(), t.respond(...), t.respondAll(...), t.sendFile(...), t.requireInputRequest(...), t.newSession(). Live turns returned by start() can wait for typed mid-turn events before cancellation or settlement. Read what came back with t.reply (the last assistant message), t.sessionId, and t.events. See Cases.
• Assert with three surfaces, covered next.

Three assertion surfaces

Each surface matches a genuinely different kind of judgment:

• Scoped methods read the final whole run on t, snapshot one independent session when invoked there, or inspect one immutable EveEvalTurn. See Assertions.
• t.check(value, assertion) grades an explicit value with a deterministic builder from eve/evals/expect, such as t.check(t.reply, includes("sunny")). Grade t.reply, an intermediate draft, parsed JSON, or anything else. See Assertions.
• t.judge.autoevals.* is the LLM-as-judge surface, like t.judge.autoevals.closedQA("cites a source"). It grades t.reply by default and uses the configured judge model, never the agent under test. See Judge.

Gate vs soft

Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.

• Gates are hard. A failed gate marks the eval failed and eve eval exits non-zero. Run-level methods, includes, equals, and matches are gates by default.
• Soft assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval scored (visible but not fatal, unless you pass --strict). similarity and every t.judge.* assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.

Override per assertion: .gate(threshold?) promotes to a hard gate, .soft(threshold?) demotes to tracked, and .atLeast(threshold) is a soft assertion with a bar.

t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
t.judge.autoevals.closedQA("cites a source"); // soft, tracked (no threshold)
t.judge.autoevals.factuality(reference).atLeast(0.7); // soft, gated under --strict at 0.7

Use await t.require(value, assertion) for a gate that must pass before the script can safely continue. Use t.skip(reason) as the first operation for an intentionally unsupported target capability.

Run evals with eve eval

eve eval                       # run all discovered evals against a local dev server
eve eval weather               # run one eval, or every eval under evals/weather/
eve eval --url https://<app>   # target an existing server or deployment

Exit code 0 means every eval passed its gates. See Running evals for the full flag list, exit codes, and CI guidance.

A good baseline

Most apps do fine with a few small smoke evals. Assert behavior with t.succeeded() plus one or two content checks, keep dataset fixtures in evals/data/, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run eve eval --strict so soft threshold misses fail the build too.

What to read next

The rest of this section covers each piece:

• Cases: single-turn evals, scripted multi-turn evals, and dataset fan-out
• Assertions: run-level methods and t.check value assertions, with matchers and severity
• Judge: LLM-as-judge grading and the judge model
• Targets: local vs remote targets for the same eval files
• Reporters: Braintrust experiments and JUnit XML
• Running evals: the eve eval CLI, exit codes, and artifacts
• Tools: the surface most evals assert on