Evaluation Is How Agentic AI Earns the Right to Act. Here's What We're Finding.

An agent books a seat before checking availability. Another approves a loan that fits the customer request but violates the firm’s risk posture. A third says it has updated a reservation, but the downstream system never changed at all. All three are failures, but of very different types, and none should be handled by the same mechanism.

This is where we see some of the current conversations about “evals” breaking down. The term has become a catch-all for runtime guardrails, policy review, task verification, benchmark scoring, and sometimes deterministic controls that are not really evaluations in the first place. The lack of precision matters. If you collapse all of those into one category, you end up using the wrong instrument for the wrong job. The industry still tends to treat agent failure as a model-quality problem, however in our experience it is increasingly a runtime-design problem.

At AI One, we have been working on this problem inside ContextOne, our enterprise agent runtime. What has become clear is that evaluation only becomes reliable when it is treated as part of the architecture. Not a score at the end. Not a judge prompt bolted onto a swollen context window. A governed system that decides what the agent may do next, what it must prove before it continues, and what the platform should learn from the result.

These are early findings, not final answers. But they suggest that evaluation is less a single technique than a stack of distinct mechanisms, each matched to a different class of failure.

In traditional software, verification is easier because the system is deterministic. A function returns the right output or it does not. Agentic systems behave differently. The same model can take different paths to the same objective. It can look reasonable while violating sequence, policy, or downstream state. Once an agent can read enterprise data, call tools, and write to live systems, “does the response look plausible?” stops being a serious standard.

Manufacturing figured this out a long time ago. You do not use end-of-line inspection to decide whether a bolt should have been tightened earlier in the assembly process, and you do not use a torque sensor to decide whether the finished engine actually runs. Different failure modes require different checks at different points in the process. Agentic systems are no different.

Consider an investment-bank operations agent running a pre-open allocation workflow. If it allocates a block trade before checking inventory, that is a sequencing failure. If it proposes an allocation that fits the mechanics but breaches client mandate or desk risk posture, that is a judgment failure. If it reports the booking as complete but the order management system still shows the old state, that is an outcome failure. The stakes are not academic: broken trades, manual repair, capital and compliance exposure, and the familiar question of why was a proof of concept allowed anywhere near production?

In practice, four different questions hide inside the word “evaluation”:

1. Can the agent take the next step?

2. Is the proposed action aligned with policy and intent?

3. Did the claimed outcome actually occur in the underlying system?

4. Does the architecture materially improve autonomous task completion across realistic workflows?

A resilient system answers each with a different mechanism.

If those questions are collapsed into one category, system design gets confused. Teams ask an LLM judge to do work that deterministic logic should have handled. They use benchmark scores as a proxy for runtime safety. Every control gets called an eval and sight is lost of what each component is there to solve.

Our research indicated we needed to settle on a three tier architecture. Instant Eval asks whether the next move is allowed. The LLM Judge asks whether the move is aligned with higher-level intent. The Agentic Grader asks whether the job was actually done.

Instant Eval enforces deterministic, real-time constraints on sequencing and preconditions of tool calls. Before an agent can execute a tool, the runtime checks whether the required prior steps have occurred. If the check fails, the agent receives actionable feedback rather than a silent rejection.

In an airline workflow, if the agent attempts to reserve a seat before it has checked availability, the call should be blocked and the agent should be told exactly which prerequisite is missing. That sounds simple. It is also the difference between a self-correcting system and a corrupted record in production. A surprising share of enterprise failure is structural, not conceptual.

Some decisions cannot be reduced to sequence. They depend on judgment, tradeoffs, and principles that are too broad or too fluid to encode as a strict set of preconditions. This is where the LLM Judge belongs. It sits at designated checkpoints and evaluates whether a proposed action aligns with higher-level objectives and constraints, using only the slice of context relevant to that assessment.

Consider a loan approval workflow. An agent proposes a $50,000 loan to a customer with a credit score of 620. The sequence may be correct. The decision can still violate the firm’s risk posture. That is a judgment problem, not a sequencing problem. This is where an LLM Judge is useful, and also where it should stop. It should not be asked to do work that deterministic logic can do better.

The final runtime question appears when an agent says it is done. In many deployments, this is still the weakest link. An agent can produce a convincing summary of a task it did not actually complete. It can claim success after making only part of the required change. It can update the wrong record and still sound confident.

The Agentic Grader verifies the outcome against the underlying system. For tasks that can be checked deterministically, it can call Python scripts, database queries, or gateway tools directly. For tasks that require a more interpretive check, it can dispatch dedicated sub-agents with introspection tools and feed their output into a separate judge. If an agent says it updated a customer’s seating assignment and the reservations system does not show the change, the task is not complete. The agent goes back and fixes the discrepancy.

That separation matters because agents fail before an action, at the point of action, and after the work is supposedly complete.

One of the most common confusions in agentic AI is treating every control surface as an eval. Some mechanisms exist to judge behavior or outcome. Others exist to make certain classes of failure impossible. Both matter. They do different jobs.

Formal constraint enforcement is the clearest example. Some business rules are too important to leave to probabilistic judgment. In ContextOne, those rules can be translated into logical constraints and checked with a formal constraint solver before execution. If a proposed action violates the rule, the action is blocked. A loan approval that conflicts with a formal credit policy should not be “graded down” later. It should fail closed.

In regulated environments, query generation belongs in the same category. For example, in ContextOne, we use the concept of Named Queries to let the agent invoke fixed, versioned query structures by name, supplying parameters rather than generating live query logic at runtime. Credentials stay out of the Agent Harness, access is enforced at runtime, and activity is recorded in a cryptographically signed audit ledger. These are not alternative flavors of eval. They are the architectural conditions under which evaluation becomes meaningful.

A useful evaluation framework cannot sit outside the agent as a wrapper around one giant prompt. It has to live where the work is being done. In ContextOne, that is the Agent Harness.

The architectural split is simple but important. The loop reasons and the harness governs. That separation creates a stable place to apply runtime checks, manage permissions, validate tool calls, gate results back into context, and halt or escalate when thresholds are reached. In ContextOne’s governed loop, we made that explicit: prompt assembly, context management, model invocation, result validation, tool execution, result collection and context gating, and control condition evaluation all happen inside a managed execution pipeline. Large tool outputs are offloaded and referenced rather than blindly stuffed back into context. Cost and iteration limits are checked before an agent is allowed to continue.

The economics matter too. ContextOne is stateless and event-driven. Agents materialize when work is required, yield resources between iterations, and resume from shared state in single-digit milliseconds. That makes governed, step-by-step execution practical at enterprise concurrency.

The practical takeaway is simple: a model that looks intelligent in a proof of concept is not production-ready until you can explain what stops it, what judges it, what verifies its work, and what it is deterministically prevented from doing.

Runtime evaluation tells you whether a live task should proceed. Benchmark evaluation answers a different question: does the architecture materially improve autonomous task completion across realistic workflows?

That is the role benchmark evaluations play. On Tau Bench 2, ContextOne achieved 95% task completion in the telecoms domain against a baseline of roughly 30%. In the airline domain, it achieved 78% on a manually verified solvable subset against a roughly 40% baseline, and 52% on the full task set against a roughly 25% baseline. The same prompts were used for both the baseline and ContextOne runs. The gains came from architecture rather than prompt optimization.

But benchmark evaluation should not be confused with runtime safety. A Tau score does not tell you whether a particular loan approval, trade allocation, or customer update should be blocked right now. It tells you whether the architecture has moved the system from constant-supervision territory toward genuine delegation. At around 30% task completion, an agent still demands babysitting. At 90%+, you are starting to talk about real business process execution, with human oversight reserved for genuine exceptions.

Before any agent moves from proof of concept to production, management should be able to answer four plain-language questions: what blocks it, what judges it, what verifies that the work actually happened, and what it is deterministically prevented from doing. “The model looked good in testing” is not a control framework.

What comes next is not a search for one universal eval. It is better architecture for separating questions that are currently collapsed into that one word.

Which actions should be blocked deterministically? Which decisions should be reviewed against principles? Which outcomes must be verified against live system state? Which failures belong in a benchmark, and which belong in the runtime? Which signals should feed memory and improve the next execution? These are the design questions that matter.

And this connects directly to memory. In our earlier SubStack article on memory, we argued that enterprise agents need governed ways to retain what happened, how tools should be used, and which contextual facts matter. Evaluation is what makes those memories trustworthy. Memory without evaluation compounds both skill and bad habits. Evaluation without memory catches the same failure forever. Joined together, they create a feedback loop in which successful trajectories become reusable patterns, failed actions become correctable guidance, and autonomy can expand only when the system has earned it.

Our current findings point in one direction. In enterprise AI, evaluation architecture may be one of the most under-leveraged variables in system performance. Models matter, and prompts matter. But in agentic workflows that touch real systems, the more consequential lever may be the architecture that determines what an agent is allowed to do, what it must prove, what it is prevented from doing altogether, and what the platform learns from the result.

We do not claim to have the final answer. The evidence so far suggests that the systems that earn production trust will be the ones that know when to block, when to judge, when to verify, and when to learn. They will treat evaluation not as a single score, but as a governed part of the runtime itself. That is the standard proof-of-concept projects should be held to, before anyone calls them production-ready.