Lesson 2 — Tool Execution and Validation

This lesson builds the Layer 3 tool execution boundary for a production AI agent: a schema-validated registry, permission-gated dispatch, and observable outcomes.

You implement a ToolOrchestrator that rejects unknown tools, missing parameters, unsafe input, and unauthorized writes before any side effect runs.

A FastAPI dashboard surfaces live tool calls, denials, blocks, and latency.

• Layer 3 tool boundary with typed schemas, permission checks, and explicit error states.

• Single orchestrator dispatch instead of ad hoc Python functions.

• Live metrics for allowed calls, denials, unsafe input, failures, and latency.

• Self-contained Dockerized FastAPI service with dashboard, CLI demo, tests, and cleanup.

• ToolOrchestrator validating tool names, parameters, and permissions.

• Registry tools: calculate, echo, write_log.

• FastAPI endpoints: /health, /metrics, /tools/call, /demo, /dashboard.

• Live dashboard for metrics and recent events.

• Docker and scripts for start, demo, test, stop, and cleanup.

Lesson 1 introduced the four-layer secure agent and least-privilege tool discovery. Lesson 2 turns that into execution—validating inputs, checking permissions, dispatching tools, and returning structured results.

Typed, permissioned, observable tool calls let later lessons add memory, persistence, and workflows without rebuilding safety checks.

• Where this component sits — Between agent logic and external capabilities; Layer 3 gates existence, permission, validation, and reporting.

• Why it exists — Model actions must not execute directly; the boundary narrows exposure and creates audit points.

• Problem solved — Blocks unknown tools, bad parameters, unsafe expressions, and unauthorized writes before runtime side effects.

registry.py owns dispatch; metrics.py owns counters; app.py serves HTTP; main.py runs CLI; scripts handle operations.

• tools/registry.py — schema, registry, orchestrator.

• tools/metrics.py — in-process dashboard metrics.

• app.py — HTTP API and dashboard.

• main.py — CLI demo.

• Dockerfile, docker-compose.yml, shell scripts — deployment workflows.

• Tool safety, schema discipline, observability, deployability, and testability.

The registry is a production allowlist declaring capabilities and requirements. Permission-gated dispatch enforces least privilege—read-only callers cannot write logs. Schema validation returns structured ToolResult errors for dashboards and audits. The calculate sandbox rejects code-execution patterns, showing each tool needs its own validation. Metrics classify ok, denied, blocked, and failed outcomes separately.

• Tool calls are authorization events, not mere function calls.

• Unknown tools fail closed; model output is untrusted.

• Expected denials prove the safety layer works.

• In-memory metrics suit lessons; production needs shared storage for replicas.

• Centralize validation; permission denied, unsafe input, and unknown tools differ operationally.

This is the execution gateway preventing an LLM from becoming an unbounded RPC caller—scaling to databases, tickets, email, payments, and infrastructure actions.

• request flow — Tool name, parameters, permissions, and label reach the orchestrator.

• execution flow — Registry lookup, permission check, parameter validation, dispatch, ToolResult wrap.

• data flow — Input through ToolOrchestrator.call(); metrics receive normalized events.

• state changes — Static registry; METRICS counters and events update at runtime.

Request → registry → permission → validation → execute → metrics → response.

• production architecture fit — Behind auth, before side effects; emits traces to shared observability.

• enterprise deployment patterns — Containers, pinned images, env config, API gateway; externalize metrics for replicas.

• scalability — Horizontal dispatch when metrics and audit logs are externalized.

• observability — Structured logs, counters, latency histograms, durable audit events.

• security considerations — Server-side identity, rate limits, approval for high-impact tools.

Support agents summarize tickets for all users; only supervisors issue refunds via permission-gated billing tools. Cloud assistants read health freely; restart and migration tools require elevated permissions and audit logging.

https://github.com/sysdr/production-ai-engineering/tree/main/lesson2/aiam-day02

JSON/CLI input passes validation, returns ToolResult; metrics get normalized events.

ToolSchema, ToolResult, ToolOrchestrator, MetricsStore.

available(), call(), record(), run_demo().

Registry allowlist → result envelope → orchestrator → tool handlers → metrics → FastAPI → dashboard → Docker packaging.

• scalability: externalize metrics for replicas.

• security: server-side identity, not request permissions.

• monitoring, logging, testing, failure handling, edge cases: timeouts, input caps, idempotent retries only.

• verification: orchestrator tests, API tests, Docker build, health checks.

• success criteria: start.sh runs dashboard, demo.sh runs CLI, run_tests.sh passes, cleanup.sh stops services.

• expected outputs: non-zero post-demo metrics, timestamped events, structured expected failures.

• production checklist: no secrets, official Python image, clear failure classification, health and metrics endpoints.