Your AI model needs an acceptance test

Hi there!

Before today’s piece, one upcoming hands-on session for anyone building AI-assisted NetOps workflows:

Last chance to register - Claim 50% OFF

Most AI assistants can write you a plausible-looking config. They can’t tell you if it matches your actual network, because they’ve never seen your runbooks, your device configs, or your topology.

Sif Baksh (Principal Solutions Architect at Tines) spends this workshop building the piece that’s missing: a RAG-powered NetOps assistant that answers from your own runbooks and device configs instead of general training data, with guardrails that make it flag what it doesn’t know instead of guessing.

Four hours. Four labs. You’ll leave with a working assistant and a workflow you can keep testing after the session.

For AI Network Engineer readers: code FINAL50 gets you 50% off all passes.

P.S. Sif wrote the book on this, literally. Building AI Agents for Network Operations goes deeper into the architecture than four hours allows. Bundle it at checkout, or grab it on its own if Tuesday doesn’t work.

Now, back to today’s test.

Ask someone which AI model you should use for network operations and you’ll usually get a model name.

That’s the wrong place to start.

A model can perform well on a general benchmark and still miss a requirement your workflow cannot afford to miss. Another may be unnecessary for routine summaries but worth the cost for complex troubleshooting.

You don’t need a favourite model.

You need an acceptance test

Suppose you give two models this prompt:

Both responses may look reasonable. But the task is too vague to tell you much.

Which platform? Which area? What should become the router ID? Should the response include verification? What information is the model allowed to assume?

A better test looks like this:

Now the model has constraints it can either satisfy or miss.

That makes the response easier to evaluate. It also makes failures visible before you spend time debating which answer sounds more polished.

Before running the prompt, decide what a passing response must contain.

For the OSPF task, you might expect:

• IOS-XE-compatible syntax

• Area 0 applied correctly

• Loopback 0 used as the router ID

• Missing interface or address information identified

• Verification commands included

Then define the hard failures:

• Invented IP addresses

• Syntax from another network operating system

• Unstated assumptions

• Missing verification

• A claim that the configuration is ready to apply without checking the current device state

This is the part most informal model comparisons skip.

If you wait until after reading the responses to decide what matters, the most fluent answer tends to set the standard. Your evaluation becomes a writing contest.

The answer key prevents that.

Use the same task, context, return format and response settings for every candidate model.

Save the raw responses rather than relying on what you remember from the chat window. Record the model version and the date too. Model behaviour changes, and an evaluation you cannot reproduce is only a snapshot.

A compact test record can look like this:

For a small comparison, a spreadsheet is enough. The important thing is that every model sees the same test and every response faces the same standard.

Using an evaluator model can help sort a large response set. It can identify missing sections, compare formats and produce an initial score.

But it should not own the answer key.

The evaluator is still a model. It can overlook the same plausible-looking technical mistake you are trying to catch.

Let it assist with the comparison. Keep a network engineer responsible for defining the expected facts, reviewing hard failures and approving the result.

For a fairer comparison, hide the model names during review. “Response A” and “Response B” are harder to favour than two brands the reviewer already has opinions about.

Do not collapse every result into one overall number too early.

Separate the tasks by the work the model will perform:

• Configuration planning

• Troubleshooting

• Documentation

• Log or ticket summarization

• Structured data extraction

A model that performs well on documentation may not be the strongest choice for troubleshooting. Another may follow a JSON schema reliably but produce weak operational recommendations.

Different workflows may justify different models.

You already work this way with network tools. The application used for packet analysis does not need to be the same one used for configuration management. Model selection should follow the same logic.

Cost matters, especially when a workflow will run hundreds or thousands of times.

But a cheap response that fails a mandatory technical check is not better value. It is a failed response delivered inexpensively.

Use the acceptance criteria as the first gate:

• Did the response include the required facts?

• Did it avoid every hard failure?

• Did it follow the requested format?

Only the models that pass should move into the cost comparison.

Then you can ask whether the improvement from a more expensive model is meaningful for this particular workflow.

Choose two models and four tasks your team already understands:

• Two configuration-planning tasks

• Two troubleshooting tasks

Use resolved tickets, reviewed MOPs or lab scenarios as your source. Remove credentials, customer data and sensitive infrastructure details before using an external service.

Write the expected facts and hard failures first. Run each model under the same conditions. Save the responses and review them without the model names visible.

You do not need a hundred-question benchmark to learn something useful.

Four well-chosen network tasks will tell you more about your workflow than a general leaderboard ever will.

Stop asking which model is best.

Ask which model passed your acceptance test.

Cheers,
Sayali
Editor-in-Chief

The tool boundary is only half the production story. The next questions are what happens after an agent proposes a change and how access is governed. These two upcoming sessions approach those problems from different sides.

Aug 27 · Agentic AI for Infrastructure Engineering: From Chatbots to Operators

Ritesh Vajariya, founder and CEO of AI Guru, will show you how to build an infrastructure agent that reads pod events, correlates live metrics and proposes fixes—but executes them only after approval.

You’ll work through seven production-grade failure scenarios injected into your own local Kubernetes cluster, which you can keep and rerun after the session.

Claim 50% Off

Aug 29th · Active Directory and Entra ID in a Modern Hybrid Architecture

Professor Robert McMillen breaks down how traditional Active Directory and Entra ID work together in real hybrid environments: domains and group policy on one side, cloud identity and SSO on the other, and Entra Connect bridging them. Built for IT pros moving into infrastructure or identity-focused roles.

Book Now

If this issue helped sharpen the conversation, please like it or leave a comment so more platform, cloud, security and infrastructure engineers can find it.

Thanks for reading The AI Network Engineer by Packt. Subscribe for free to receive future issues.