AI Accountability Needs Boundaries

I came away from Black Hat and DEF CON with one recurring thought: the debate over whether AI needs governance is largely behind us.

The more important question is what that governance has to become.

Marcos Semola captured part of the concern coming out of Black Hat when he described organizations accumulating “AI governance debt” as adoption moves faster than the controls surrounding it. (LinkedIn)

At the same time, researchers and technology companies are exposing why closing that gap will not be as simple as writing better policies.

Novee Security’s Black Hat research showed vulnerabilities in real-world Claude Code, Gemini CLI and Codex workflows created by hidden trust handoffs between different stages of an agentic system. Novee makes a particularly useful distinction: the model produces intent, while the surrounding “harness” includes the tools, permissions, execution environment, sandboxing, routing, memory and observability that connect that intent to the real world. (Novee)

Viakoo recently highlighted another problem that deserves more attention: persistent memory itself can become compromised. Commenting on the RufRoot vulnerability, Viakoo pointed to the operational implications of attackers poisoning memory that AI agents later rely on to make decisions. Dark Reading reported that the underlying vulnerability allowed attackers to modify persistent agent memory in ways capable of influencing later behavior even after the original software vulnerability had been patched. (Dark Reading)

And while we are wrestling with those governance questions, industrial AI is not standing still. Siemens describes its Eigen Engineering Agent as moving beyond assistance toward actually planning, writing, validating and delivering automation-engineering work. Siemens CEO Roland Busch recently said the platform is already bringing agentic AI into industrial workflows at scale, with hundreds of customers onboard. (LinkedIn)

These may look like separate developments.

I think they point to the same architectural conclusion.

The model is no longer the system

We still tend to talk about “the AI” as though a model receives a prompt and produces an answer.

Increasingly, that is not what is happening.

A consequential transaction might move from a user into an application, retrieve prior memory, collect additional data, invoke a model, hand work to an agent, call a tool, cross an API, invoke another agent and eventually reach an operational system.

The user sees one interaction.

The system executes a graph.

That matters because we cannot establish accountability for the entire transaction simply by deciding that each individual component is trusted.

Novee’s research is a good illustration. A control applied at one stage of a workflow can be undermined when a later stage interprets the same trusted state differently or with greater capability. (Novee)

The problem is not necessarily that one component has no security.

The problem is what happens between the components.

Accountability therefore needs boundaries

My conclusion is increasingly straightforward: if we want dependable AI accountability, we have to compartmentalize the environment and control the boundaries between those compartments.

Models, agents, tools, data, credentials, tenants, memory and operational systems should not exist inside one implicit zone of trust.

Compartmentalization limits how far influence can propagate.

And each meaningful transition should force the system to re-establish what is allowed to cross.

• Who or what is acting?

• For what purpose?

• What information is being carried forward?

• Which tool or destination is permitted?

• What authority applies?

• What consequence could result?

Those decisions should not be left solely to the model or agent requesting the action.

Memory has to be governed as influence

Memory is especially important because it is easy to treat it as passive storage.

It’s not.

Anything retained and later placed back into an agent’s context can influence the next decision.

A maintenance exception that was legitimate yesterday may be wrong today. A previous task state may no longer apply. A retrieved record may have been superseded. A poisoned memory may be internally consistent and still produce a dangerous assumption.

Viakoo’s warning makes this practical rather than theoretical: if persistent memory can be manipulated, patching the original vulnerability may not remove the influence already left behind. (Dark Reading)

A bounded agent is not truly bounded if uncontrolled prior context can cross the boundary with it.

And accountability has to leave evidence

The other half of bounded architecture is evidence.

Logs can tell us that something happened.

For consequential AI, we need to be able to reconstruct more:

• What influenced the transaction?

• What memory or context entered it?

• Which model and agent participated?

• What crossed each boundary?

• Which controls were applied?

• What was permitted or denied?

• What action followed?

• What happened as a result?

At NexGenomics, this is why we have approached AI governance as an architectural problem rather than a policy wrapper around a trusted model. We compartmentalize the environment, control the transitions among those compartments, keep consequential authority outside the participating AI, govern retained context as a source of influence, and preserve the evidence required to reconstruct what occurred.

The objective is not to prevent AI from becoming useful.

It is exactly the opposite.

The stakes change when AI moves from answers to effects

Siemens’ industrial direction makes that transition easy to see.

An AI system that summarizes a document presents one class of risk.

An agent that can plan engineering work, generate automation code, validate it and move that work closer to execution presents another. (LinkedIn)

The closer AI gets to consequential systems, the more deliberately we need to replace implicit trust with explicit control.

The governance discussion is necessary. But governance has to survive contact with the architecture.

If AI can remember, retrieve, delegate, use tools and act, accountability cannot depend on asking the model to behave.

Compartmentalize it. Bound it. Control what crosses the boundaries. Govern the memory. Preserve the evidence.

That is how accountability becomes an engineering property instead of an aspiration.

Govern every influence. Prove every outcome.