Reading · AI Agents Simplified

Welcome to the 63rd issue of AI Agents Simplified 🍻

This issue is brought to you by HubSpot

Every AI agent demo works, and that’s exactly what makes them dangerous. You have one user, one channel, a model that answers in under a second, and a path through the code that you’ve walked a hundred times. Everything you’d need to worry about at scale is simply absent from the room.

Then you connect it to a real Slack workspace. A plugin throws an exception on startup, somebody pastes forty thousand tokens of log output into a thread, a webhook fires while two hundred sessions are mid-turn, and the thing you thought was an agent turns out to have been a prompt with no runtime underneath it.

The work of closing that gap has a name, and it’s a useful one to know: harness engineering. It’s the discipline of building the machinery around the model, the gateway that receives traffic, the permission system that decides what the agent may touch, the memory that survives a restart, the telemetry that tells you what happened, and the recovery paths that catch things when they go wrong. Almost nobody writes about it, and it’s the part that determines whether your agent survives contact with actual users.

If you haven’t read our previous blog post about Harness Engineering, this is the best time to take a look at it:

What follows are seven failure modes I keep running into, and the pattern that addresses each one. None of them are exotic. All of them are cheap to check for, and most teams discover them the expensive way.

Picture a plugin that fails while loading. Maybe a dependency is missing, maybe there's a syntax error, maybe a service it depends on can't reach its backend. In a naive architecture, that exception travels straight up through the startup sequence, and the result is that your entire gateway never comes online. One misbehaving component, and nothing works.

The instinct is to write more careful plugins. The better move is to assume plugins will fail, because eventually they all do and structure the system so that failure stays where it happened.

The pattern that solves this is called a bulkhead and the name comes from shipbuilding, where you divide a hull into sealed compartments so a breach in one doesn't flood the whole vessel. In an agent platform, the compartments are the plugin lifecycle phases (loading, registering, starting, running), and each one gets wrapped in its own error boundary. So when a plugin throws while loading, that boundary catches the exception, records it as a diagnostic, and lets the loader carry on to the next plugin. Your gateway still comes up, and you get a log line naming the plugin that's broken, which is considerably more useful than a stack trace and no service at all.

It’s worth understanding what this protects you from and what it doesn’t, because the distinction catches people out. If your platform runs in a single process, these boundaries contain errors but they can’t contain resource consumption, a plugin stuck in a tight loop will still eat the CPU its siblings need, and no amount of try/catch will help. What you’re buying is containment of failure rather than true isolation between components, and that’s genuinely valuable as long as you know which of the two you have.

Sitting above that is an idea I find more interesting which is that a platform shouldn’t only have two states. Instead of being either up or down, it can descend through levels. At full operation everything runs. Drop a level and you’re in core-only mode, drop again and you reach read-only diagnostics. At the bottom is offline, where health checks respond and nothing else does.

Each of those rungs is a deliberate engineering decision about what to sacrifice and in what order, which is why Ken Huang spends a full chapter on it in OpenClaw AI in Production, Chapter 3 walks through all four levels, including which function owns each one and where the boundary actually gets enforced. That last part matters more than it sounds, because "graceful degradation" is a comfortable phrase that hides a lot of fiddly implementation.

This is my favourite failure on the list, because it doesn’t look like a failure at all. It looks like security.

Here’s the setup. You write a tool policy for your agent. You define a deny list and an allow list, both are present, the config validates cleanly, and you ship it feeling reasonably good. What you may not have realised is that in a deny-first matcher, an empty allow list usually means permit everything rather than permit nothing. So a policy block that reads as tightly locked down is in fact granting every tool that you didn’t explicitly name in the deny list.

That isn’t a bug, to be clear. It’s a coherent design called allow-unless-denied, and it’s a sensible default for a system that has to stay usable. The problem is purely that it’s not what most people think they configured, and the gap between those two beliefs is where trouble lives.

The model underneath is a composition stack rather than a single setting. Built-in defaults sit at the bottom, then global config, then per-agent config, then channel group policy, then sandbox policy, then subagent depth limits, and finally per-session overrides at the top. What makes it a composition stack rather than a simple override chain is that all seven layers get consulted together, a tool is permitted only when nothing in any layer denies it, and every layer that does define an allow list agrees to it. Deny always wins, no matter how high up the permission came from.

The practical upshot is that you get two workable strategies:

You can set restrictive global defaults and selectively relax them for individual agents
or you can set permissive defaults and tighten them where the risk is highest. Either works fine on its own.

Chapter 5 of the book lays out the full precedence stack along with the glob expansion that sits underneath it, including a detail that catches people regularly:

group expansion is only single-level, so if you’ve defined nested tool groups expecting them to resolve recursively, they don’t.

^{You already know email gets buried and calls go unanswered. But texts? They get read — 98% of the time, within minutes. That’s why 230,000+ businesses use EZ Texting to reach their customers on the channel they engage with most}

^{Whether you’re sending promos, appointment reminders, or urgent alerts, EZ Texting makes it easy to send to 10 or 10,000 contacts in just a few clicks. No tech skills required.}^{AI Agents Simplified’s}^{readers can try it free — no credit card needed.}

Start Your Free Trial with EZTexting

^{Offer valid for new customers only. Use promo code}^EZWORKFLOW^{at checkout.}

Conversations outgrow context windows, and everyone knows that. What gets underestimated is what specifically gets destroyed when you compact one.

The prose survives compaction perfectly well, because summarizing prose is what language models are good at. The UUID does not. Neither does the file path, the container ID, or the API endpoint the agent was halfway through calling. What happens is that the model does something helpful and reasonable and now every tool call that depends on that identifier fails, several turns later, for reasons that look completely unrelated to compaction.

Three things make the difference between compaction you can trust and compaction that quietly corrupts your session.

The first is triggering it early. You want to compact somewhere around 70-80% of the context window rather than waiting until you’re up against the ceiling. That’s because token estimators are heuristics and they undershoot, and a buffer of roughly 20% is what stands between compacting on your own schedule and overflowing the window in the middle of a tool call, which is a much worse place to find out.

The second is chunking by token weight instead of message count. If one assistant turn is six thousand tokens long, it should fill a chunk on its own, even if it’s surrounded by dozens of two-word user replies. Summarising coherent segments independently and then merging them produces noticeably better results than compressing a thousand messages in a single pass, because the model can actually hold each segment in view while it works.

The third is pinning identifiers explicitly, which means your summarization prompt needs a hard instruction to reproduce opaque strings exactly as written It feels like an obvious instruction until you see what happens without it.

Chapter 6 goes considerably deeper than I can here, into append-only event logs, the timing of memory flushes before compaction runs, and how to merge summaries that cover overlapping turns. If you’ve ever watched an agent lose the thread halfway through a task and couldn’t explain why, that chapter explains the mechanism.

Networks fail, so you add retries. It’s the most natural reflex in distributed systems and it’s usually correct. But now the same Slack event can get processed twice which means → the same tool fires twice which means → the same order gets created twice and you’ve turned a transient network problem into a data problem that’s much harder to unwind.

Retries without deduplication aren’t resilience. They’re duplication with a scheduler attached.

What makes retries safe is a deduplication key built from the message’s provenance, the provider it came from, the account, the session scope, the peer, the thread, and the message ID, all joined into a single fingerprint. When a message arrives, you check whether you’ve seen its fingerprint before. If you have, you skip the work entirely and return the result you produced the first time. Pair that with bounded exponential backoff on the model invocation, which is the phase most likely to fail for transient reasons, and retrying becomes something you can do freely instead of something you have to reason about each time.

Two details in that design are worth borrowing:

The first is that the key should return null when provenance is incomplete rather than falling back to something approximate.
The second is that deduplication is a property that wraps the whole request flow rather than a stage inside it. It isn’t step four of six; it’s a guarantee that holds across all six.

Chapter 2 traces the entire six-phase request path and shows precisely where the dedupe check and the backoff sit relative to context assembly, authorisation, and the model run, which is the kind of thing that’s much easier to get right when you can see it laid out.

An agent enters a reflection loop on a Friday evening. Nothing is down, no health check fails, no alert fires. It is simply thinking, in a circle, at per-token prices, until somebody looks at a dashboard on Monday…

How are you finding this blog post so far? My goal is to write articles that are genuinely useful for our readers. If you'd like me to write the second part of this post, leave a comment and share your thoughts.

7 Failures Nobody Warns You About Shipping An AI Agent

Comments