Most teams building multi-agent systems suffer from a 'Model-First' bias: they spend weeks benchmarking the latest LLMs, but zero hours defining an agent’s permissions, its unit economics, or its handoff contract. They obsess over intelligence and ignore architecture. That’s how a routine database lag triggers a $47,200 'rogue' feedback loop before Monday morning.
You’re the Lead AI Architect at a retail bank. Your team built a four-agent fraud investigation pipeline that runs 24/7 with no human on duty. When a transaction is flagged, the pipeline investigates automatically and takes action, freezing accounts, sending customer emails, issuing refunds. Every agent runs on Claude Opus, and given access to a set of tools.
On a Saturday night, the Monitor Agent sees a number of unconfirmed transactions in a 10-min period, flags 105 of them as suspicious, and passes them to the Investigation Agent. The Investigation Agent, uncertain about several ambiguous cases, re-invokes Opus 22 times on a single transaction to re-examine its own reasoning. Mid-investigation, it pulls the full transaction history, merchant records, and case archives for every account even loosely related to the flagged batch. The Report Agent, while writing summaries, spots a discrepancy in one frozen account and issues a $1,200 refund on its own initiative before writing the audit entry. By Monday morning: $47,200 in API costs, three fraud alert emails sent to innocent customers, six accounts frozen, one unsanctioned $1,200 refund issued.
Final bill: $47,200. Three customers received fraud alerts that didn’t exist. Four accounts incorrectly frozen. One unsanctioned refund.
Root cause: a routine database maintenance window causes a 10-minute delay in transaction confirmations, a known, scheduled event that resolves on its own.
⏳⏳ How do you fix this system? 💭
Better system prompts. Stricter instructions. A max retry count on the Investigation Agent. Human escalation triggers. And for Opus, swapping it with a newer model?
That’s not wrong, you should absolutely have some of those. But it treats the symptom, not the disease.
A retry limit on the Investigation Agent wouldn’t have stopped the Report Agent from issuing a refund. A human escalation trigger wouldn’t have stopped the Monitor Agent from sending premature fraud alerts to innocent customers. And swapping Opus for a newer frontier model doesn’t change the fact that you’re still running your most expensive model on every node, including the ones doing nothing more complex than reading a log file and writing a summary paragraph.
These aren’t runtime bugs you can patch with a better prompt or a smarter model. They’re architectural decisions that were never made. Three of them, specifically.
Fixes: the unsanctioned refund, the premature fraud emails, the out-of-scope account freezes
Every agent had access to tools it didn’t need. The Report Agent had issue_refund. The Monitor Agent had freeze_account. Nobody made those choices deliberately, they were defaults nobody questioned.
The principle: an agent should only be able to do what its role requires, and nothing more. The Monitor Agent watches and flags — it needs read tools, nothing else. The Report Agent writes a summary, it needs read_fraud_case_history and write_audit_log, full stop.
This matters because multi-agent failures are rarely one agent going rogue. They’re a chain of small, individually plausible decisions that compound. Least-privilege scoping breaks the chain. A Report Agent that cannot call issue_refund cannot issue a refund, regardless of what it hallucinates. The tool is the last line of defense, and it costs nothing to set. Only the Action Agent can move money or contact customers. Every other agent operates in read-only or write-to-log mode. This is 40% reduction in tool access compared with the original design!
Fixes: the 22-call reasoning loop, the bulk of the $47,200 bill
Opus at every node isn’t a safety choice — it’s an absence of one. Every agent has a different cognitive demand, and treating them as equally complex is the most reliable way to burn budget without improving accuracy.
The Monitor Agent is classifying patterns. The Report Agent is formatting known facts into a summary. Neither task needs a frontier model. The Investigation Agent is the one doing real work, reasoning across ambiguous signals, merchant history, and prior cases. That node earns Opus. The rest don’t.
The mental model: match the model to the cognitive demand of the task, not to the stakes of the pipeline it lives in.
The Investigation Agent is the only node that earns Opus. The Monitor Agent is classifying patterns, use a small model. The Action Agent is executing a decision already made, medium size. The Report Agent is formatting known facts, small. One Opus call per case instead of 22.
Fixes: the Action Agent misreading a confidence signal, the inability to reconstruct what happened
The Investigation Agent passed its output as free-text prose. The Action Agent read something that sounded confident and acted on it. No rule was broken, the handoff was just a conversation, which is expressive, flexible, and completely unverifiable.
Every agent handoff should be a structured contract:
Now the Action Agent can’t misread 0.43 as high confidence. You can write a hard rule: if fraud_confidence < 0.75, route to human, no model reasoning involved. And when something goes wrong, you have a full trace: every agent’s input, output, model, confidence, and timestamp. The Monday morning post-mortem becomes a log review, not a forensic investigation.
Observable agents doesn’t mean dashboards. It means every agent emits a structured, loggable output before the next one acts on it.
“In a multi-agent system, the damage an agent can do is a function of the tools it has, not the instructions it was given.”
Instructions are behavioral. Tools are structural. You can tell an agent not to issue refunds, but if it has access to issue_refund, you’re relying on the model following instructions perfectly, every time, under every possible input. In production, that’s not a guarantee, it’s a prayer.
The three fixes above aren’t about making your agents smarter. They’re about making the system resilient to the moments when they aren’t. Scope the tools so the blast radius is bounded. Right-size the models so the cost is bounded. Structure the handoffs so the failure is always traceable.
A multi-agent system that does all three can still make mistakes. But when it does, you’ll know exactly which agent made them, why it had the ability to, and how to fix it, before it happens again next weekend.
📖 This scenario is drawn from The Agentic AI Book — a production-first guide to building AI systems that actually work.
Grab early access: book.ryanrad.org
Until next dose — Dr. Ryan Rad
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.