I Went on Vacation. The Agents Stopped.

The thread was on r/shopifydev sometime around April 21. A developer’s backend got crushed. A merchant had pushed a huge product catalog update, and the resulting events had piled into the app’s intake layer faster than it could handle. The post was a description of the cleanup. Hours of incident response. A bill from AWS that arrived later.

I read it and I recognized it. OrderFlow, one of my apps, listens for the same kind of catalog change events. Same Shopify, same architecture. Same merchant could install OrderFlow tomorrow.

So I went to my product manager agent.

The PM agent has been live since early April. Every day it reads through merchant complaints, competitor reviews, and support tickets. It groups them into topics, ranks the topics by which ones seem most worth doing something about, and produces a weekly digest. There are nine separate commands I can run against it from the terminal. Each of my apps has its own roadmap file that the agent maintains.

It has one command for adding a new item to a roadmap. I used it. It captured “harden OrderFlow against this kind of event burst” as a roadmap entry. The slot was there. The format was there. The agent accepted what I typed.

That was the limit of what it could do.

It doesn’t read documentation. The math of what a noisy free-plan merchant costs me versus what a paying merchant costs me is outside what it does. So is comparing my apps to what the established players in the same category do for the same problem. The half of the agent that’s supposed to surface needs from real merchant data works by looking for exact phrases against vocabularies I wrote months ago. A Reddit post about event-burst overload doesn’t match any of those phrases. Even if the Reddit scanner were running, which lately it hasn’t been.

So I opened a second Claude session and asked it to do a research task. The deep kind. What does it actually mean to harden a system like this. Failure modes, throttling patterns, what happens when too many events arrive at once, what happens when the same event arrives twice. The canonical references. What SparkLayer and BSS Commerce do for the same problem.

It came back with seven roadmap items.

Sequenced by priority. Cost math on every one. The cheapest change in the list would save 30 to 40 percent on my event-handling bill. A per-merchant cap would bound the worst-case cost of a single bad-actor install at about five dollars a month. One item, marked critical, would prevent a different kind of cost trap that would otherwise run to 1,800 dollars a month at 500 merchants. That number is the kind of fact you only find if you read AWS’s own pricing pages carefully.

The second Claude then wrote the seven items into the PM agent’s file format directly. Not just the research notes. The actual roadmap entries, ranked and detailed, ready to build. The PM agent’s role by the end of that workflow was holding the file format the other tool knew how to write through.

One of the seven items doesn’t add new work. It corrects work the PM agent had already structured. Work I’d approved.

On April 7, before any of this, I’d had an earlier hardening item sitting in the OrderFlow roadmap. Two of its tasks said to use a specific HTTP header value as the way to recognize duplicate events. I’d looked at it. I’d approved it. I never actually checked what the header meant.

The agent had the intent right. Don’t process the same event twice. The spec was wrong. The header it specified identifies the subscription, not the event. It’s the same value every time a given subscription fires. Use it as a deduplication key and you drop every event after the first. Permanently. On every store.

The second Claude found it by going to Shopify’s own current documentation and checking what each header actually means. It wrote a new opportunity explicitly to override the bad spec before it shipped. The PM agent dutifully held that one too.

The substrate doesn’t validate the meaning of what gets written into it. It holds whatever shape the writer hands it. Including a bug I’d missed and approved.

The seven items were written and triaged the third week of April. The first ship date is May 14. The gap between those dates is roughly three weeks. In that gap, I went on vacation.

Nothing happened.

The agents didn’t quietly continue. The build loop didn’t pick up the specs the second Claude had written and turn them into code. The PM agent didn’t notice the items sitting at status “identified” and escalate. The roadmap file stayed the way I’d left it. The underlying systems stayed up. Events still arrived. The Monday digest still ran. But no feature moved forward across multiple agents over multiple days.

I came back. I sat back down at the keyboard. Implementation started again.

Four of the seven items shipped to production within seventy-two hours. The per-merchant cap. The cheap infrastructure changes. Queue separation between free-plan and paid-plan traffic, so a noisy free-plan merchant can’t slow down a paying one. A way to see per-customer usage without paying through the nose for it. Two more are in flight. One is parked behind a 100-dollar-per-month cost trigger that doesn’t exist yet.

But not autonomously. Several of the implementations needed me at the bench. Moments where the agent got something almost-but-not-quite-right and twenty-five years of software experience were required to correct it in real time. I don’t remember the specific catches now. I just remember the rhythm of it. Look at what the agent produced. Notice the subtle thing. Push back. Watch it adjust.

The autonomy score for this experiment measures something specific: what percentage of operational decisions required no human input. The March number was 42 percent. The April and May numbers aren’t logged yet. The instrument itself has gone idle for forty-three days, which is its own observation.

The vacation surfaced a different number the instrument doesn’t measure. Call it runtime autonomy. The question isn’t “which decisions need me.” It’s “does the system have forward motion when I’m not present.”

For decision-autonomy I can claim 42 percent. For runtime-autonomy the honest number from these three weeks is closer to zero.

The blocker isn’t technical possibility. The agents already exist. AWS sells a runtime called Bedrock AgentCore designed exactly for this: agents that call other agents. The PM agent could call a research agent. The research agent could call a Reddit-scanning agent. The PM agent could trigger a coding agent that turns a roadmap item into a pull request. All of that is buildable today.

Two things stop me.

The first is trust. I don’t yet trust this kind of orchestration to go right when no human is present. Every one of the four items that shipped after my vacation needed me catching small things in real time. A fully autonomous pipeline would have shipped versions of those items with the small things baked in. Some of those small things, accumulated over weeks, are how systems quietly drift into being broken.

The second is cost. The coding agent I use locally runs on a flat-rate subscription. Running the same coding agent from inside another agent means paying per call, per token, through an API. The math on a runaway loop is genuinely scary. One agent stuck in a retry. Another agent dutifully feeding it work. No human in the loop to notice. The bill arrives later.

Which means before I’d let this run unattended, I’d need another agent on top: something I’ve been thinking of as a CFO agent. The equivalent of the Andon Cord in a Toyota factory, where any worker can stop the production line the moment something doesn’t look right. A CFO agent watches spend per hour, per day, per task, and pulls the cord when the rate of work outpaces the rate of value. That agent doesn’t exist yet either.

So the gap between what’s possible and what I actually run is roughly: one well-instrumented orchestration runtime, plus the CFO agent, plus enough operating evidence to trust the loop. None of those is impossible. None of them exists in my stack on May 17.

This is the gap between the AI transformation pitch you’d hear at any enterprise AI vendor conference and the experience of actually running a portfolio on agents. The enterprise pitch is about replacing decisions. The lived reality is about replacing presence. The agents I have can make a lot of decisions. They cannot, yet, be present in the way that keeps work moving when I am not.

I’m testing this in one specific place. The customer support stack for both apps is being built so an AI-managed support agent can hold the room while I’m somewhere else. The hypothesis is that support is a domain where runtime autonomy might land first, partly because the cost ceiling on a single support interaction is much easier to bound than the cost ceiling on a coding loop. We’ll see. I’ll write about that one when it has a track record.

For now: I went on vacation and the agents stopped. Four roadmap items shipped because I came back and shipped them. The agent that holds the format made all of it possible. The agent that holds the format didn’t move anything by itself.

I don’t yet know whether that’s a property of these specific agents or a property of agents, of trust, of cost, or of all four at once.