The Sovereign Stack series works through the migration of Digital Coworker toward an air-gapped on-premises target. This post addresses the general mechanism that migration must solve: what replaces the managed durability layer when the cloud is not reachable.
On a public cloud, a managed workflow scheduler gives you durability for free — a crashed run resumes from the last checkpoint because the cloud service owns the state. Pull that service out and you are left with an agent loop that has no memory of what it completed before it died. In a regulated environment — energy, defense, healthcare, or any deployment where the managed backend is unreachable — a run that cannot be resumed from its exact failure point is not just inefficient; it may be unsafe. Duplicate side effects from a re-run that does not know what already executed are a correctness problem, not a retry problem.
The self-hosted substrate that reconstructs this guarantee has three interlocking parts: durable execution, idempotency gates, and liveness monitoring.
Durable execution: the state store as checkpoint#
Every tool call, model invocation, and agent handoff must be written to a durable, self-hosted state store the moment it completes — committed immediately, not batched. The state store is the execution record.
On crash, the orchestration layer does not restart from the beginning. It replays the committed record: completed steps are retrieved from the store and their results reinjected into context. Execution resumes from the exact failure point.
The substrate technology is standard infrastructure: a local Postgres or Redis instance running within the same network perimeter. What matters is write-on-complete discipline: a step is not considered done until its record is committed. In a cloud deployment the managed service owns this discipline. Self-hosted, you own it explicitly.
Idempotency gates: making retries safe#
Durable execution handles recovery from crashes. Idempotency gates handle a distinct problem: a step that completes but whose record write fails, or a network partition that causes the same step to be dispatched twice.
Without idempotency, a retry of a non-idempotent tool — a configuration push, a write to a data store — produces a duplicate side effect. The retry looks identical to the original dispatch, and the tool executes again.
The gate is explicit execution tracking on the state store. Before a tool executes, the orchestration layer checks for a record with the same step identifier. If the record exists and is complete, the step is skipped and its prior result returned. For non-idempotent tools, a transactional write lock around the check-and-execute sequence ensures two concurrent dispatches of the same step cannot both pass simultaneously.
The discipline is categorical: classify every tool at registration time as idempotent or not. Non-idempotent tools get a gate. This is a design requirement, not something deferrable to the retry path.
Liveness monitoring: the separate supervisor#
A crashed agent does not announce itself. Without managed infrastructure watching for failed processes, a silent agent is indistinguishable from a slow one until tasks stop completing.
The liveness layer is a separate supervisor process with one job: monitor heartbeats and manage leases. Each active agent emits a heartbeat at a fixed interval to the state store. When an agent’s last heartbeat exceeds the lease expiry threshold, the supervisor marks its pending tasks as unassigned and routes them to an active worker.
The heartbeat interval must be short enough to detect failures within an acceptable window; the lease expiry long enough not to false-positive on transient slowness. Task reassignment must be idempotent — the new worker consults the state store before executing any step, inheriting the existing checkpoint rather than starting over. The supervisor itself is lightweight: access to the state store, a heartbeat reader, and the ability to mutate task assignment records.
The OT example#
An air-gapped plant-floor cell running a multi-agent diagnostics workflow illustrates the failure mode this substrate is built for. The cell has no external connectivity — the OT network is physically isolated. If an agent process dies midway through a topology correlation query that spans three agent handoffs, there is no cloud backend to restart it. Without the self-hosted substrate, the entire run is lost. With it, the surviving agents detect the liveness gap, the supervisor reassigns the stalled tasks, and the new worker resumes from the last committed checkpoint. The workflow completes without re-executing the handoffs that already landed.
The plant-floor cell is one example. The same substrate requirement applies to any air-gapped environment where multi-agent runs must survive process failures without re-triggering completed side effects.
Where Dapr fits#
A durable-execution runtime such as Dapr implements the persist-replay-resume pattern as a managed primitive — automatic checkpointing after every activity, effectively-once execution semantics via an actor placement service and idempotency-safe activity replay, and crash recovery by replaying the persisted event log. It is a category exemplar for what the self-hosted substrate needs to provide.
The concrete Strands and Dapr integration for the Digital Coworker on-premises migration is the subject of the Sovereign Stack series, which works through that implementation as the validation cycle produces evidence. This post is the framework-neutral mechanism; the Sovereign Stack series is the Digital Coworker-specific cut.
Topology and team structure on top of this substrate#
The durable substrate is what makes the run survivable. Which topology you run on top of it — hierarchical, DAG, peer-to-peer — is a separate question, driven by the task shape, the coordination economics of the specific workload, and the team’s operating model. The fan-out/fan-in coordination economics are covered in Agentic AI on AWS: Orchestrating Agent Teams. The full topology decision space, including the attack-vector considerations that come with private-CA TLS and A2A under air-gap constraints (covered in a follow-up post on the protocol security layer), lives in the Agent Collaboration Patterns Blueprint. The organizational dimension — topology as org decision — is the companion piece at javatask.systems.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.