Declarative Without the Elasticity Tax: Margo + Quadlets for Cloned Production Cells

Battery assembly. Two — or twenty — physically identical production cells. Each cell is a redundant ring of three switches, the same model, the same firmware, the same control logic, the same 192.168.1.x address space. Last quarter a ring reconfiguration in one cell halted production. The team wants centralized monitoring. They cannot have it.

Every cell sits behind a NAT firewall. Centralized SNMP polling from the backbone is dropped at the boundary — by policy and by design. Giving the SOC inbound reach into each cell is a worse cybersecurity posture than no monitoring at all, and the 192.168.1.x collision across cells means the backbone could not address a switch by IP even if the firewall let it through.

The solution inverts the data flow. One small device sits inside each cell, polls the three switches locally via SNMPv3, and publishes outbound — through the firewall, through the NAT, to a backbone MQTT broker. The device is identical across all N cells. The two workloads on it are identical. Atomic update, bit-identical recovery, no inbound exposure.

How that device is put together lives in two companion deep-dives: The Spec-Driven Stack: Quadlet, Margo, and the WASM Migration Path — the five-layer stack and the one-field WASM migration path — and Two Workloads, Outbound Only: Quadlet Units for NAT-Isolated Production Cells — the two .container units, directive by directive. This post stays on the decision: how to compare, how to score, and where Margo enters.

This post is the talk from DevConf.CZ 2026 Brno (room E112, June 18, 2026) made durable — shipping five days later so the room’s strongest objections could be folded in. The case this talk makes: Podman Quadlets offer a way out of the triangle — for this domain — by treating the deployment spec as a file-system artifact, not daemon state.

The full talk — co-presented with Louis Thomann (Cloud Native Specialist) — is now public:

DevConf.CZ 2026, Brno — with Louis Thomann (Cloud Native Specialist).

Part 1: The Constraint Domain#

The opening story, stripped to its constraints, is an ISA-95 (IEC 62264) Work Cell — battery assembly here; a welding cell or a packaging line elsewhere. Naming those constraints before scoring any technology is what makes the framework honest rather than retro-fitted:

• Statically configured, deterministic clone. Every battery cell is an exact replica of its siblings — same switches, same workload set, same configuration. The requirement is identical deployment state across all N cells, with atomic forward and rollback — not dynamic scheduling.
• 10-year unattended operational horizon. Nobody walks that factory floor to babysit deployments. Updates arrive over the network and must apply atomically or not at all.
• IEC 62443 SL2 target. The same posture that drops inbound SNMP at the firewall: every privileged daemon and process boundary not directly serving production must be justified and documented.
• Sub-1 GB RAM gateway. The one device inside each cell is an industrial ARM gateway in the 512 MB–1 GB range. The K3s single-node floor of ~1.4 GB is a hard architectural exclusion, not a tuning question.
• No horizontal elasticity requirement. Resilience and atomic update delivery are requirements; distributing workloads across additional nodes is not.

Every score that follows is a score within this domain. The same framework applied to a different domain reveals different fits — which is the point, not a hedge.

Part 2: How to Compare — The Three Criteria#

A deployment primitive for a 10-year OT window deserves a comparison axis that does not shift depending on which tool is being defended. Before scoring any technology, three criteria need naming — each measuring a different dimension.

Criterion 1 — Cloned-Cell Drift Control (lifecycle, temporal). Whether the deployment unit pins and owns its dependencies, AND whether N sibling cells stay bit-identical across an 18-month redeploy cycle. A deployment that silently diverges from its declared state is not suitable for an unattended 10-year horizon. A fleet that cannot guarantee bit-identical recovery across sibling cells is not suitable for cloned-cell deployment.

Practitioner question: if I image cell-7 today and recover cell-23 from a power cut next year, will they be bit-identical?

Criterion 2 — Cybersecurity & Audit Surface (security, audit). Capability boundaries, daemon trust surface, audit-trail-per-workload, and attestation cost. This criterion combines workload-level isolation (the namespace boundary that bounds a breach) with platform-level audit surface (the trust surface an IEC 62443 auditor must walk). The two move together less often than they move apart, so the score reflects the enforced boundary a workload actually gets, with the platform’s audit-surface cost noted per technology rather than averaged away.

Practitioner question: how many distinct artifacts does an auditor read to certify one workload — and how large is the trust boundary of the platform itself?

Criterion 3 — Operational Footprint (human + resource — inverted axis: HIGH = lighter). The subsystems the on-call team must understand AND the RAM/CPU floor consumed before any workload runs. Two operational dimensions that move together on constrained OT edge hardware: a heavier platform demands both more memory and more specialised skills. HIGH on this row means lower demand on both dimensions; LOW means the platform demands a sustained team plus a gigabyte-class floor.

Practitioner question: at 3 a.m. on a constrained ARM gateway, what does the on-call engineer need to know, and how much RAM is the platform itself eating?

No technology is named in this section. The criteria stand on their own.

Part 3: Scoring Each Technology Honestly#

Three criteria, four technologies, twelve cells. The matrix shows where each technology earns HIGH — and where it does not. The load-bearing methodology line at the bottom of the figure is the same as before: higher is not universally better; domain fit, not score totals, determines the right choice.

Packages — apt/dnf + Ansible#

Packages earn LOW on Cloned-Cell Drift Control. apt-cache state drifts between playbook runs; Ansible governs only the state it knows about. Over an 18-month redeploy cycle, the gap between playbook-declared state and actual device state widens — bit-identical recovery across sibling cells is not a property apt/Ansible can guarantee by design.

LOW on Cybersecurity & Audit Surface is structural: there is no workload-level kernel namespace boundary between co-located processes, and the audit surface for an IEC 62443 auditor is the entire host. No per-workload isolation boundary exists unless added explicitly outside the package management layer.

HIGH on Operational Footprint is the genuine win. The deployment platform is the OS itself: kernel + systemd + apt, zero resident daemon overhead. An OT-native sysadmin faces no new learning surface. For environments staffed by experienced Linux sysadmins running standalone workloads, Packages is the appropriate tool.

Compose — Docker / Podman Daemon#

MED on Cloned-Cell Drift Control: image digests pin workload state, but compose.yml drifts at the host level without a fleet abstraction. Two cells that started from the same compose file can diverge through manual host-level changes over 18 months. The workload layer is pinned; the platform layer is not.

MED on Cybersecurity & Audit Surface: approximately 15 inline security knobs — security_opt, cap_drop, no-new-privileges, read_only, sysctls — are real controls. The highest-risk element is the Docker socket running as root; Podman rootless mitigates the socket vector without changing the knob count. The audit surface is the compose file plus host configuration — wider than a single systemd unit file, narrower than a full Kubernetes cluster.

HIGH on Operational Footprint reflects the staffing profile that Compose suits. DevOps engineers from cloud-native backgrounds onboard quickly; the YAML stack file is a familiar artefact. Compose wins where the staffing profile is cloud-native, not OT-native.

Kubernetes (K3s)#

HIGH on Cloned-Cell Drift Control: etcd-backed reconciliation plus GitOps continuously heals divergence between declared and running state. This is the strongest drift-control mechanism in this comparison — continuous reconciliation rather than point-in-time enforcement.

HIGH on Cybersecurity & Audit Surface — but realised only when a team with CKA-equivalent skills is present to configure, audit, and maintain it. Unconfigured K8s defaults are no safer than Compose. The policy depth — RBAC, NetworkPolicy, Secrets-store CSI, SecurityContext, Pod Security Standards, admission controllers (OPA Gatekeeper, Kyverno) — is the richest in this comparison, but that depth requires sustained expert operation to activate.

LOW on Operational Footprint reflects both dimensions of the criterion. The resource floor for the standalone single-node install (server and agent co-located) sits at ~1.4 GB of RAM at zero workloads, per official K3s resource profiling data — and later versions trend higher as the control-plane surface grows. Disabling the bundled add-ons (--disable for traefik, servicelb, metrics-server, and local-storage) trims the footprint, but the kube-apiserver, controller-manager, scheduler, kubelet, and embedded datastore remain — the irreducible single-node control plane sits in the gigabyte class, not the half-gigabyte class an OT gateway budget assumes. The commonly cited half-gigabyte figure is the agent-only worker-node minimum in a multi-node cluster — a different scenario where the control plane runs on a separate server node. On a sub-1 GB industrial gateway, the single-node floor is a hard architectural exclusion, not a tuning question. The staffing dimension compounds it: RBAC, secrets-store integration, network policies, control-plane patching, and etcd backup require a sustained team.

Margo + Quadlet — systemd-native#

HIGH on Cloned-Cell Drift Control: each .container file IS the deployment specification — there is no daemon state to diverge from. Margo’s ApplicationDescription schema delivers the same workload spec to every cell in the fleet. The combination of workload-level pinning (the .container file) and fleet-level synchronisation (the ApplicationDescription) is what makes bit-identical recovery across sibling cells a designed-in property rather than an operational discipline.

HIGH on Cybersecurity & Audit Surface: each Quadlet unit runs under its own cgroup v2 limits and kernel namespace boundary enforced by the OS. The unit exposes more than 100 security-relevant directives across systemd.exec(5), systemd.resource-control(5), and related man pages — the full breakdown is in Why Quadlet Is Different, Part 1. Unlike the K8s HIGH, this one does not carry a team-skills qualifier for the security boundary itself — the boundary is enforced by the kernel regardless of operator expertise.

MED on Operational Footprint is the honest cost. systemd familiarity is necessary but not sufficient. The Margo fleet layer — ApplicationDescription schema, agent, OCI delivery pipeline — adds learning surface beyond what an experienced systemd operator already holds. The differentiator from Kubernetes is the resource floor: Quadlet adds zero resident RAM before the first workload, because systemd is already PID 1.

Part 4: Reading the Matrix Against the Domain#

Given the constraint set — ISA-95 Work Cell, 10-year horizon, IEC 62443 SL2, sub-1 GB gateway, no elasticity — the matrix column that fits the domain is Margo + Quadlet: HIGH on Cloned-Cell Drift Control, HIGH on Cybersecurity & Audit Surface (kernel-enforced, no team-skills qualifier), MED on Operational Footprint as the honest cost of the fleet layer. The sub-1 GB floor excludes K3s regardless of its top two cells. No other column matches all three domain constraints simultaneously. The same matrix applied to different constraint sets reveals different fits: Kubernetes for multi-node clusters with a resident CKA team; Compose for cloud-native DevOps backgrounds; Packages for single-machine OT deployments staffed by experienced Linux sysadmins.

The case this talk makes is that Margo + Quadlet fits the domain as I have drawn it. Push back at the scoring (any cell in the matrix) or at the domain boundary itself (Part 1) — the domain is stated, the scores are reasoned, and the post-talk addendum slot below is where the room’s objections land.

Post-talk addendum (folds in after June 18): the strongest objection room E112 raised against this framework will be added here before publication — the durable version earns its claims from the room, not ahead of it.

Part 5: Where Margo Enters — Three Open Proposals#

Quadlet solves the device-level problem. Margo solves the fleet-level problem. The ApplicationDescription schema delivers the workload specification; Quadlet executes it on the device. Three independent proposals under active TWG review describe how the fleet-level spec grows to describe Quadlet workloads natively.

A single monolithic proposal covering the publishing pattern, Compose, and Quadlet was closed and re-submitted as three independent proposals — so disagreement about one component type cannot block the others.

docs: SUP-00 The Helm Way — canonical OCI component publishing pattern names the five-element OCI publishing pattern Helm already implements — storage, reference, media types, integrity, parametrization — so future component types conform by citation. Informational; no normative changes. Deep-dive: The Helm Way — Margo OCI Publishing Pattern.

Compose OCI registry publishing and archive structure applies the pattern to Compose: OCI registry storage, two Margo media types, normative archive structure, versioned compose.v1 profile type, removal of packageLocation.

Quadlet component type (Podman 5.0+ / systemd) applies the same pattern to Quadlet: quadlet deployment profile type, QuadletDeploymentProfile and QuadletComponent schema classes, Margo media types, normative Quadlet Archive Structure, LinkML desired-state patch. Additive only — zero breaking changes to existing Margo APIs or OCI registries.

All three remain open proposals under active TWG review. None is ratified; none is part of Margo PR1 or the current pre-draft specification. The TWG Chair, Armand Craig, convened the Compose/Quadlet working group at the bi-weekly meeting at my request. The shaping window for Margo GA1 (targeting 2026) is open now. Engineers who wait for ratification before engaging have already missed the opportunity to shape what gets ratified.

Closing: The Case, Open for Scrutiny#

The framework produces a constraint-domain-specific fit, not a universal ranking.

Applied to the cloned production cell domain — ISA-95 Work Cell, 10-year horizon, IEC 62443 SL2, sub-1 GB gateway, no elasticity — the case this post makes is that Margo + Quadlet fits the constraint set as stated. The .container file is the deployment specification. The ApplicationDescription is the fleet-level contract. The atomicity guarantee is file-system semantics. The security perimeter is cgroup v2 and kernel namespaces, enforced by the OS.

Two places to push back: the scoring (any cell in the matrix is open to challenge) and the domain boundary (Part 1 draws it — argue it is drawn wrong, and a different column may score differently). The post-talk addendum in Part 4 is where the room’s strongest objections from E112 land before the 2026-06-23 ship date. That slot is not decoration — it is how the durable version earns its claims.

If you have constraint sets not represented here, or a matrix cell that does not hold in your environment, the WG mailing list, margo/specification-enhancements GitHub discussions, and the comments on this post are the right places. I genuinely want to hear what I am missing, what could be done better, and whether this is a real alternative for your domain.

Background: Podman Quadlets: A Lightweight Deployment Profile for the Industrial Edge. Series entry points: Part 1, Part 2, Part 4.

The leadership dimension of writing a standard in public — including the Doodle reply that said “Thursday works” — is explored in the companion piece at javatask.systems.