The Software-Defined Industrial Edge, Part 4: Sovereign Inference — Local-LLM Serving on the Box

When an anomaly detector on a press line needs to query a language model, there are two architectures to choose from. In the first, the inference call leaves the factory network, traverses the WAN, reaches a cloud endpoint, and returns a completion. Latency is measured in hundreds of milliseconds. The data crosses a network boundary. The model’s license depends on a cloud subscription that requires continuous internet connectivity to validate.

In the second architecture, the inference happens on the box the press line is already connected to. The call never leaves the OT network. Latency is measured in tens of milliseconds. The license is validated locally, against a hardware fingerprint, without any outbound traffic.

The proof point is stark: on the e-Kanban node, gated on-box inference costs ~€22–33 a year in electricity where every-cycle cloud inference would run ~$700–$40,800 a year depending on model tier — a 30×-to-over-1,700× structural gap (FTPFI Part 3).

These two architectures have different security postures. A high-security OT zone (SL2 or above) under IEC 62443 does not permit arbitrary outbound connections to cloud endpoints. That is not a preference — it is a control requirement. The first architecture is structurally incompatible with it. The second is not.

This post is about building the second architecture: what the inference stack looks like, how to package it as a KitOps ModelKit (the lifecycle mechanism from Part 3), and how to solve the licensing problem that every air-gapped deployment hits.

The air-gap licensing paradox#

Sidebar — The Air-Gap Licensing Paradox

Most commercial model-serving tools and SaaS inference endpoints use an online license-validation model: the software contacts a remote server on startup, periodically, or per-inference call. On an air-gapped OT network, that contact fails. The software will not start, or degrades to an error state.

This is the air-gap licensing paradox: the network isolation that makes a deployment IEC 62443-compliant breaks the licensing mechanism of the tool running inside it.

The resolution is host-based offline licensing — a license token the software validates locally, against a cryptographic signature and a hardware fingerprint, without any outbound call. A license daemon runs as a local service. Queries go to a UNIX domain socket or a loopback REST endpoint. Nothing leaves the host.

This is not a new pattern. Industrial automation software has used dongle-and-daemon licensing for decades. What is changing is that the LLM ecosystem is being asked to meet the same requirement — and a small but growing number of inference tools now support it.

On-box inference: GGUF and NPU offload#

The lowest-friction path to on-box LLM inference is llama.cpp serving a GGUF-quantized model. GGUF is a single-file format that carries model weights, tokenizer vocabulary, and inference parameters as a self-contained bundle. Quantized variants — Q4_K_M for maximum compression, Q8_0 for better accuracy at moderate size — reduce DRAM requirements by 4–8× compared to full-precision weights. That matters on edge hardware with 8–32 GB of unified memory.

NPU and iGPU offload is what brings inference latency into a range that is useful for factory diagnostics. llama.cpp’s --ngl <N> flag (number of GPU layers) offloads transformer layers to the NPU or integrated GPU. On hardware with a dedicated edge NPU — Intel Core Ultra via the OpenVINO backend, AMD Ryzen AI via the Ryzen AI SDK (Ryzen AI NPU backend, currently experimental in llama.cpp), Qualcomm Snapdragon X Elite — a 7B-parameter model runs at latencies acceptable for non-real-time anomaly reasoning. Neither path requires a discrete GPU; both are available on the class of x86 and Arm edge box that an OT team can actually deploy to a Zone 1 cabinet.

Vision inference sits alongside the language layer. Ultralytics YOLO26 is NMS-free — the non-maximum suppression postprocessing step that previously required custom CPU code is eliminated from the architecture — and is designed from the start with edge-class hardware resource budgets in mind. On an edge box handling line-of-sight inspection, YOLO26 processes frame-by-frame object detection while the GGUF language model handles diagnostic reasoning over the structured detection output.

Sensor streams and video feeds on a factory floor are continuous and voluminous. Routing that data to a cloud endpoint adds latency, creates a network dependency, and, in many regulatory contexts, creates a data-residency problem. On-box inference resolves all three. The data does not move; the model does.

Packaging the inference stack as a ModelKit#

In Part 3, apps and model artifacts were placed on the same OCI lifecycle using KitOps ModelKits — versioned, signed, pulled from a registry, rolled back on failure. The local-LLM inference stack is a model artifact with exactly that shape.

A Kitfile for the edge inference server:

# Kitfile — local-LLM ModelKit for edge inference
manifestVersion: "1.0.0"

package:
  name: edge-llm-server
  version: 0.2.0
  description: "llama.cpp server with 7B GGUF model, NPU offload config"
  authors:
    - "Andrii Melashchenko"

model:
  name: Mistral-7B-Instruct-Q4_K_M
  path: ./models/mistral-7b-instruct.Q4_K_M.gguf
  framework: gguf
  license: Apache-2.0

code:
  - path: ./config/
    description: "llama.cpp server launch config, NPU backend flags, license socket path"

docs:
  - path: ./README.md
    description: "Deployment notes for air-gapped OT environments"

The llama.cpp launch config inside ./config/server.sh:

#!/usr/bin/env bash
# server.sh — llama.cpp edge-LLM server, NPU/iGPU offload
llama-server \
  --model /models/mistral-7b-instruct.Q4_K_M.gguf \
  --ngl 33 \               # full NPU/iGPU offload (33 > 32 layers → clamps to all)
  --ctx-size 4096 \
  --n-predict 512 \
  --host 127.0.0.1 \       # bind to loopback only — never expose to factory network
  --port 8080 \
  --log-disable

--host 127.0.0.1 is not optional. An LLM server bound to a factory network interface is a remote code execution risk under any OT threat model. Loopback binding is the minimum; an application-layer firewall rule that blocks 8080 at the zone boundary is the belt to go with those suspenders.

This ModelKit is pulled from the OCI registry by a Podman Quadlet unit. It carries the same version, the same rollback semantics, and the same pull-on-change lifecycle as the application containers running beside it. The runtime is Podman — a CNCF Sandbox project since January 2025 — supervised by systemd.

Offline license validation — the pattern#

With inference on-box, the licensing handshake is the remaining WAN dependency to eliminate.

A host-based license daemon listens on a UNIX domain socket:

/var/run/edge-license/license.sock

On startup, the LLM server queries the daemon over that socket:

# Verify license locally — no internet, no outbound traffic
curl --unix-socket /var/run/edge-license/license.sock \
     http://localhost/v1/verify \
     -d '{"product": "edge-llm-server", "hwid": "'$(cat /etc/machine-id)'"}'

The daemon verifies three things without leaving the host:

1. The license token was issued for this hardware fingerprint (/etc/machine-id or a TPM-backed identity)
2. The token’s cryptographic signature is valid (offline RSA or Ed25519 check — no certificate authority call)
3. The license has not expired

The response is {"status": "valid", "expires": "2027-06-01"} or a structured error. The application proceeds or halts locally. No DNS query. No TCP connection to an external host. No dependency on network reachability during runtime.

This satisfies IEC 62443 zone-and-conduit network isolation requirements: no unsolicited outbound traffic from the control-zone host, and no runtime dependency on cloud reachability. The license token is provisioned once, offline, and carried with the deployment. Renewal is a provisioning event, not a network event.

What this unlocks — and what it does not#

On-box inference closes the WAN dependency in the inference layer. The model is versioned in the same OCI registry as the application containers (Part 3). It runs on locally licensed hardware inside the OT zone. The data-residency constraint is satisfied by design, not by policy.

What it does not do is make agent behavior reliable in production. Inference capability is necessary but not sufficient. The 88% of agentic pilots that do not make it past proof-of-concept do not fail because the model could not generate text — they fail because there is no deterministic harness governing what the model is allowed to do, with what context, and under what safety constraints. That harness is the subject of Part 5 — The 88% Wall.

For the governance lens — who owns the read-only boundary between the agent and the OT system, and which team holds the contract for that boundary — the companion post on javatask.systems covers the organizational design decision.

For the full open-standards stack that frames where this inference layer sits — GGUF/ModelKit as the model-packaging layer, Podman Quadlets as the runtime, Margo as the interoperability layer — see the hub post: The Open Standards Stack of the Software-Defined Edge.