Day 50: Tamper-Proof Metadata — Signing Tags and Timestamps at Kernel Speed

Here’s the ticket a junior engineer picks up: “Every request that flows through our multi-tenant WASM pool needs a metadata tag — a routing hint, a priority class, a cache key — and that tag has to be tamper-proof. A malicious or buggy tenant component must not be able to forge or replay it.”

The junior solution, nine times out of ten, looks like this:

Wrap the tag in a JWT. Sign it with jsonwebtoken + ECDSA (P-256). Verify it downstream with the same crate. It compiles, the demo works, everyone claps.

Then it goes into a NexusCore pool doing 400K requests/second across 200 co-resident tenant components, and it falls over — not because ECDSA is “slow” in the abstract, but because of what the crate is doing underneath, on every single call, that you never see:

jsonwebtoken deserializes the header and claims through serde_json, which means a fresh heap allocation graph per token — a String for every field, a HashMap for arbitrary claims, dropped and reallocated per request.

ECDSA verification via ring/openssl bindings pulls in BIGNUM arithmetic paths that were designed for TLS handshake frequency (tens per second per connection), not per-packet frequency. Each verification is a nontrivial modular inversion over a 256-bit field — hundreds of multiply-and-reduce steps, none of them vectorizable in the way symmetric primitives are.

The signing key lives behind a Mutex<SigningKey> in a shared service. At 400K QPS that mutex becomes a cache-line ping-pong target across cores — every acquire invalidates the line on every other core holding it, which is a MESI protocol tax you pay whether or not there’s real contention on the data.

None of this shows up in a load test against localhost with warm caches. It shows up in production as p99 latency that quietly triples under multi-tenant load, and nobody can point to a single slow function because the damage is distributed across allocator churn and cache coherency traffic.

Two mechanisms compound here.

Heap fragmentation from JWT parsing. serde_json allocates dozens of small, short-lived objects per parse. Under sustained load, the allocator (glibc malloc or jemalloc) starts serving these from increasingly fragmented arenas. You don’t see this in top — RSS looks stable — but you see it in perf stat as a climbing dTLB-load-misses count, because fragmented small allocations spread your working set across more 4KB pages than a compact representation would need. On a 200-tenant pool with a 2MB L2 per core, that’s the difference between staying resident and thrashing.

TLB shootdowns from key-rotation. If you rotate the signing key (you should — NexusCore rotates every 90 seconds), a naive implementation calls mprotect or reallocates the key material, which triggers a TLB shootdown: an IPI (inter-processor interrupt) to every core that might have cached a mapping to that page, forcing them to invalidate and refill. At rotation time, on a 32-core box, that’s 31 IPIs landing simultaneously on cores that were mid-flight on unrelated tenant work. Measured on a c6i.8xlarge under a synthetic 400K QPS mixed workload, a naive mprotect-based key swap costs a 340µs global stall — three orders of magnitude past our per-request latency budget.

The naive JWT approach isn’t wrong because ECDSA is “insecure” or “too slow” as a primitive. It’s wrong because it inherits a general-purpose cost model — arbitrary claims, arbitrary algorithms, arbitrary key material — for a problem that is actually narrow and fixed-shape: sign 128 bytes, verify 128 bytes, do it a few hundred thousand times a second, without touching the heap.

We split tamper-proofing across three planes, matching the trust boundary to the layer that can enforce it cheapest:

1. Kernel plane (eBPF/XDP): a fast, symmetric integrity check — a keyed hash, not a signature — applied at line rate as packets carrying metadata tags cross the XDP hook. This catches gross tampering and replay before the packet ever reaches userspace, at sub-microsecond cost, using a per-CPU key held in a BPF_MAP_TYPE_ARRAY so there’s zero cross-core contention.

2. Userspace plane (Rust loader): the actual asymmetric signing authority. A single-writer, batch-signing loop that takes tags off a RINGBUF map, signs them with Ed25519 (not ECDSA — more on why below), and writes the signed record into a second ring buffer that WASM components read from. Key rotation is double-buffered, never mprotect-in-place, so there’s no shootdown.

3. WASM plane (no_std verifier component): each tenant component links a tiny no_std verification routine compiled to wasm32-wasip2. It checks the Ed25519 signature over the 128-byte metadata block using a bump arena allocator — one fixed-size scratch buffer, reused every call, zero heap traffic, zero malloc.

Why Ed25519 over ECDSA/P-256 here: Ed25519 verification is a fixed, branch-free sequence of Curve25519 field operations — no modular inversion in the hot path, no reliance on a CSPRNG per-signature (EdDSA is deterministic), and it’s roughly 2-3x faster to verify on commodity cores. It also has no “cofactor” footguns for batch verification, which matters when the WASM verifier wants to check a burst of tags in one epoch tick rather than one at a time.

https://github.com/sysdr/nexus-core-devops-engineering-p/tree/main/lesson50/nexuscore-day50-tamper-proof-metadata

The signed record. Every tag is packed into a fixed, cache-aligned 128-byte struct — never a self-describing format like JSON or CBOR:

Fixed offsets mean the WASM verifier never parses — it slices. No branches on field presence, no allocation, no bounds-checking beyond a single length assert on the imported memory region.

Kernel-side fast check. The XDP program keeps a per-CPU BPF_MAP_TYPE_ARRAY holding the current epoch’s symmetric key. On each packet it computes a truncated SipHash-1-3 over tag_id || timestamp_ns (cheap: two 64-bit multiplies, a handful of rotates) and compares it against a tag embedded in the packet’s metadata by the loader. Mismatch → XDP_DROP before the packet costs you a single userspace cycle. This is not the authoritative signature — it’s a kernel-cheap tripwire that rejects the 99% of tamper/replay attempts that don’t bother forging the SipHash, so only genuinely suspicious traffic pays the Ed25519 cost downstream.

Lock-free key rotation. The Rust loader holds two Ed25519 signing keys in a [SigningKey; 2] array behind an AtomicU8 index. Rotation is bpf_atomic_cmpxchg-style: write the new key into the inactive slot, then flip the index with a single atomic store. Readers (the signing loop and the kernel map updater) always read the index first, then the slot — no reader ever observes a half-written key, and there’s no mprotect, so no TLB shootdown.

WASM verification. The no_std component imports a 128-byte memory region per call and a single static 4KB bump arena (enough for the Ed25519 verification scratch state, freed by resetting the bump pointer, not by free). wasmtime‘s PoolingAllocationConfig keeps the component’s linear memory pre-allocated across invocations, so there’s no mmap per call either — the entire verify path from “bytes in” to “bool out” touches zero syscalls

Metric Target Why it matters Kernel SipHash check latency < 80ns Line-rate tripwire; anything higher and you’re back to userspace-speed filtering Ed25519 sign throughput (loader) > 250K sigs/sec/core Batch-signing headroom above peak tag-issuance rate WASM verify latency < 3µs Must stay inside the epoch-interruption budget, not dominate it Key rotation stall < 5µs (p99) Should be invisible to in-flight requests; anything near the old 340µs mprotect number is a regression Heap allocations per verify call 0 Any nonzero count means the bump arena reset logic broke

Export all five via the Axum /metrics endpoint on the loader; the kernel-side numbers come from a PERCPU_ARRAY histogram map read by bpftool map dump and re-exported as Prometheus counters.

Requirements: Rust 1.80+ with wasm32-wasip2 target, cargo-component, wasmtime 25.x, libbpf + clang/llvm for CO-RE compilation, root or CAP_BPF/CAP_NET_ADMIN for XDP attach.

Extend the loader so that key rotation is triggered not on a fixed 90-second timer but on a Count-Min Sketch-based anomaly signal: track per-tenant tag-issuance rate, and if any tenant’s rate crosses 3 standard deviations above its trailing EMA, force an out-of-cycle rotation scoped to only that tenant’s key slot (you’ll need to extend the fixed-key-pair design to a small per-tenant HASH_OF_MAPS). Report the added kernel-side lookup cost this introduces, and whether it still holds the sub-80ns SipHash budget.