Notes on making LLMs run well on-device — MLX, Apple Silicon, tokenizers, and the seams where models meet real hardware.
The Vesuvius Challenge's July 2026 Progress Prizes named my Zarr 3 compatibility fix and chunk cache for the Python vesuvius library. Here's what was actually awarded, and why the boring data-plumbing work was worth writing up.
I tried to turn the best-scoring pockets from my PHerc1203 readability atlas into a readable surface. They were torn. An opposite coherence-first search found a different failure mode: the seven successfully inspected candidates were densely packed. Together, the searches motivate a joint coherence-and-separability test.
A CPU-only survey of PHerc1203 found that most sampled sites fail a local geometry gate, while a few rare pockets initially looked comparable to known-readable PHerc1667. Follow-up raw-CT inspection showed that the top pockets were torn fragments, clarifying what the metric can and cannot detect.
I ran the public ink model on a 2.4 µm scan of sealed PHerc1203 and got speckle instead of letters. Chasing why turned up a mistake in my own measurement, corrected it against the community's winding atlas, and left me with something more useful than a failed read: a CPU-only gate that flags rendered windows unlikely to isolate one papyrus sheet — before you spend the GPU.
I tested the public sarvam-translate model on three Windows 11 laptops with NVIDIA GPUs from 2014 and 2023. It ran fully GPU-offloaded on all three, with llama.cpp generation ranging from about 17.5 to 73.6 tok/s.
There is no single best zarr chunk size for reading the Herculaneum scrolls. On one region of Scroll 1, re-chunked five ways and read three ways, read amplification ranged from 1.3× to 176× — same bytes, only the chunk geometry and access pattern changed. I measured it on a laptop, and put the missing chunk cache back on the Python reader while I was there.
The same idea — store attention keys and values in fewer bits — measured on two stacks: MLX's --kv-bits on an M3 raises RAM and slows decode 4×, while vLLM's fp8 KV cache on an L4 doubles capacity and speeds decode up to 28%. KV quantization is a property of the implementation and hardware, not the technique.
A measured audit of Llama-3.1-8B on a rented L4: what vLLM's real, untouched default actually costs, what FP8 quantization gets you on top of it, and why a plausible-looking concurrency guess cost throughput instead of adding it. Every number here is measured, including a second, larger benchmark pass that fixed a real gap the first one had.
John Carmack argued AI inference should stream model weights from cheap flash instead of expensive HBM, since weight access is deterministic. That's correct for weights — but the KV cache grows, gets rewritten every token, and reads a shifting range. I measured what happens when you force it to behave like a fixed resource anyway.
Capping a model's KV cache instead of letting it grow without bound plateaus peak memory — but the instant a conversation outgrows the cap, recall of five planted facts collapses from 5/5 to 0/5, every trial, with zero partial credit. The attention-sink trick doesn't help, and here's why.
I ran the same long-context test on a 16 GB fanless M3 and a ₹23 rented NVIDIA L4. The laptop fits a 32k context on an 8B model and keeps every planted fact — but prefill balloons to seven minutes and its decode speed can't even be measured, because the fanless chip throttles. A measured, cross-hardware look at the KV-cache tax.
The applied chapter of a from-scratch project: after building tokenization, attention, gradient descent, a tiny GPT, and LoRA by hand, I ran a real QLoRA fine-tune — teaching Llama-3.2-1B to call tools on a MacBook, then measuring honestly what changed and what the adapter costs at inference. A 2.8M-parameter adapter (0.23% of the model) clearly helped on a small test; the debugging taught me the…
I wrote five GPU kernels from scratch on a 16 GB M3 to learn how LLM inference works at the metal. The most useful thing wasn't a kernel — it's a decision rule: never hand-write elementwise ops (the compiler already fuses them), reach for a kernel the moment a reduction appears, and remember the famous trick is rarely the hard part.
Atomic Chat advertises TurboQuant as 8x faster inference and 6x less memory. I tested the local MLX TurboQuant KV path on a 16 GB M3. It saved about 3-5% total peak memory and did not speed up generation — a useful reminder that KV-cache microbenchmarks do not automatically become whole-chat product claims.
Self-speculative decoding lets a model draft its own tokens by skipping layers — speculative decoding's speedup with no extra memory. I built it for MLX and swept 24 configs on an M3. Every one was slower than baseline, even though all were lossless. Here's why, and the paper that fixes it.
Single-stream decoding on Apple Silicon is bottlenecked by reading the model's weights out of memory, not by the math. Three techniques attack that directly — speculative decoding, diffusion generation, and self-speculative layer-skipping. I measured all three on a 16 GB M3. Each is right in theory and backfires in its own way: the bottleneck just moves one step further from the arithmetic.
LLaDA2.0-mini, a diffusion language model, runs on a 16 GB M3 and ties Qwen3-4B for the best answer-quality score I've measured (20/21). But it's slower than the fastest autoregressive model and uses 4× the memory of the lightest — and the exact reason I expected it to be fast on bandwidth-bound hardware turned out to be why it isn't. A measured look at where the bottleneck actually moved.
Apple shipped an official Python SDK for its on-device Foundation Model at WWDC 2026. I put the ~3B model through the same 21-task quality suite I use for MLX models: it ties a 4-bit Llama-3.1-8B (18/21), one question behind Qwen3-4B. Quality is the only fair axis to compare — and that limitation is itself the interesting part.
On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.
A 1B draft model speeds up Llama-3.1-8B by 20% on an M3 — at num_draft_tokens=2. Push that dial to 4 and decoding gets 25% SLOWER than using no draft at all. Here's the measured curve, and why low draft counts win when decode is bound by memory bandwidth.
Google's Gemma-4 E4B posts the same math and factual scores as the full 12B on an M3 MacBook Air — in 6.6 GB instead of 11.4, at 8.2 tok/s instead of 2.7 — so on a 16 GB Mac the E4B is the one to run. This is a size win, not a QAT one: the 12B's own QAT build doesn't shrink or speed it up. Honest numbers, measured under a real 2048-token load.
Re-running my own LLM benchmark, I found a bug that had inflated the quality scores in posts I'd already published. Then a second bug. Then a third. Here's how a wrong number looks exactly like a right one — and why you spot-check the failures, not the passes.
On an M3 MacBook Air, Qwen3-4B with the thinking trace turned off scores 20/21 on a verifiable suite — beating Llama-3.1-8B's 18/21 at half the memory and nearly double the speed. With thinking on, the same model drops to 7/21. The flag is enable_thinking=False, and here's exactly what it changes and why it matters.
Google's Gemma 4 12B uses 11.4 GB of RAM and runs at 2.7 tok/s on an M3 MacBook Air — 2.4× the memory of Llama-3.1-8B at well under half the speed. Its math and factual answers are flawless; its coding can't be cleanly scored. Here's the honest picture, the multimodal tax, and the benchmark bug I found correcting this post.
StreamingLLM (2023) found that keeping four specific tokens in the KV cache prevents catastrophic perplexity collapse at long contexts. mlx-lm implements this as RotatingKVCache(keep=4). Here's what that means, why it works, and what our measurements on M3 actually show.
Benchmarking Gemma-3-12B, Qwen3-14B, and Llama-3.1-8B on a 16 GB MacBook Air (M3) with MLX. A quantization-aware 3-bit 12B ties a naïve 3-bit 14B on overall answer quality while running faster and in less memory. On a memory-bound Mac, a well-quantized smaller model can match a bigger naïvely-quantized one — so parameter count alone is the wrong thing to shop on.
Honest local-LLM benchmarks on a base M3, 16 GB — tokens/sec, peak RAM, and exactly where it hits the wall. The numbers nobody publishes because they run on H100s.
Debugging why tekken-v13 models emit Ġ instead of spaces through mlx-lm's server, and the one-line root cause in MLX's detokenizer routing.