oMLX Review: SSD KV Cache Fixes Local Coding Agents

Key highlights:

• 19,977 GitHub stars, 1,701 forks, ~1,100 stars gained this week

• Apache 2.0, Python, created 2026-02-13 — roughly six months old

• Latest release: v0.6.3rc1 (2026-08-19) — very active, ~984 open issues

• Tiered KV cache: hot in-memory + cold SSD blocks in safetensors format, survives restarts

• Continuous batching via mlx-lm's `BatchGenerator` (default 8 concurrent requests)

Local inference on Mac has had a good year. We covered [Ollama's switch to the MLX backend](https://andrew.ooo/posts/ollama-mlx-apple-silicon-review/) (1.6–2x faster) and [Rapid-MLX](https://andrew.ooo/posts/rapid-mlx-fastest-apple-silicon-llm-server/) (2–4x faster than Ollama on raw throughput). Both of those posts were about the same metric: tokens per second.

oMLX is interesting because it attacks a metric nobody was optimizing: time to first token in a long, mutating conversation.

Here's the thing raw benchmarks hide. When you benchmark a local model you send one prompt and measure decode speed. Impressive numbers. But a coding agent doesn't work like that. It sends dozens of requests where the system prompt, tool definitions, and file contents keep shifting in the middle of the context — a file gets edited, a tool result gets inserted, the conversation gets compacted. Standard prefix caching only helps when the entire prefix matches exactly. Change one token 40k tokens in, and every server on the market recomputes the whole thing.

oMLX started from vllm-mlx v0.1.0 as its basic serving layer. Everything above that — SSD tiering, continuous batching, VLM support, the Anthropic API surface, the native macOS app — is original work, per the author.

The architecture is worth reading in full because it explains the design:

├── EnginePool (multi-model, LRU eviction, TTL, manual load/unload)

This is the single most important practical detail in the project, and it deserves its own section because it will silently cost you 30x performance.

A plain pip install -e . does not build oMLX's native Metal custom kernels. Affected model families — GLM-5.2, MiniMax M3, Qwen3.5 — then fall back to much slower generic paths, and use more memory doing it. The README quantifies it: for GLM-5.2 the fused DSA prefill is roughly 30x faster with the kernels — a measured 845 tok/s versus ~29 tok/s on an M3 Ultra.

Building them requires the Metal toolchain, which Command Line Tools alone do not provide (you'll hit xcrun: error: unable to find utility "metal"). You need full Xcode:

This is a condensed version. The full article includes architecture diagrams, code examples, comparison tables, and practical use cases.

👉 Read the complete analysis on andrew.ooo

Topics: omlx, mlx, apple-silicon, local-llm, kv-cache