Running LLMs locally on a Mac

1.1 Runtimes on Apple Silicon

The compute backend is the runtime that runs the matmuls — where on the chip the work actually happens. Three of them cover local text inference on the M-series, and which one a tool picks determines both its speed and how quickly it supports new models.

• PyTorch + MPS (Metal Performance Shaders) is the baseline. Most ML code reaches Apple Silicon through PyTorch, so coverage of new architectures arrives first — it is the lingua franca. My embedding code runs here; speed is acceptable, if not amazing.
• llama.cpp brings its own hand-written Metal kernels rather than going through MPS. It is the engine under Ollama and llama-server, fast and wide-coverage, and the one that consumes GGUF.
• MLX is Apple’s own array framework — faster still on the models it supports, less mainstream, and lagging by months on new architectures. Osaurus, mlx-lm, and the JANG stack all use it.

Not treated here because it is only relevant to image models: CoreML on the Neural Engine for the lowest-footprint path, and Draw Things’ custom Swift + Metal stack.

1.2 Weight formats and quantization

The storage format is what the weights ship in, subject to whatever quantization (if any) has been applied etc. RAM is usually the main constraint on consumer hardware, which is why quantization is helpful. A 70B model at full fp16 is ~140 GB and will not fit on even a 128 GB Mac. A 4-bit build of that same 70B lands near 40 GB, which is tolerable. There are several formats in play.

• safetensors — the Hugging Face baseline, full precision; what PyTorch + MPS loads when the model fits without help.
• GGUF — the llama.cpp format. It is not Apple-specific (it also runs on CUDA and CPU), and it has the widest coverage and the finest quant ladder — down to the very-low-bit IQ2/IQ3 imatrix quants.
• MLX (mlx-community) — the MLX-native format, Apple-only. Also quantized to fit but with added speed on the M-series; coverage tends to lag.
• JANG — mixed-precision extension to MLX: per-tensor bit-widths instead of one width for the whole model.

Generally we prefer an MLX build for the speed when one is published; GGUF when no MLX port exists yet, and classic safetensors when neither is available.

1.3 Mixed-precision MLX

A quantized model does not have to spend the same number of bits on every tensor, and the ones that do are leaving accuracy on the table: attention and routing layers are a small share of the parameters but a large share of the behaviour, so they repay 8 bits where an expert MLP is fine at 4. MLX supports this out of the box. Its quantization config takes per-module overrides — a quantization block in config.json carrying a {bits, group_size} entry per tensor alongside the global default — and as of mid-2026 two toolchains publish calibrated mixed-precision builds into mlx-community using nothing but that.

Both document their calibration-data hygiene. Neither declares a custom format, so anything that loads mlx-lm loads them: mlx-lm itself, mlx-vlm, vllm-mlx, LM Studio, oMLX. Documented calibration plus no new format to support is the whole reason I pull an OptiQ or oQ build first.

The alternative was to declare a new format, and I used to think it would win. JANG — a custom mixed-precision format with its own converter, model zoo and runtime, which gets a section of its own below because it is a whole ecosystem — was my guess for how mixed precision would arrive, at the cost of being fringe. Half right: mixed precision did win, the custom format did not.

3.1 What the session costs on top of the weights

The weights are a fixed cost, and the only one a model card tells us about. The longer a session runs, the more memory we need on top of that: every token in the current context lives in the KV cache, which grows as the conversation does. This is why a model that loads fine can still OOM an hour later, and why so many of the server flags below are about capping context or shrinking the cache.

How much it costs depends on the architecture. A classic dense transformer keeps a key and value vector per layer for every token, so the cache scales with context × layers × width × word length apiece — gigabytes for a long context.¹ kipply’s inference-arithmetic post has a per-token formula; the apxml VRAM calculator looks it up per model, Apple Silicon included. Grouped-query attention already shrinks this, and sparse-attention or SSM-hybrid models (Nemotron-Cascade, DeepSeek V4) change the whole scaling relation to be sub-linear in length.

When the cache is the part that will not fit, we may be able to quantize it, cap the context, or move to one of the cheaper architectures.² Which server exposes which of those is tabulated below.

4.1 Osaurus

Osaurus (MIT, brew install --cask osaurus, osaurus-ai/osaurus) is Swift-native, no Electron, no Python, and behaves like a proper Mac app.

It seems efficient and easy. It also locks me into the Mac ecosystem, so it might not be for everyone. Also, it’s run by one person, so the bus factor is 1, which is a very small number. But — it’s so good!

The window has a model picker, a chat pane, and a status indicator; the inference engine underneath is Apple’s fast MLX, so it gets many tokens per second. It is also the intended runtime for JANG, the same author’s mixed-precision quantization format, which is most of what its curated model catalogue holds — a fact that mostly matters when a model refuses to load somewhere else.

Osaurus is not just a chat client but a full native macOS agent harness. It supports various hip features like persistent memory and sandboxed working folders in an isolated Linux VM via Apple’s Containerization framework. It understands agentskills.io-format skills (and whole Claude plugins) from GitHub or local files, selecting them by RAG at runtime, and speaks MCP in both directions, as server and client. The harness layer is model-agnostic, fronting cloud providers as happily as the local MLX-ish runtime.

There is no CLI download command; the in-app Model Manager (⌘⇧M → Models) browses a curated catalogue of models, especially JANG ones, and will sideload others too — though not all of them work equally well. Nemotron 3 Nano Omni 30B A3B JANGTQ4 seems like a reliable workaday default. Osaurus also discovers anything dropped into its models directory:

hf download mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit \
  --local-dir ~/MLXModels/mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit   # mirror the org/repo path
osaurus list    # confirms discovery, and gives the exact API name for `osaurus run`

Osaurus is also a first-class server, covered below.

4.2 Jan

Jan (brew install --cask jan) is a FOSS cross-platform option — a full frontend + harness + server + runtime bundle. The UI is built on the mildly cursed Electron, but the plus side is that it runs on Linux, Windows, and macOS. It supports both llama.cpp (via Cortex) and MLX backends, so it seems well-suited to brute-force running non-Apple-optimized models.

It looks nice, and the Projects / Assistants / Agents / MCP Connectors quartet gives it a tool-calling agent loop — connect MCP servers under Settings, and Agents mode runs multi-step autonomous workflows (Jan v2 VL is pitched as a 49-step multimodal agent). Jan Server is the self-hosted orchestration variant.

4.3 LM Studio

LM Studio (brew install --cask lm-studio) is closed-source, relatively slick and turnkey, and not free for commercial use. It runs both llama.cpp and lately its own MIT-licensed mlx-engine (mlx-lm + Outlines + mlx-vlm). Like the others, it is a bundle — frontend + runtime + server — with an OpenAI endpoint it can expose headlessly (lms server). I’m mildly sceptical of it because so many Cool LLM Technologies ship special bug fixes or alternate install paths for LM Studio, which hints at a slightly non-standard stack — though that might just be sampling bias, since more people file bug reports when more people run the thing. Also the licence sucks.

5.1 Osaurus as a server

If we already have Osaurus running, we are mostly done: it is already that daemon, exposing OpenAI-, Anthropic-, and Ollama-compatible endpoints on localhost:1337 all at once. Anything we want to point at a local model can talk to it. The reasons to install something else anyway are that this is an idiosyncratic single-author stack, and that it exposes far fewer settings than the Python servers below.

To keep two models resident at once (say the agentic daily-driver plus the maths model), set Settings → Local Inference → Model Management to Flexible — under the default Strict policy, loading one evicts the other.

Context length is automatic — Osaurus picks a sane per-model default and does not expose it as a plain setting, so the actual ceiling is hard to read off (the cheat-sheet has the detail).

To drive all this from the terminal, there is one gotcha: the osaurus command is embedded in the app bundle, and only the Homebrew install links it onto PATH automatically. If it is missing, we symlink it: ln -sf "/Applications/osaurus.app/Contents/Helpers/osaurus" "$(brew --prefix)/bin/osaurus", or use the button in the settings menu that does the same thing.³ From there the CLI does osaurus serve / stop / status / list / run <model> / mcp, plus a plugin manager.

5.2 Ollama

Ollama (brew install ollama) is a llama.cpp wrapper with its own model registry — fast enough, wide model coverage, and notably good for embedding models:

brew services start ollama
ollama pull qwen3.5  # LLM/chats etc
ollama pull mxbai-embed-large    # also handles embeddings

Anything OpenAI-API-compatible can now point at http://localhost:11434/v1. Ollama handles the lifecycle hands-off: it loads a model on the first request, keeps several resident at once (up to OLLAMA_MAX_LOADED_MODELS, default 3), and unloads each after OLLAMA_KEEP_ALIVE of idleness (default 5m). Left unbounded, the pool tanks the machine — OLLAMA_MAX_LOADED_MODELS=1 forces evict-on-switch, OLLAMA_KEEP_ALIVE=0 drops a model the moment it idles (or 15m to keep it warm longer). The context window is num_ctx: set it per request (options.num_ctx), bake it into a Modelfile (PARAMETER num_ctx), or lean on the OLLAMA_CONTEXT_LENGTH default — which on recent Ollama versions auto-scales to VRAM (4k / 32k / 256k) rather than the old fixed 2048.

Gotchas:

• The .gguf files come from Ollama’s registry, not Hugging Face.
• Some weird reimplementation headaches — e.g. the tokenizer baked into the GGUF can differ from the original for unclear reasons.
• The “llama.cpp wrapper” framing is loosening: the registry now ships -mlx tags for some models (e.g. qwen3.5:35b-mlx).

# The OpenAI /v1/embeddings dialect has no num_ctx field, so bake it into the
# model. Overwrite the SAME tag: a client asking for the original name once a
# minute evicts a differently-named derived model on every single request.
printf 'FROM qwen3-embedding:0.6b\nPARAMETER num_ctx 4096\n' > /tmp/qwen3emb.Modelfile
ollama create qwen3-embedding:0.6b -f /tmp/qwen3emb.Modelfile   # -f wants a path, not stdin

# Revert with `ollama pull qwen3-embedding:0.6b`. Weights are content-addressed,
# so neither direction re-downloads anything.

# Confirm: CONTEXT 4096, SIZE ~2.2GB, once something reloads it.
ollama stop qwen3-embedding:0.6b
ollama ps

P=~/Library/LaunchAgents/homebrew.mxcl.ollama.plist

# One model resident at a time.
/usr/libexec/PlistBuddy -c "Add :EnvironmentVariables:OLLAMA_MAX_LOADED_MODELS string 1" "$P"
# Ollama multiplies the context by this before handing it to llama.cpp, so
# letting it autoscale would undo the num_ctx fix above.
/usr/libexec/PlistBuddy -c "Add :EnvironmentVariables:OLLAMA_NUM_PARALLEL string 1" "$P"

# No sudo — gui/501 is our own login session. Two statements, not `&&`: chained,
# the reload arrives before the unload finishes, fails, and leaves Ollama down.
launchctl bootout gui/501/homebrew.mxcl.ollama
launchctl bootstrap gui/501 "$P"

launchctl print gui/501/homebrew.mxcl.ollama | grep -E 'state =|OLLAMA_'

: > /opt/homebrew/var/log/ollama.log

LOG="${OLLAMA_LOG:-/opt/homebrew/var/log/ollama.log}"
MAX_BYTES="${OLLAMA_LOG_MAX_BYTES:-104857600}"   # 100 MiB
KEEP_TAIL_LINES="${OLLAMA_LOG_KEEP_TAIL_LINES:-2000}"

[ -f "$LOG" ] || exit 0
size=$(stat -f%z "$LOG" 2>/dev/null || echo 0)
[ "$size" -gt "$MAX_BYTES" ] || exit 0

# Staged in the log's own directory to stay on one filesystem.
tmp=$(mktemp "$(dirname "$LOG")/.ollama-log-trim.XXXXXX") || exit 0
trap 'rm -f "$tmp"' EXIT

tail -n "$KEEP_TAIL_LINES" "$LOG" > "$tmp" 2>/dev/null || : > "$tmp"
: > "$LOG"
cat "$tmp" >> "$LOG"

OLLAMA_LOG_MAX_BYTES=1048576 sh ~/.local/bin/ollama-log-trim.sh

5.3 mlx-lm and mlx-vlm

mlx-lm is Apple’s reference language-model runtime on MLX. mlx-vlm is its sibling package: same MLX backend, same mlx-community/<repo> weights and HF cache, but for VLMs (“vision-language models”) and omni models—image/video/audio in, text out—instead of pure text LLMs. Where this page says “VLM,” we mean a model in that family; mlx-vlm is what runs one locally, e.g., DeepSeek-OCR. Osaurus and JANG take inspiration from MLX-style Apple-Silicon-friendly execution, but mlx-lm is the original.

uv tool install mlx-lm drops a family of commands onto PATH, all reading the same weights:

• mlx_lm.generate --model mlx-community/<repo> — one-shot completion from the CLI.
• mlx_lm.chat — an interactive REPL in the terminal.
• mlx_lm.server --model mlx-community/<repo> — an OpenAI-compatible daemon; it holds one model, swapping on demand per request (evict + reload, not a restart). Two live at once means two processes on two ports.
• mlx_lm.lora — its LoRA fine-tuning path.

mlx-vlm mirrors this shape (mlx_vlm.generate, mlx_vlm.chat, mlx_vlm.server) but takes an image/video/audio argument alongside the text prompt.

It uses the standard Hugging Face links: mlx-community/<repo> resolves straight to Hugging Face, and the weights land in the shared HF cache (~/.cache/huggingface/hub).

mlx_lm.server has no --ctx flag, so it grows the KV cache to fit whatever we send—up to, presumably, the model’s declared max, capped only by RAM, at which point it presumably kernel-panics the machine. We cap the context in the harness (limit.context / contextWindow) and mind the memory budget.

A reason to keep this around even with Osaurus installed is that it’s great when it runs, but its Swift engine’s coverage lags the Python MLX options. A plain mlx-lm loads interesting MLX conversions that Osaurus can’t (e.g., Cascade-2).

It has also quietly grown the features I left it for. --decode-concurrency and --prompt-concurrency are continuous batching, --prompt-cache-bytes is a prefix cache, --draft-model is speculative decoding, and the sampling defaults are per-launch flags—--temp, --top-p, --top-k, --min-p—which matters because top-k is one of the knobs vllm-mlx accepts and drops. Measured on VibeThinker-3B-8bit: 58 tok/s at \(k=1\), 146 tok/s aggregate at \(k=4\). It also splits thinking into a reasoning field with no parser flag at all, spelled differently from vllm-mlx’s reasoning_content.

5.4 vllm-mlx

vllm-mlx (uv tool install vllm-mlx, Apache 2.0) is a vLLM-style inference server for Apple Silicon. It is active and popular by this page’s standards — 1,300+ stars. The core pitch: continuous batching, paged KV cache with prefix sharing, an SSD-tiered cache for spilling prefixes to disk, and both OpenAI (/v1/*) and Anthropic (/v1/messages) endpoints from one process.

This is the longest section here because I ran it hardest and therefore know its failure modes; it is also the one I eventually swapped out for llama-swap, for reasons downstream of the multi-model registry rather than the engine. The engine-level gotchas are worth reading even if we never touch the registry, since oMLX and Rapid-MLX are both forks and inherit most of them.

It has many fancy bonus features:

• native TTS (Kokoro, Chatterbox, VibeVoice, VoxCPM) and STT alongside text/image/video/audio chat
• multi-token prediction (--enable-mtp)
• an embedding and reranker endpoint in the same process (--embedding-model, --rerank-model)
• MCP tool integration (--mcp-config)
• Prometheus metrics (--enable-metrics) and a built-in benchmarker (vllm-mlx bench-serve)

MTP here is the self-drafting kind: --enable-mtp drives the model’s own MTP head via cache snapshot and restore rather than loading a separate draft model, so the head must live inside the model directory. The acceptance test makes it a poor fit for the fan-out workload I run. The draft token is always sampled greedily regardless of the temperature we request, and it is accepted when the full model’s greedy next token agrees with it — so every accepted position emits a greedy token into a stream we asked to be sampled, eating exactly the diversity maj@\(k\) needs. The comparison is also all-or-nothing across the batch, one list against another, so a single request’s mismatch throws away the drafts of every request sharing the step, and the acceptance rate falls as we widen the fan-out. --mtp-optimistic skips the check entirely and accepts every draft, which trades verification for speed rather than fixing either problem.

Single-model serving is a one-liner:

uv tool install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

For a chat window over this endpoint — one that renders the equations the maths models emit — Open WebUI points at it unmodified: add http://localhost:8000/v1 as an OpenAI connection and skip the bundled-Ollama path the tutorials assume.

vllm-mlx serve ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit --port 8000 --continuous-batching

manager:
  # WEIGHTS only — the KV cache is a separate pool, sized on the serve command.
  # 68 against the 98.6 GB ceiling below, plus --cache-memory-mb *per resident
  # multimodal engine* on top, which nothing here counts. See the note after.
  memory_budget_gb: 68
  contention_policy:
    strategy: wait_then_preempt
    wait_timeout_s: 45
    preempt_after_s: 15

models:
  - name: driver
    path: /Users/dan/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit
    continuous_batching: true
    estimated_memory_gb: 22

  - name: solver
    path: /Users/dan/MLXModels/mlx-community/VibeThinker-3B-8bit
    preload: true
    estimated_memory_gb: 3

vllm-mlx serve --models-config ~/.config/vllm-mlx/models.yaml \
  --port 8000 \
  --gpu-memory-utilization 0.88 \      # ceiling = 0.88 × the wired limit, ≈98.6 GB — not 112
  --continuous-batching \
  --use-paged-cache \
  --cache-memory-mb 30720 \
  --max-cache-blocks 16384 \
  --max-num-seqs 16 \
  --max-tokens 131072 \
  --max-request-tokens 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser auto \
  --reasoning-parser qwen3 \
  --enable-metrics \
  --kv-cache-quantization \
  --kv-cache-quantization-bits 8 \
  --ssd-cache-dir ~/.cache/vllm-mlx/kv \
  --ssd-cache-max-gb 40 \
  --timeout 1200

INFO  Registry memory budget: 68.0 GB of model weights; Metal allocation ceiling
      98.6 GB (88% of 112.0 GB, from serve default); prefix-cache maximum none
      configured

uv tool install --force vllm-mlx

5.5 oMLX

oMLX (jundot/omlx, Apache 2.0, brew tap jundot/omlx https://github.com/jundot/omlx && brew install omlx) is a fork of vllm-mlx with a different frontend grown on top, so it inherits that feature set — continuous batching, multi-model residency, both OpenAI and Anthropic endpoints. Three things set it apart:

• Its SSD prefix cache is automatic and block-addressed — hot blocks in RAM, cold blocks spilled to disk, longest-prefix matched and surviving a restart — where llama-server’s --slot-save-path and ds4’s --kv-disk-dir are manual slot-save knobs. We aim this at agentic coding, where the pitch is TTFT dropping from 30–90s to under 5s on the second turn of a long context.
• An explicit Claude Code accommodation: it rescales reported token counts so auto-compact fires at the right time, and holds the connection open with SSE keep-alives through a long prefill. The frontend is a signed SwiftUI menu-bar app (not Electron) with a web admin panel, and it reuses an existing LM Studio model directory.
• It is the one non-Osaurus server with merged JANG support, so it can load the mixed-precision JANG quants that make sub-4-bit MoE models behave — otherwise Osaurus-only.

omlx serve --model-dir ~/MLXModels \
  --paged-ssd-cache-dir ~/.omlx/cache \
  --hot-cache-max-size 16GB \
  --max-concurrent-requests 8

The same caveats as the rest of the page apply (bus factor 1, MLX-only, benchmarks from an M3 Ultra 512GB), but the clean lineage and the oQ converter make it worth a run as our headless daily-driver if the Osaurus/mlx_lm.server pair leaves us wanting persistent prefix reuse. The SSD cache is less of a differentiator than I first thought: vllm-mlx has the same cold tier behind --ssd-cache-dir, just switched off by default rather than on.

hf download gabfssilva/VibeThinker-3B-MLX-BF16  --local-dir ~/MLXModels/gabfssilva/VibeThinker-3B-MLX-BF16  # solver, hi-fi bf16
hf download mlx-community/VibeThinker-3B-8bit   --local-dir ~/MLXModels/mlx-community/VibeThinker-3B-8bit   # solver, near-lossless and faster
hf download mlx-community/Qwen3.6-35B-A3B-4bit  --local-dir ~/MLXModels/mlx-community/Qwen3.6-35B-A3B-4bit  # the driver

omlx serve --model-dir ~/MLXModels \
  --memory-guard-gb 110 \          # 128 GB Mac: driver (~22 GB) + solver (~3 GB) + KV headroom
  --max-concurrent-requests 16     # the fan-out width; default is 8

5.6 Rapid-MLX

Rapid-MLX (Apache 2.0, brew install rapid-mlx) seems like a variant of interest. The pitch is a polished single-model MLX server — continuous batching, a radix prefix cache, KV quantization down to int4, 17 tool-call parsers, both OpenAI and Anthropic wires, and agent clients wire-verified against real weights every release.

It is a fork of vllm-mlx, (upon installation we find a vllm_mlx module) and what it forked away is the multi-model registry — its model_registry.py is 185 lines (against vllm-mlx’s 1195), because it does not even try to manage multiple models and their memory contention. OTOH it does not seem that vllm-mlx is actually doing a great job when it does try anyway (cf #627 and #712). So maybe it is an alternative with some bugs amputated.

Usefully --reasoning-parser takes vibethinker by name! Also using the flags --kv-disk-checkpoint-interval plus its pflash flags it gets something close to the oMLX’s persistent cache. It is in homebrew-core rather than a personal tap, which is a maturity signal the rest of this page mostly cannot claim. Untested here beyond installing it.

5.7 llama.cpp / llama-server

llama.cpp ships its own server (brew install llama.cpp): llama-server -m model.gguf exposes an OpenAI-compatible endpoint with no daemon, no registry, no opinions.

Ollama wraps this same engine. However, llama-server is more configurable, exposing llama.cpp flags that Ollama does not, notably: YaRN context extension on a GGUF (which Ollama cannot do at all), the finer KV-cache quant ladder, speculative decoding with a draft model, and per-slot KV save/restore to disk.

It loads a .gguf from disk — no ollama pull into a separate store, no background service — which suits scripted or reproducible runs.

It loads the one model named at launch, though a newer router mode (start it with no -m) does dynamic multi-model load and unload. Context is the -c / --ctx-size flag, defaulting to 0 (the model’s full trained window). Other useful config options can be found in the cheat-sheet.

5.8 One process per model, with llama-swap in front

That is every server I have run. The thing they leave us with is a pile of configs and no single source of truth: nine models, three servers, and a different place to write down each model’s temperature.

Worse, every server above that holds several models makes each setting global. One process owning nine models means one temperature for nine sampling recipes, one reasoning parser, one cache size, and a memory budget that then needs an eviction policy and a contention strategy to go with it. Both bugs I filed against vllm-mlx are that shape — a process reasoning about an aggregate nobody can measure. I ran it that way for months before working out that it was the source of most of my grief.

The alternative is to invert it. llama-swap (Go, one binary, brew install) reads the model field of an incoming request, starts the right server if it is not up, proxies, and stops it on a TTL. It contains no inference code, so it is a much smaller thing to trust. Underneath, each model is its own process — mlx_lm.server here, but a cmd: is an arbitrary command line, so vllm-mlx serve, omlx serve and ./ds4-server can all live behind the one port too.

models:
  "vibethinker-8bit":
    cmd: |
      mlx_lm.server --port ${PORT}
      --model ~/MLXModels/mlx-community/VibeThinker-3B-8bit
      --temp 1.0 --top-p 0.95 --top-k 0 --decode-concurrency 16
    ttl: 900

What this buys, in rough order of how much it improved my life:

• Per-model settings stop being compromises. The card’s recipe goes on that model’s launch line. There is no server-wide temperature to be wrong about half the registry.
• Memory becomes measurable. One process per model means footprint -p <pid> reports that model’s Metal allocation under IOAccelerator (graphics) — 4.7 GB of weights on disk shows up as 5.2 GB RSS and 5510 MB of footprint. Unified memory means MLX buffers are ordinary physical pages, so nothing is hidden. The old setup needed a hand-maintained estimated_memory_gb per entry precisely because nine models in one address space made per-model memory unmeasurable.
• Swapping is cheap enough to ignore. Cold starts on my machine: 2.4 s for a 3 GB model, 7.7 s for a 20 GB one.
• Co-residency is five lines. A groups block keeps the small utilities loaded together and lets the heavyweights take turns — enough to hold a driver and a solver up at once. No budget, no LRU tuning.

Name the models after the weights, not the job — half the time which build answered is the question. Roles then layer on top, and the useful one is a warm selector: driver resolves to whichever driver is already loaded and only cold-starts when none are, where a plain alias would force a 20 GB swap because the other driver happened to be up. setParamsByID gives one resident process two names and two recipes (qwen3.6-35b at temp 0.6 for coding, qwen3.6-35b:thinking at 1.0), and profiles repoint a default at runtime via PUT /api/profiles/active with {"name": "..."} — not profile, not id, both of which fail claiming the name is required. The same setParams / stripParams filters rewrite requests on the way through, which is where we drop the parameters vllm-mlx accepts and ignores and rename Open WebUI’s repeat_penalty.

A peers block folds another whole server in behind the same address — Osaurus on :1337 becomes osaurus/<model> on the same port, which is worth it for the JANG builds nothing else loads. It is federation, not scheduling: llama-swap forwards the request and nothing else, so it cannot start the peer, cannot stop it, and does not count its memory against the groups. Only list models the peer alone can serve — a peer entry duplicating a model llama-swap already owns is a second path to a second resident copy that neither side accounts for. And only list chat models, per the warning above.

What I gave up is the SSD prefix-cache tier: mlx_lm.server caches prefixes in memory only, so a long agentic second turn re-prefills from scratch. A swap presumably discards the in-memory prefix cache outright, which makes that loss worse rather than better. It is a real cost and the reason I keep watching Rapid-MLX, whose --kv-disk-checkpoint-interval is the nearest replacement. One further macOS wrinkle: llama-swap’s docs suggest running Python servers under Docker for clean SIGTERM handling, which is not available to us, since containers on macOS cannot use the Metal backend.

6.1 Sampling defaults

Sampling — temperature, top-p, top-k, and the output-token budget — is a decode-time choice set on each request. A server does not, in general, pick up the model’s recommended sampling settings for us.

mlx_lm.server, for example, defaults to temperature=0.0 (greedy), top_p=1.0, top_k=0, and max_tokens=512, and it does not read the model’s generation_config.json. So a reasoning model’s advised settings — VibeThinker advises temperature 1.0 / top-p 0.95 and a 64K-plus output budget — need the client to specify them; each model table below carries a Recommended sampling column with the per-model picks. Notably temperature=0.0 breaks maj@k voting, since every sample comes back identical.

6.2 Which sampling parameters actually work

vllm-mlx 0.4.1 accepts parameters that it ignores. On the text path the sampler is built as make_sampler(temp, top_p, min_p), and logits processors are constructed only from repetition_penalty. So:

The dead ones are logged on the way in and dropped on the way through. So any top-k 20 in a model card’s recipe — and most of the recipes in the tables below say exactly that — does nothing for a text model, while the same request against a VLM or omni build is honoured, which is a difference to remember before concluding a knob works. Two more traps in the same family:

• mlx-lm’s repetition_context_size defaults to 20 tokens and vllm-mlx never overrides it, so the penalty sees a 20-token window. It will break a short stutter but will not touch a repeated paragraph.
• vllm-mlx reads a model’s generation_config.json for stop tokens only, so a card’s recommended temperature never reaches the server on its own.

When the registry has no per-model sampling — vllm-mlx’s does not — a server-wide default is the only lever, and that makes it a compromise rather than a setting. Mine splits: 0.6 for the agentic models, 1.0 for Cascade-2 and VibeThinker, so any single temperature is wrong for half of them, and the 0.7 fallback is a defensible midpoint. min_p and repetition_penalty are the two that are safe to set globally, because they truncate the degenerate tail without pinning a preferred temperature:

--default-min-p 0.02 --default-repetition-penalty 1.05

Where to pin values depends on which layer issues the prompt.

• mlx_lm.generate — flags per call: --temp 1.0 --top-p 0.95 --max-tokens 40000 (--top-k already defaults to 0). Wrap it in an alias.
• mlx_lm.server — launch-time defaults via --temp / --top-p / --top-k / --min-p, overridden per request; Osaurus keeps the same defaults in its app settings, also overridden per request.
• transformers — a GenerationConfig passed at generate() time.
• Ollama — an exception, baking in a per-model default through a Modelfile: PARAMETER temperature 1.0, PARAMETER top_p 0.95, PARAMETER num_predict 40000 for the output budget, PARAMETER num_ctx 65536 for the context window.

Output budget and context window are separate limits: the first caps how much the model may emit, the second how much prompt-plus-output the KV cache holds. A long-reasoning model can need both raised, or it truncates mid-derivation — and Ollama in particular drops the overflow silently once num_ctx is exceeded.

6.3 What the client is able to send

The server-side story above assumes we can send a value with a prompt, and which ones we can vary with the client — Goose sends temperature and nothing else, OpenCode sends arbitrary keys past one badly-named flag, Open WebUI pins a full set per model preset, Jan does too but only on a custom provider. That table lives with the frontends, since it is the same table whatever the server.

What makes it bite here is that vllm-mlx’s registry has no per-model sampling of its own, so there is no server-side fallback: whatever the client cannot send is a setting we simply do not have. That is why Open WebUI’s presets matter disproportionately on this stack — a preset per model is the only place the per-model temperatures in the tables below can actually be written down.

Gotcha: Open WebUI’s Advanced Params panel names the penalty repeat_penalty — Ollama’s spelling — while vllm-mlx reads repetition_penalty, so it is dropped in transit and the model keeps looping. The panel suffixes eight Ollama-only fields with (Ollama) — num_ctx, keep_alive, think and friends — but repeat_penalty is not among them, so it looks portable and is not. The fix is the Add Custom Parameter button at the bottom of the panel: add repetition_penalty by hand and skip the built-in row.

6.4 Prefix caching

The KV cache is the only state these servers keep between requests — there is no session object; we resend the whole history each call. What they reuse is the prefix: the shared start of the conversation (system prompt, tool definitions, history so far) keeps its KV, so an agentic loop prefills only the new suffix. Matching is content-addressed — on the tokens, not a session ID — so it happens by itself, and there is nothing to switch on.

The consequence worth planning around is that the match is on a prefix, so anything that edits the front of the history throws the whole thing away. A harness that trims old turns to fit the context window, or rewrites the system prompt between calls, re-prefills from scratch every time and we pay for it in time-to-first-token without any error to tell us why. Appending is free; editing is not.

Different servers hold this cache for different lengths of time:

• mlx_lm.server — an in-memory LRU, longest-prefix matched, reporting cached_tokens in the usage block; bounded by --prompt-cache-size / --prompt-cache-bytes, held for the process.
• llama-server — --cache-prompt is on by default (one KV per slot; --cache-reuse even salvages chunks after a mid-prompt edit), and slots can be saved to disk.
• Ollama — the same llama.cpp reuse, alive as long as the model stays loaded (OLLAMA_KEEP_ALIVE).
• Osaurus — automatic; headless under osaurus serve the cache lives for the server process (governed by the Strict/Flexible policy), and in the GUI it is per chat window, warmed the moment one opens.

Prefill is chunked and continuously batched besides, so concurrent requests interleave rather than queue — but the prefix skip is the bigger win.

6.5 Speculative decoding and MTP

Emitting one token at a time is bound by memory bandwidth rather than arithmetic: each token drags the active parameters across from RAM, and the compute units spend most of that time waiting for them. Speculative decoding spends that idle arithmetic. Something cheap drafts the next few tokens, the full model scores the whole draft in a single forward pass — the pass that would otherwise have produced one token — and every drafted token it agrees with is kept. A miss costs the pass we were going to pay for anyway, so the arrangement is roughly free when it fails and worth several tokens per pass when it works. Done properly the output distribution is unchanged and only the speed differs, which is the entire appeal, and which is also the part the vllm-mlx implementation gives up.

The draft has to come from somewhere, and there are two answers.

Draft model

A second, much smaller model does the drafting — say a 0.5B proposing for a 30B — sharing a tokenizer with the big one. This is what --draft-model means in mlx_lm.server and llama-server. The cost is a second set of weights resident in RAM, plus finding a small model that pairs with the large one.

MTP head, multi-token prediction

The model drafts for itself. A checkpoint trained with multi-token prediction (Gloeckle et al. 2024) — Qwen3.6 and Qwen3.8 here, the DeepSeek lineage before them — carries a small extra head that predicts a position or two past the ordinary next-token head. Nothing extra loads, because the head ships in the model directory with the rest of the weights. This is what --enable-mtp means in vllm-mlx.

The verification step is the same either way, so a server’s rule for what happens when the draft and the full model disagree applies to both.

6.6 Stretching the context window

Deep lore for the optimizers.

A model’s positional encoding is trained out to some fixed length, and that trained length is baked into its knowledge of the context window. Some of the huge context numbers advertised are that same window stretched at load time, rather than a property of the weights. Qwen3.6, for example, trains its rotary positions to 256K (262,144 tokens); the 1M figure quoted for it is that same window extended ~4×, and getting the extension costs us something in both quality and RAM.

The mechanism is RoPE interpolation. RoPE encodes each token’s position as a rotation; interpolation rescales those rotations so a position past the trained window maps back into the range the model saw during training, instead of falling off the end into rotations it has never seen. The common variant is YaRN (“yet another RoPE extension”), which rescales per frequency rather than uniformly. A factor of 4.0 takes Qwen3.6’s 256K to ~1M, for example.

Every mainstream implementation I’ve seen applies the rescaling statically, fixing it at load and applying it to every prompt regardless of length. A model loaded with factor 4.0 rescales a 2K-token prompt exactly as hard as a 900K one, and short prompts lose some accuracy for a long window they aren’t using. So we switch YaRN on only when we want the long window, and set factor to the longest context we actually expect rather than the largest the model will accept.

How to turn it on depends on the runtime.

transformers and mlx-lm read a rope_scaling block straight from the config.json that ships in the model’s own directory:

"rope_scaling": {
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}

• llama.cpp / llama-server take flags: --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 262144, alongside the usual -c.
• Ollama exposes no YaRN settings. We inherit whatever the person who converted the GGUF baked into its metadata.
• Osaurus’s Swift engine has the YaRN code, but its Qwen3 and Llama wrappers do not route through it, so YaRN is unavailable for now.

YaRN increases the maximum context window, but does not shrink the KV cache cost of storing that context, so we remain RAM-constrained.

6.7 Configuring each server

All the settings in one place.

Flash attention does not get its own column because it is no longer a per-server decision: llama.cpp defaults -fa to auto and turns it on wherever Metal supports it; Ollama switches it on per architecture for the families here (Qwen3.x, Nemotron, Gemma, gpt-oss); and the MLX servers plus ds4 always run a fused attention kernel. The one place it still needs a hand is Ollama’s KV-cache quant, which only takes effect with OLLAMA_FLASH_ATTENTION=1 set alongside it.

Sensible headless starting points:

# 64K context (below): a useful bound well under the trained max — raise or lower for your RAM
# llama.cpp — flash attention is automatic; quantize the cache, choose a context
llama-server -m model.gguf -c 65536 --cache-type-k q8_0 --cache-type-v q8_0

# Ollama — env vars; cache quant needs flash attention enabled
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# mlx-lm — no cache-quant/context flag; set the model’s sampling values, cap context in the harness
mlx_lm.server --model mlx-community/<repo> --temp 0.6 --top-p 0.95 --top-k 20

# oMLX — point it at the shared MLX tree; turn on the SSD cold tier and a memory ceiling
omlx serve --model-dir ~/MLXModels --paged-ssd-cache-dir ~/.omlx/cache --memory-guard-gb 96

The --paged-ssd-cache-dir on that last line persists across restarts and gets rewritten every session, so it is the one path here that we want to keep out of backups and Spotlight.

Osaurus is tuned in its Settings pane rather than via a command line, and ds4’s full flag set is in its own section.

8.1 For mathematical reasoning

The Role and Target columns are the reasoning notes’ taxonomy — the role is which slot in a harness the model fills, the target is what it emits, and that determines which checker we owe it. A generalist does maths alongside chat and tool-use, takes whichever target we ask it for, and can be driven by an ordinary agent. A solver emits a boxed final answer, trading chat fluency for reasoning, and wants a specialized loop — tool-integrated reasoning where the model runs code, maj@k voting where it does not. An informal prover emits a proof in natural language, which only a judge can grade. A prover emits Lean, which a compiler accepts or rejects.

The solver rows above are pure CoT, innocent of tool use. They can fit into a larger workflow still, as a solve() oracle a general agent dispatches to. In this case we would serve it on a batching endpoint that keeps it resident beside the agentic driver, and let the driver call it when it needs some help on a sub-problem.

So the Target column is a procurement decision, not a label. maj@k voting pays off on a solver and buys nothing on either proof target, because a set of arguments has no mode to take — download Goedel and we are also committing to a Lean toolchain; download nomos-1 or QED-Nano and we are committing to a judge instead. Which is why nomos-1 ships its own harness and why the two are not interchangeable on this page, however similar their file sizes look.

VibeThinker is a case where the harness matters a lot. Its Claim-Level Reliability Assessment lifts AIME26 from 94.3 to 97.1 off the same 3 GB checkpoint (Xu et al. 2026).

8.2 For agentic flows

These are the weights a coding harness needs. Sizes here are for the specific MLX build named in the Run via column, since a 4-bit and an 8-bit of the same model are different procurement decisions.

8.3 Diffusion models

Everything above is autoregressive. Diffusion LLMs are a thing though. Do any run locally? I know of one: DiffusionGemma — Google’s experimental Gemma 4 variant, which denoises a whole 256-token “canvas” in parallel instead of emitting tokens left to right. OsaurusAI/diffusiongemma-26B-A4B-it-MXFP8 runs natively in Osaurus through its vmlx-swift block-diffusion engine, ~26 GB on disk and ~24 GB resident.

Manage expectations on speed. Docs report 28–42 tok/s at 48 denoising steps: Osaurus defaults to 16, roughly twice as fast as the bundle default and still coherent, and the output falls apart below 12. Quality trails plain Gemma 4 as well. Vision, tool-calling, and a reasoning channel all work in this checkpoint; audio and video do not.

Beyond speed, there is a family-level reason I have not chased Gemma further, and it is why no Gemma appears in the tables above. In my brief tests Gemma models come across as brittle and anxious — quick to hedge, and prone to a sort of performed distress when pushed. Soligo et al. report the same at scale, and locate it in post-training rather than the base weights: instruct-tuned Gemma expresses substantially more distress than its own base model does, whereas instruct-tuned Qwen and OLMo express less (Soligo, Mikulik, and Saunders 2026). They also take high-frustration responses from 35% to 0.3% with DPO on 280 preference pairs at no capability cost, which reads as a shallow trait rather than a baked-in one — just not one fixed in any checkpoint I can download.

So it is not the agentic daily-driver. I am interested in the interaction model though: bi-directional attention over the canvas makes it potentially useful for infilling and structure-preserving rewrite, which is a different way to drive a coding tool than streaming tokens into a chat box.

10.1 One developer, four products

The same person wrote every piece above — Jinho “Eric” Jang (Irvine, California), who is also Osaurus’s lead and only engineer, and the author of the parallel desktop app MLX Studio. Runtime, quant format, model zoo, two GUIs, one developer. That vertical integration buys fast iteration and a coherent feature set across the chain: new model architectures land in JANG within days of release, faster than the bigger stacks manage, and at the high end of Apple Silicon he is doing things nobody else is doing.

The downside is that if Jang loses interest, switches jobs, or gets hit by a bus, the lot becomes abandonware — including any JANG-format weights we have on disk, which no other converter emits. There is some community wariness about this; see the r/LocalLLaMA “Is MLX Studio legit?” thread. It is all open source, so in principle we could maintain it ourselves if he walks away.

10.2 MLX Studio

MLX Studio is the JANG/Osaurus author’s other Mac desktop app — Electron + Python rather than Swift, broader feature surface (image generation via Flux and Z-Image, ~26 built-in agentic tools, in-app GGUF→MLX and MLX→JANG conversion, an Anthropic-compatible API). Install via the signed DMG on the releases page, or engine-only with uv tool install vmlx and vmlx serve mlx-community/<repo> (OpenAI-compatible on localhost:8000).

11.1 Running DwarfStar via the pi stack

The default harness for ds4 seems to be: pi, an MIT-licensed agent harness by Mario Zechner (badlogic, of libGDX fame) — itself a strong offline coding agent once a model is behind it. There is an easy install via the pi extension by Armin Ronacher (mitsuhiko, of Flask): mitsuhiko/pi-ds4. It handles process management for ds4-server — per-PID leases, watchdog shutdown, OpenAI-compatible local endpoint on 127.0.0.1:8000:

pi install https://github.com/mitsuhiko/pi-ds4

A first-time install clones antirez/ds4, builds it, downloads the GGUF (~87 GB), and registers a ds4/deepseek-v4-flash model with pi. Subsequent runs spawn the server on demand and shut it down when no client process holds a lease. OpenClaw embeds pi, so the same extension can in principle load there.

Running pi from the terminal opens a TUI (“textual user interface” — I believe that’s what it means, as it lives in the terminal).

Audrey Tang maintains audreyt/pi-ds4, a fork that swaps in cyberneurova’s abliterated IQ2XXS quants and turns on uncertainty-mode directional steering by default — an activation-space edit that puts the model into “this is a contested question” mode on CCP-sensitive topics (Taiwan, Crimea, Kashmir, Western Sahara).

11.2 Manual setup for non-pi harnesses

Outside the pi ecosystem, the manual setup requires four commands plus a config edit.

# antirez/ds4 for upstream; also audreyt/ds4 looks cool
# optimizations + steering-vector work — pick one
git clone https://github.com/audreyt/ds4
cd ds4
make
tmutil addexclusion -p (realpath ./gguf)
./download_model.sh                     # ~87 GB into ./gguf/
./ds4-server                            # listens on 127.0.0.1:8000

For lifecycle, we could wrap ./ds4-server in a launchd plist with KeepAlive: true; this probably isn’t what we want on a typical laptop, where we do other things besides inference — like, you know, use it as a laptop. I think pi is more automatic in that regard.

ds4-server’s context window is set at launch via --ctx <tokens> (max accepted per conversation); output length is a per-request API field, not a launch flag. --kv-disk-dir <path> (with --kv-disk-space-mb <n>) persists the KV cache to disk, so a prefix survives restarts and session switches rather than being reprocessed — durable prefix storage, not a long-context spill. Thinking mode is on by default, toggled per request, running DeepSeek’s reasoning mode. DeepSeek V4 Flash nominally supports 1M tokens, but ds4 is RAM-bound: the 2-bit IQ2XXS weights are ~81 GB, and a full 1M-token KV/index sits around 26 GB on top. Rough budget on unified memory:

• 64 GB: 50k–150k --ctx with headroom.
• 96 GB: 150k–250k works but is tight; quit Slack.
• 128 GB: 200k–300k is comfortable; >300k starts risking OOM.
• 1M: only with very generous memory and nothing else running.

If a client (Hermes, OpenClaw, OpenCode, anything OpenAI-compatible) advertises a context larger than --ctx, requests will get cut off — match the client’s contextWindow / limit.context to the server’s --ctx. DeepSeek’s sparse attention means raising --ctx doesn’t blow up compute the way dense attention would, but RAM is still a constraint. For most interactive coding, 32k–100k plus a retrieval layer beats brute-forcing the whole history into the prompt. See antirez/ds4’s README and the OpenClaw ds4 provider docs for the full flag list and client-side config.

Reasonable defaults:

./ds4-server --ctx 200000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

audreyt’s fork uses environment variables for directional steering rather than flags, so it composes with the above:

DS4_DIR_STEERING_FFN=-0.75 \
DS4_DIR_STEERING_ATTN=0 \
DS4_REPRODUCIBLE=1 \
./ds4-server

To use with Hermes, add an OpenAI-compatible provider entry to the Hermes config (sketch — confirm the exact schema with hermes config):

# ~/.hermes/config.yaml
custom_providers:
- name: ds4
  base_url: http://127.0.0.1:8000/v1
  model: deepseek-v4-flash
  models:
    deepseek-v4-flash:
      context_length: 200000

From inside Hermes, /model ds4/deepseek-v4-flash — matching the provider name in the YAML. Done.

Anyway, this provides a generic token endpoint, so we’re free to plug in whatever we like on the front end.

The protagonists of this play have a lot of clout — antirez (Redis), mitsuhiko (Flask), badlogic (libGDX), and audreyt (Taiwan’s former Digital Minister, Pugs / Perl 6). Some kind of critical mass seems feasible for a certain type of nerd.

12.1 When a model can’t spell a space

Text from mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit came back like this:

DidĠyouĠmeanĠ'chubs'ĠasĠinĠtheĠtongueĠtwister?

Ġ (U+0120) and Ċ (U+010A) are how byte-level BPE spells a space and a newline in the vocabulary, so seeing them raw means the round-trip was not in fact round. The fault seems to be at encode time, where "hello wide world" tokenizes to ['h', 'ellow', 'id', 'eworld'], i.e. spaces gone before BPE runs. The DeepSeek-R1-Distill-Qwen tokenizer omits the ByteLevel post_processor that closes the loop — which worked with tokenizers<0.22.2 — so that whole generation of Qwen distills shares that problem with its upstream progenitor checkpoint.

That lineage is old enough by now that I won’t bother fixing this particular bug to reactivate it. What I will do is remember this failure mode, because it sneaks up. The model loads, serves, streams, reports 200 OK and answers on-topic; it is only that small matter of being unreadable. I had a small preloaded utility model sitting wasting RAM for weeks unnoticed, just casually gaslighting the other LLMs that invoked it. Open WebUI finally surfaced it. So: eyeball the output of anything we leave running, especially the small utility models nobody looks at.

12.2 Two stacks, two tokenizers, one model

I came to understand the transformers/llama.cpp split by breaking it.

The Hugging Face and Ollama versions of mxbai-embed-large are nominally the same model — same upstream weights — but each stack implemented its own tokenizer. On plain prose the two mostly agree, I think; on markdown they can disagree by a few percent on how many tokens a chunk takes. That is survivable for chat, where tokenization stays internal to one stack and we can see the output, but it is not survivable for embeddings, where a few percent of drift is a silently different vector and nothing downstream can tell. So for embeddings on this blog I went all-transformers, and for chat through a live server Ollama is fine.