7 posts tagged “quantization”
KV Cache Quantization Is 4× Slower on My Mac and 28% Faster on a Rented L4
The same idea — store attention keys and values in fewer bits — measured on two stacks: MLX's --kv-bits on an M3 raises RAM and slows decode 4×, while vLLM's fp8 KV cache on an L4 doubles capacity and speeds decode up to 28%. KV quantization is a property of the implementation and hardware, not the technique.
Running Llama 3.1 8B with FP8 on vLLM Cuts Cost from $1.00 to $0.36 per Million Output Tokens
A measured audit of Llama-3.1-8B on a rented L4: what vLLM's real, untouched default actually costs, what FP8 quantization gets you on top of it, and why a plausible-looking concurrency guess cost throughput instead of adding it. Every number here is measured, including a second, larger benchmark pass that fixed a real gap the first one had.
A rotating KV cache saves 36% of your memory and 100% of your recall
Capping a model's KV cache instead of letting it grow without bound plateaus peak memory — but the instant a conversation outgrows the cap, recall of five planted facts collapses from 5/5 to 0/5, every trial, with zero partial credit. The attention-sink trick doesn't help, and here's why.
I turned on MLX's memory-saving flag and ran out of memory
On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.
Gemma 4 on a 16 GB Mac: the E4B matches the 12B at 42% less RAM and 3× the speed
Google's Gemma-4 E4B posts the same math and factual scores as the full 12B on an M3 MacBook Air — in 6.6 GB instead of 11.4, at 8.2 tok/s instead of 2.7 — so on a 16 GB Mac the E4B is the one to run. This is a size win, not a QAT one: the 12B's own QAT build doesn't shrink or speed it up. Honest numbers, measured under a real 2048-token load.
Gemma-3-12B QAT vs Qwen3-14B 3-bit: same quality on a 16 GB Mac, but the smaller model runs lighter and faster
Benchmarking Gemma-3-12B, Qwen3-14B, and Llama-3.1-8B on a 16 GB MacBook Air (M3) with MLX. A quantization-aware 3-bit 12B ties a naïve 3-bit 14B on overall answer quality while running faster and in less memory. On a memory-bound Mac, a well-quantized smaller model can match a bigger naïvely-quantized one — so parameter count alone is the wrong thing to shop on.
What actually runs well on a 16 GB MacBook
Honest local-LLM benchmarks on a base M3, 16 GB — tokens/sec, peak RAM, and exactly where it hits the wall. The numbers nobody publishes because they run on H100s.