2 posts tagged “gpu”
KV Cache Quantization Is 4× Slower on My Mac and 28% Faster on a Rented L4
The same idea — store attention keys and values in fewer bits — measured on two stacks: MLX's --kv-bits on an M3 raises RAM and slows decode 4×, while vLLM's fp8 KV cache on an L4 doubles capacity and speeds decode up to 28%. KV quantization is a property of the implementation and hardware, not the technique.
Running Llama 3.1 8B with FP8 on vLLM Cuts Cost from $1.00 to $0.36 per Million Output Tokens
A measured audit of Llama-3.1-8B on a rented L4: what vLLM's real, untouched default actually costs, what FP8 quantization gets you on top of it, and why a plausible-looking concurrency guess cost throughput instead of adding it. Every number here is measured, including a second, larger benchmark pass that fixed a real gap the first one had.