10 posts tagged “inference”
My readability metric found pockets in a sealed scroll. They were shredded.
I tried to turn the best-scoring pockets from my PHerc1203 readability atlas into a readable surface. They were torn. An opposite coherence-first search found a different failure mode: the seven successfully inspected candidates were densely packed. Together, the searches motivate a joint coherence-and-separability test.
I mapped local readability cues inside a sealed Herculaneum scroll
A CPU-only survey of PHerc1203 found that most sampled sites fail a local geometry gate, while a few rare pockets initially looked comparable to known-readable PHerc1667. Follow-up raw-CT inspection showed that the top pockets were torn fragments, clarifying what the metric can and cannot detect.
Why a 2.4 µm Herculaneum scan still reads as noise — and what actually blocks it
I ran the public ink model on a 2.4 µm scan of sealed PHerc1203 and got speckle instead of letters. Chasing why turned up a mistake in my own measurement, corrected it against the community's winding atlas, and left me with something more useful than a failed read: a CPU-only gate that flags rendered windows unlikely to isolate one papyrus sheet — before you spend the GPU.
KV Cache Quantization Is 4× Slower on My Mac and 28% Faster on a Rented L4
The same idea — store attention keys and values in fewer bits — measured on two stacks: MLX's --kv-bits on an M3 raises RAM and slows decode 4×, while vLLM's fp8 KV cache on an L4 doubles capacity and speeds decode up to 28%. KV quantization is a property of the implementation and hardware, not the technique.
Running Llama 3.1 8B with FP8 on vLLM Cuts Cost from $1.00 to $0.36 per Million Output Tokens
A measured audit of Llama-3.1-8B on a rented L4: what vLLM's real, untouched default actually costs, what FP8 quantization gets you on top of it, and why a plausible-looking concurrency guess cost throughput instead of adding it. Every number here is measured, including a second, larger benchmark pass that fixed a real gap the first one had.
Carmack's right about the weights. The KV cache is the part his argument skips.
John Carmack argued AI inference should stream model weights from cheap flash instead of expensive HBM, since weight access is deterministic. That's correct for weights — but the KV cache grows, gets rewritten every token, and reads a shifting range. I measured what happens when you force it to behave like a fixed resource anyway.
A rotating KV cache saves 36% of your memory and 100% of your recall
Capping a model's KV cache instead of letting it grow without bound plateaus peak memory — but the instant a conversation outgrows the cap, recall of five planted facts collapses from 5/5 to 0/5, every trial, with zero partial credit. The attention-sink trick doesn't help, and here's why.
At 32,000 tokens, the costliest thing my MacBook did was wait seven minutes to speak
I ran the same long-context test on a 16 GB fanless M3 and a ₹23 rented NVIDIA L4. The laptop fits a 32k context on an 8B model and keeps every planted fact — but prefill balloons to seven minutes and its decode speed can't even be measured, because the fanless chip throttles. A measured, cross-hardware look at the KV-cache tax.
I built self-speculative decoding for MLX. On an M3, naive layer-skip never beats baseline — 24 configs, 24 losses
Self-speculative decoding lets a model draft its own tokens by skipping layers — speculative decoding's speedup with no extra memory. I built it for MLX and swept 24 configs on an M3. Every one was slower than baseline, even though all were lossless. Here's why, and the paper that fixes it.
Speculative decoding on a 16 GB Mac: a 20% win that becomes a 25% loss
A 1B draft model speeds up Llama-3.1-8B by 20% on an M3 — at num_draft_tokens=2. Push that dial to 4 and decoding gets 25% SLOWER than using no draft at all. Here's the measured curve, and why low draft counts win when decode is bound by memory bandwidth.