Quantized KV Cache Helps Six of My Nine Models and Actively Hurts the Two Newest Ones
Nine models on one pinned build, three cache types each. Where quantizing the KV cache pays, where it costs, and what runs at all on 128GB.
The Frontier Lab
Engineering's view on local frontier AI — what actually runs on hardware you control.
Latest posts
I Rebuilt the Box, and the Cliff Was Gone. Then I Spent an Evening Proving Which Patch Did It
A Fedora upgrade turned into an attribution run. Five builds from one base, and the one patch that removes the dense flash attention collapse at depth.
A Reader Fixed Quantized KV Cache on Strix Halo. I Verified It at 262k Context, and His Theory About My Dense Cliff Survived Half Its Test
A reader's fork makes quantized KV cache faster than f16 on Strix Halo. Verified on a 128GB box up to the full 262k native context. His cache-spill theory for the dense cliff survived half its test, and the failed half is the interesting part.
Turning Off the IOMMU Made My Dense Model 37% Faster. Five More Strix Halo Tips, Measured, and One Crashed the GPU
Two guides said turn the IOMMU off. I was the outlier running it on. Measured: +34 to +38% dense prompt processing. Plus five more tips, mostly busted, and one that crashes the GPU.
I Measured the Strix Halo Tuning Tips. One Flag Matters More Than All of Them
Same box, same build, one flag: +41% for the MoE model, -51% for the dense one. The Strix Halo tuning tips, measured and ranked, including the ones that failed.
One llama.cpp Update Made My MoE Model 38% Faster and My Dense Model 45% Slower
Same box, same models, same flags, three builds in three weeks. What moved, what broke, and the routine that keeps my numbers honest.
The Strix Halo Reference Setup Pack
A proven production setup you can copy: BIOS values, kernel params, working systemd units, and real benchmark numbers.
Buy or Wait: Reading the Local LLM Hardware Question in a Memory Crunch
The instinct is to wait for the next box. In a memory crunch, that's backwards. How to read the buy-or-wait question for Strix Halo, DGX Spark, and Mac Studio.
What to Buy for Local LLM Inference: Strix Halo, Mac Studio, DGX Spark, or a GPU Rig
A buyer's guide to local LLM hardware, ranked by the one spec that actually decides generation speed: memory bandwidth. Plus where ROCm really stands on Strix Halo.
AMD Is Selling "First-Class ROCm" on Strix Halo. I've Run the Same Chip for Six Months.
On June 8, AMD opened pre-orders for a $3,999 box built around the exact chip I've run in production since the start of the year, marketed on full ROCm support, with one of its own demos running on the exact model my board can't load under ROCm.
A BIOS Update Won't Fix #6182 — I Tried the Newest One
The Bosgame M5's ROCm bug is board-specific, not chip-specific — so firmware is the obvious lever. I flashed Bosgame's newest official BIOS hoping to dodge it. It didn't work, and the negative narrows where the fault actually lives.
Full Context on a Vulkan-Only Strix Halo: The Decode-Drop Reproduces, but the Sweet Spot Moves
kmarble showed ROCm decode collapses 64% at full context on Strix Halo, and ROCm+MTP cures it. My board can't run ROCm. The Vulkan half reproduces the drop — but the MTP sweet spot from last week walks left at depth: by 76k, drafting too deep is slower than no speculation at all.
MTP Defaults Are a Trap: What 260 Runs Showed About Speculative Decoding on Qwen3.6
Until May 19, the llama.cpp speculative-decoding default was 16. On Qwen3.6's single MTP head, that default cost up to 75% of generation throughput. Here's where the real sweet spots are — and why they're architecture-specific.
ROCm 7.x on the Bosgame M5: 14 Configurations, 14 Failures
We promised a ROCm 7.x revisit. We got a comprehensive workaround sweep instead. Both are useful.
Vulkan/RADV vs ROCm 6.4 on Strix Halo: What 128 Benchmark Runs Actually Showed
The headline isn't where Vulkan wins. It's where ROCm doesn't run at all.