Engineering blog on distributed systems, LLM infrastructure, and observability
I wanted to actually understand LLM-as-a-judge instead of nodding along the way I used to with CAP theorem before I'd actually read the paper. So I built a proper research pipeline, tried to find something genuinely novel to say about it, and watched all five of my "novel" ideas get killed by papers published in the last few months. Here's everything I learned along the way — with the receipts.
The LLM Serving Stack You Actually Need: From NGINX to llm-d to GPU
Grafana AI Observability launched on April 21. I spent a few days building a five-agent incident response system to put it through its paces. Here's what actually happened.
I deployed llm-d's P/D disaggregation guide — separate prefill and decode pods, NIXL sidecar, the full architecture. The pods ran. The NIXL KV transfers showed No Data. Here is what that tells you about what disaggregation actually requires.
The stack is deployed. Now let's see what it actually does. EPP prefix cache routing, 81.1% KV cache hit rate, TTFT at 15ms p50, and what those numbers mean for teams building multi-tenant inference at scale.
I deployed llm-d on a Lambda Labs GH200. Nothing worked first try. Here is the honest account of what broke, why, and how to fix it — so you don't spend your GPU budget finding out the hard way.
I treated an M4 Mac Mini as a production-like inference environment — wired up Prometheus, Grafana, a kind cluster with nginx, and ran real load tests. Here's what the numbers actually showed.
An Engineer's annotated tour through what actually happens when you hit send — from bytes to tokens to embeddings to attention to the word your model finally spits out. No skipped steps. No "and then magic happens."
Translating SQL to NoSQL: Architecture Deep-Dive
Part 2 of 2: Empirical evaluation of classification accuracy, routing performance, and cost attribution — with honest analysis of failure modes