hi, i'm emre — ml systems & performance engineer. i build and optimize inference systems for production: low-latency audio models, custom CUDA/CuTe-DSL ...
This is the story of a megakernel: the 28-layer text decoder of Qwen3-ASR-0.6B, plus its lm_head, running as a single persistent CUDA kernel on an RTX 5080. The baseline fires about 466 kernel launches per generated token. This fires one. Every op in it is hand-written - the GEMVs, the norms, the rope, the KV-cache write, the attention, the sampling argmax - and every op was validated against…
If you do kernel-level inference optimization, you eventually hit the cold truth: the enemy is not compute, it is VRAM bandwidth. FP32 numbers are huge in memory. The fix is to squeeze them into 4-bit boxes. The internet is full of repos that do this. They say "find amax, divide by this, shift the bits, here is your FP4," and move on. I copied those formulas into my own code at first - and could…
This is the story of one CUDA kernel pair in cublade, my personal kernel library. The kernels do per-tensor symmetric FP8 (E4M3) quantization and dequantization. They started at 429 GB/s on a v1 build, then NCU shoved them to roughly 880 GB/s at the kernel level - about 98.5% of the RTX 5080's real DRAM peak. Plus a dequantizer that landed DRAM-bound on its first compile. I'm going to walk through…
My STT journey started with Whisper. One of Freya's STT models was a fine-tuned Whisper Large-V3. I optimized it with TensorRT, got it to 130ms, around 94x realtime. Solid, but we hit a wall. The architecture has a heavy encoder and a lightweight decoder, and TRT could only help with the encoder side. We needed something better. STT is the first link in our voice agent chain: STT → LLM → TTS. When…
In Phase 1 , we demonstrated that outliers in transformer models are not "statistical noise"—they are critical carriers of model intelligence and context. When we zeroed outliers in weight matrices, model outputs degraded catastrophically, producing repetitive loops and nonsensical text. But here's the problem : Quantizing weights from BF16 to INT8 reduces memory by ~50%, but it also destroys…
Introduction Many assume that large language models (LLMs, Transformers) simply “memorize data” due to their scale. In reality, some part of their "intelligence" and creative capacity is packed into outlier values, extremely large or small numbers in their weight matrices and activation tensors. In this post, we’ll zero out these outliers and directly observe how this disrupts the model’s output…
Hey there! In this post, I’m going to walk you through the building blocks of the Transformer architecture. But don’t worry—this isn’t going to be one of those dry academic reads. Think of it more like we're sitting down for a coffee and chatting about how all of this works. The goal? No more “What the heck is this, bro?” moments. Everything will be clear, with examples and just enough math to…