Hi anon, I am Vishal, currently pretending to be an ML engineer at Sync. Currently, I am learning about GPU optimizations (or so I am telling myself)...
Sequel to "The kernel that wasn't there" . Last time I profiled my way out of writing a kernel and shipped streaming instead. This time I wrote them: a skinny bf16 GEMV that beats cuBLAS by up to 1.36x, and a four-stage SSD scan that loses to Triton by about 2x. The two outcomes together are the point. Last post ended on a promise. Profiling a Mamba2 decode step told me the scan everyone points at…
Or: what 7% taught me before I wrote a single kernel. I set out to write a fused Mamba2 decode kernel for Zonos. The plan, in writing, said 2 to 3x on the per-frame decode step. The first thing profiling told me was that the kernel I was about to write would buy me about 7%, and that the part of it everyone treats as the bottleneck was already sitting at the memory roofline. So I didn't write it.…
I wrote a fused decode-attention kernel for an RL training loop, got it 2.2× faster than the SDPA path it replaces at the microbenchmark level, dropped it into HuggingFace's generate , and watched the decode step get nearly 3× slower . The kernel was doing exactly what the microbench said it would. The integration broke an auto-compile path that the baseline was quietly benefiting from. This post…
After writing Flash Attention 1 in plain CUDA, I wanted to take the next step and port Flash Attention 2 to CuteDSL , the Python front-end NVIDIA ships with CUTLASS 4. CuteDSL is new, under-documented, and surprisingly fun once you wrap your head around it. I do not write kernels every day, and I'm still building intuition for both GPU hardware and CuteDSL itself, so this project involved a lot of…