Hello, I'm Ali. I am a CS student interested in how systems work on the lower-level, and hope to keep learning and building until I can optimize every system...
Two teams are running the same model on the same hardware. Team A's p99 time to first token (TTFT) is 340ms. Team B's is 1.2 seconds. Both are using vLLM (an inference server for LLMs) with default settings. Team B has been tuning batch size for a week. The difference isn't the hardware or the model, but rather that Team A knows their workload is prefill-heavy and their KV cache is hitting 90%…
I've been getting into CUDA for the past month, and profiling GPU kernels (functions that run in parallel on the GPU) always felt painful. You run NSight Compute (NCU, a kernel profiling tool for debugging), on v1, stare at the metrics, run NCU on v2, then try to hold both profiles in your head simultaneously. AMD actually ships a tool for this, ROCm Compute Profiler has a baseline comparison…
Mamba has been a key figure in the topic of efficiency; the SSD layer is a linear recurrence that scales better than attention at long contexts. The recurrence itself is sequential, each hidden state depends on the previous one. Unrolling the computation shows a parallel prefix scan, and the implementation recognizes this at the algorithm level and parallelizes across chunks, but inside each chunk…
I always wanted a go-to place for F1 statistics, every other alternative seemed out-dated and didn't really have a compelling UX. A lot of the current F1 websites have the same format: search for driver/race track -> look at basic statistics. I wanted to combine the both into a single website, and the best solution looked like Statmuse . The great part of the UX is it allows twitter fans to get…