Personal website for some random tidbits I work on
In this post, we’ll dig deep into how TileIR works, from how it generates instructions to analyzing its different passes. We’ll trace how a Mixture-of-Experts (MoE) kernel written in CuTile gets compiled down through cuda_tile → nv_tileaa → nv_tileas → NVVM → LLVM → SASS. Here’s what to expect: What is CuTile? — The tile-centric programming model Running Example — An MoE kernel we’ll trace through…
This post will be about going through https://abseil.io/fast/hints.html#performance-hints , a blog post written by the power duo Jeff Dean and Sanjay Ghemawat who argubly made google to what it is today. This is a knowledge distillation from the both of them with many examples from the internal codebase. Hopefully I can a thing or two professionals who have worked in the industry longer than I…
Motivation So I was browsing Hacker News and came across this interesting post: Fp8 runs ~100 tflops faster when the kernel name has “cutlass” in it . This was from Triton tutorial where someone noticed that adding “cutlass” to their kernel name gave them an additional 100-150 TFLOPs. That’s a huge improvement just from… a name? Mentions 100 TFLOPs improvement (Image source: Github pull ) Mentions…
Series An Intro to DeepSeek’s Distributed File System A Reality Check on DeepSeek’s Distributed File System Benchmarks Network Storage and Scaling Characteristics of a Distributed Filesystem Table of Contents The Benchmarking Pyramid Network Baseline Benchmark Storage Baseline Benchmark 3FS Performance Analysis Scaling Block Size Scaling Number of Nodes Wrapping up Refresher In my first post , I…
AI usage has become universal. Teams everywhere are building RAG, generating embeddings, and training increasingly sophisticated agents. Most distributed LLM training guides focus on model architecture and hyperparameters while ignoring a critical bottleneck: infrastructure configuration. Network and storage choices often determine whether training takes hours or days. I ran benchmarks finetuning…
AI 2027 and related works This will be my thoughts about AI 2027 by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifl and, Romeo Dean. It will also cover two other works, Gradual Disempowerment and AI-espionage since they are related. These essays/blogs were recommended to me by someone (have not asked for permission, so will not put name here) My thoughts of the different works AI 2027 -…
Always Measure One Level Deeper Thoughts about Always Measure One Level Deeper by John Ousterhout. Before we dive into this, this was written in 2018 when John was not retired yet (I think) Thoughts along the way Performance measurement is one of the most important parts of software development. In academic research a thorough performance evaluation is considered essential for many publications to…
Paul Graham - Why Nerds Are Unpopular Thoughts about Why Nerds Are Unpopular by Paul Graham. Before we dive into this, this was written in 2003. This was when Paul Graham was 38, when he was not married or have kids. This is also a rather long essay… Thoughts along the way We sat at a D table, as low as you could get without looking physically different. We were not being especially candid to…
Series An Intro to DeepSeek’s Distributed File System A Reality Check on DeepSeek’s Distributed File System Benchmarks Network Storage and Scaling Characteristics of a Distributed Filesystem How should we analyze 3FS? In my previous post , I introduced DeepSeek’s 3FS distributed file system – exploring its architecture, components, and the CRAQ protocol that provides its consistency guarantees.…
Paul Graham - What to Do Thoughts about When To Do by Paul Graham. Thoughts along the way What should one do? That may seem a strange question, but it’s not meaningless or unanswerable. It’s the sort of question kids ask before they learn not to ask big questions. This statement about kids kind of took me off guard - I do see it happen (at least in myself). Why though? Does it see in his children…