Inside TPU and GPU Clusters: The Anatomy of Collective Communication
A deep dive into all-gather, reduce-scatter, all-reduce, and all-to-all communication on TPUs and GPUs.
Aleksa Gordić's Blog
Long thinking.
Latest posts
Inside the Transformer: The Life of a Token
A deep dive into a modern dense transformer: YaRN, hybrid attention, soft capping, QK normalization, FLOPs/token, cluster sizing, and more.
From Oura Ring to Dataset: Analyzing and Visualizing a Year of Biometric Data
Visualizing and analyzing sleep, heart rate, HRV, activity, stress, and SpO2 data from the Oura Ring API.
Inside NVIDIA GPUs: Anatomy of high performance matmul kernels
From GPU architecture and PTX/SASS to warp-tiling and deep asynchronous tensor core pipelines.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.
Neocambria
Humans in the post-ASI world.
ELI5: FlashAttention
How does Flash Attention really work?
Landing a job at DeepMind
How i landed a job at DeepMind as a research engineer without an ML degree.
Learn RL
How I got started with reinforcement learning.
Learn Geometric ML
How I got started with geometric/graph machine learning.
Learn transformers
How I got started with transformers.
Computing machinery and intelligence - Alan Turing (II)
Breaking down ideas from Turing's seminal paper - part 2.
Computing machinery and intelligence - Alan Turing (I)
Breaking down ideas from Turing's seminal paper - part 1.
Learning how to learn
Learnings from Coursera course.
Learn ML
My journey into machine learning.