Inside TPU and GPU Clusters: The Anatomy of Collective Communication

A deep dive into all-gather, reduce-scatter, all-reduce, and all-to-all communication on TPUs and GPUs.