My first Multi-GPU kernel: Writing All-to-all for AMD MI300X

Last month, I participated in the AMD Distributed Challenge, hosted by GPU MODE . This was very exciting for me as it was the first time I learned how to write a multi-GPU kernel! Although I had a brief understanding of how DDP and FSDP worked under the hood via collective primitives like all-reduce and reduce-scatter, I didn&rsquo;t know it was possible to perform remote memory access directly…