Local SGD and DiLoCo Research Musings

Here are some notes I wrote over this topic. I&rsquo;ve switched my master&rsquo;s thesis to a different topic, but there were many interesting research directions I found in this area. Local SGD and DiLoCo Overview # It is October 15th, 2025. For my last year of my master&rsquo;s, I decided to a thesis around distributed low-communication training. Essentially, how can we train large models…