Comparing Ulysses and Megatron Sequence Parallelism

Introduction I recently built a small (mainly pedagogical) distributed training engine to understand how different forms of model parallelism work in practice. This post compares sequence parallelism as used in Megatron-LM and DeepSpeed. Although both systems keep the transformer residual stream partitioned as [B, S/P, H] (where P is group size), the mechanisms they use around attention are quite…