Parallelism strategies for neural network training

Any large-enough neural network training job (e.g., training a LLM or an image or video generation model) employs parallelism strategies. In fact, such large scale training jobs often employ multiple strategies in conjunction. There are two reasons for using parallelism. First, the models are large enough that they don't fit on a single accelerator's memory and hence need to be split up with each…