Multi-Node Training on TACC Vista

Introduction SLURM Script Environment Variables Testing test_mpi.py test_simple_distributed.py test_lightning_mpi.py Multi-node training can be tricky. When training on multiple nodes, machines must communicate with one another, synchronizing gradients and state. Most frameworks handle most of the heavy lifting behind distributed training algorithms (like torch's DDP implementation). However,…