How to make LLM checkpoint writes fast

Unless you have been living under a rock for the last couple of years or so, you already know what a large language models (LLM) is. For the rock-dwellers, here is a quick note: a LLM is a machine learning model (more specifically, a neural network) trained on a large corpus of data (think: all the text on the internet) to predict a sequence of words. The outcome is a seemingly "smart" model that…