Continual training of Llama-3.1-8B for 809B tokens

Over the last couple of months, I’ve been continually training the pretrained Llama-3.1-8B model (with a context length of 8192 tokens) for 809B tokens. This was my first truly large-scale distributed training experience and in this post, I’d like to share some of what I’ve learned so far. Why First of all, why am I… Continue reading Continual training of Llama-3.1-8B for 809B tokens →