LLM pretraining on TPU v6e with a $50 budget

Andrej Karpathy has the nanochat project with the description “The best ChatGPT that $100 can buy”. He evolved a model architecture and training setup that reaches the performance of GPT-2 while costing 600 times less than the original OpenAI run from 2019. This is an inspiring example, showing that pretraining experiments can now be available even to individuals without corporate/university…