Reproducing GPT2-125M

Summary This is a detailed account of Reproducing GPT2-small (GPT2-125M) on Open-source Frameworks, Datasets and publicly available GPU cloud infra. The overall stack is : GPU Infra : Runpod Dataset : OpenWebText (via HF/datasets) Training : HF Trainer, Pytorch, Accelerate Eval : LLM-eval Benchmark Monitoring : Wandb Conclusion: Reproduced GPT2-125M open weights off HuggingFace for Hellaswag…