Why do OpenAI's GPT-2 weights beat mine?

When I finished my project training an LLM from scratch , I was left with a minor mystery. Why were my models worse at instruction-following than the original OpenAI GPT-2 small weights? I had an evaluation that I was running, based on the instruction fine-tuning code in chapter 7 of " Build a Large Language Model (from Scratch) ". The process was to train a model on samples from the Alpaca…