Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining

The GPT-2-style models that I've been training work really well, and I've even managed to train some that perform better than the original OpenAI small model in terms of cross entropy loss on a test set. But as I wrote previously , there's a mystery: why do they perform worse on my instruction fine-tuning evaluation? I had various theories about why that might be, and to me, the most…