Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix

I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post . While I was writing up the results of my first experiment into possible causes, I ran the post past ChatGPT -- I always use an "editorial board" of AIs to check my posts for flow, style, and any technical errors (though all writing is always…