Understanding GPT-OSS architecture

An in-depth look at the architecture of GPT-OSS, an open-source large language model released by OpenAI on 5th August 2025. It is a MoE GPT-2-style Transformer with 36/24 layers, 128/32 experts with top-4 routing, RMSNorm, Grouped Query Attention + RoPE attn, 131 K context via YaRN, 4-bit MXFP4 packs 120 B on 80 GB & 20 B on 16 GB, and SwiGLU activation.