Building intuition about LLM parameter counts

When I was building my GPT-2 implementation in JAX , I started with just token embeddings for the input, and a separate output head (as I was not using weight tying ). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train…