Gemma 3 270M’s Digit Embeddings
Google released a tiny Gemma model recently, Gemma 3 270M. What’s interesting about the size of this model is that out of 270 million parameters, 170 million of them are embedding vectors, and only 100 million parameters are for transformer blocks. It is uncontroversial to say… a lot of information is encoded in these embedding vectors.