Trying out other peoples’ ideas
In response to my post announcing my second modded-nanogpt medium world record, several people made suggestions for variations of or alternatives to the technique introduced in the article.
snimu’s blog
Thoughts on LLMs, AI, and language model training
Latest posts
modded-nanogpt medium world record: Re-using intermediate activations in the output latents
Adding the output of layer 11 to the final output latents in a learned, weighted sum leads to a separation of concerns for early model layers: it enables them to focus only on providing context to the next layer, without directly impacting the final prediction. Backing them out of the residual stream this way improves model performance and led me to a new modded-nanogpt medium track world record.
modded-nanogpt world record: Decoupling embedding size from model dimension
I have achieved a (not yet official) modded-nanogpt medium track world record.
modded-nanogpt medium world record: adding value embeddings
Adding more value-embeddings to modded-nanogpt led me to an (as of yet unofficial) modded-nanogpt medium record as seen in PR#119.
modded-nanogpt: Analyzing value-embedding-, UNet-, and x0-lambdas
The modded-nanogpt medium-track makes use of multiple tricks to improve performance, which rely on learned scalar values for mixing two vectors:
Separating Simulacra: tags for smarts and safety
I’ve seen multiple papers where LLM training data included tags providing metadata, which improved the models’ downstream performance. However, as far as I can tell, they always end training on pure text data so that the models can be used without tags. I don’t understand why that should be desireable.
Model stacking doesn’t work
In my article about Model Stacking, I proposed a method for decentralized pre-training of models.
Infinite tool use
An LLM should never output anything but tool calls and their arguments.
Schizo embeddings: Initializing special tokens the complicated way
I’ve read some complaints about Llama using too many special tokens. One of the issues with this is that those tokens are typically initialized randomly and then trained for a very short time in post-training, which leads to poorly trained embeddings and weird behavior.
My dream VLM
Edited on: 2025-04-11; 2025-05-04.