Schizo embeddings: Initializing special tokens the complicated way

I’ve read some complaints about Llama using too many special tokens. One of the issues with this is that those tokens are typically initialized randomly and then trained for a very short time in post-training, which leads to poorly trained embeddings and weird behavior.