What's Wrong with Chat-Templates Format for LLM

This is going to be a long read (I rarely write these), but I believe it's a useful and interesting one with some non-trivial twists along the way. Chat-templates are broken, and I'll explain why.