Making LLMs More Efficient With Search Heads

It has always seemed a little crazy to me that LLMs are expected to re-learn how to do things that can be done trivially on a CPU. For example, it takes billions of FLOPS for an LLM to compute 2 + 2. While mathematics is definitely something I want to tackle in the future, I’ve started with something simpler that LLMs are expected to learn: how to copy sequences in the input string to the output.…