BlaGPT notes on Softpick, Canon Layers, Parallel Transformer

Recently, I’ve been testing different architectural modifications in my BlaGPT benchmark to see how they affect performance. I wanted to share what I’ve learned about three interesting techniques: Softpick, Canon Layers, and Parallel Transformer blocks. Softpick: A Different Approach to Attention Softpick offers an interesting alternative to the standard softmax function in attention blocks. It…