Deriving Linear Attention and DeltaNet (Part 1)

Showing that softmax attention and other sub-quadratic models (linear attention, SSMs, etc.) belong to the same class of models that differ in how they store, optimize, and forget memories.