Native Sparse Attention

Wow, just read the #DeepSeek Native Sparse Attention paper, and they basically do exact same things I considered most promising from experiments back in 2022. On the screenshots: top-k settings from an experiment that mirrors their Token Selection (section 3.3.2 of the paper)