About DeepSeek Sparse Attention

DeepSeek Sparse Attention , some considerations about it after reading the paper. DeepSeek-V3.2-Exp is defined as an experimental sparse-attention model . Its architecture is the same as DeepSeek-V3.1-Terminus, except for the introduction of DeepSeek Sparse Attention (DSA). The core idea is to reduce the number of key-value pairs each query token looks at, instead of attending to all tokens.…