# prefix sum (blogs) — RSS Amplifier Recent posts from the 2 feeds in the RSS Amplifier directory that cover prefix sum. Page: Feed: --- ## [Softmax and Cross-Entropy Backward Pass](https://shreyansh26.github.io/post/2026-07-06_softmax-cross-entropy-backprop/) _2026-07-06 · Shreyansh Singh · blank_ A step-by-step derivation of softmax, logsoftmax, and cross-entropy backward passes: how the softmax Jacobian turns into a row-wise dot product, and why the logits gradient is p - y. ## [Decompose-K: From torch.compile to Hand-Tuned Triton Kernels for Skinny Large‑K Matmuls](https://shreyansh26.github.io/post/2026-06-21_decompose-k-triton/) _2026-06-21 · Shreyansh Singh · blank_ An implementation deep dive into Decompose-K matmul: why splitting the K dimension helps skinny large-K GEMMs, what torch.compile and Inductor custom-op autotuning emit, and how a vectorized split-reduction Triton kernel ends up beating both. ## [KV Cache Compaction and Compression: From Attention Sinks to Learned Memory](https://shreyansh26.github.io/post/2026-06-01_kv-cache-compaction-compression/) _2026-06-01 · Shreyansh Singh · blank_ A code-first guide to KV cache compression: why the cache dominates long-context serving, how token-eviction methods work, and how Cartridges and STILL turn compact KV tensors into reusable memory. ## [Paper Summary #17 - Engram](https://shreyansh26.github.io/post/2026-05-17_engram-layers/) _2026-05-17 · Shreyansh Singh · blank_ A technical explainer for DeepSeek's Engram layers: conditional memory, hashed n-gram lookup, context-aware gating, sparse-capacity allocation, and the implementation path inside Transformer blocks. ## [Paper Summary #16 - Canon Layers](https://shreyansh26.github.io/post/2026-05-16_canon-layers/) _2026-05-16 · Shreyansh Singh · blank_ A deep dive into Canon Layers: why sequence models need cheap horizontal token flow, how residual causal depthwise convolution implements it, and where Canon-A/B/C/D fit inside Transformer and linear-model blocks. ## [Paper Summary #15 - Hyper-Connections and mHC](https://shreyansh26.github.io/post/2026-05-15_hyper-connections-mhc/) _2026-05-15 · Shreyansh Singh · blank_ From residual-stream basics to manifold-constrained mixing: why widening the residual path helps, why unconstrained products destabilize depth, and how Sinkhorn-Knopp turns HC into conservative feature routing. ## [Deep dive into CUDA Scan Kernels: Hierarchical and Single-Pass Variants](https://shreyansh26.github.io/post/2026-02-19_cuda-scan-kernels/) _2026-02-19 · Shreyansh Singh · blank_ A guided tour of hierarchical and single-pass CUDA scan kernels with coarsening and warp-level optimizations. ## [Paper Summary #14 - Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://shreyansh26.github.io/post/2026-01-17_physics-of-lms-3-1-knowledge-storage-and-extraction/) _2026-01-17 · Shreyansh Singh · blank_ My notes from the Physics of Language Models series of papers. ## [Understanding Multi-Head Latent Attention (MLA)](https://shreyansh26.github.io/post/2025-11-08_multihead-latent-attention/) _2025-11-08 · Shreyansh Singh · blank_ A mathematical and code deep-dive on one of the key innovations from Deepseek - Multihead Latent Attention (MLA) ## [Reach prospects with real signals (Sponsored)](https://crawlproof.com/a/NPogTLC6cdSg) _2025-11-07 · **Sponsored**_ Find prospects talking about your problem and get context-backed moments to engage ## [Deriving the Gradient for the Backward Pass of Layer Normalization](https://shreyansh26.github.io/post/2025-06-04_layernorm-gradients/) _2025-06-04 · Shreyansh Singh · blank_ Understanding the math behind Layer Normalization and deriving the gradients for the backward pass.