GPU MODE Lecture 13: Ring Attention

Tip This post is part of the following series: GPU MODE Lecture Notes : My notes from the GPU MODE reading group lectures run by Andreas Kopf and Mark Saroufim . Introduction and Overview Motivation: Long Context Transformers and Applications Background: Multimodal Input and Transformers Challenge: Memory Limitations Vanilla Attention Recap Compute Scaling with Context Length Online Softmax and…