GPU MODE Lecture 12: Flash Attention

Tip This post is part of the following series: GPU MODE Lecture Notes : My notes from the GPU MODE reading group lectures run by Andreas Kopf and Mark Saroufim . Introduction Background: The Need for Optimization Memory Hierarchy and Performance Attention Mechanisms: A Brief Overview Flash Attention: Tiling Strategy The Softmax Challenge Flash Attention 2: Advanced Techniques Implementation…