Retrieval-Augmented Training: RETRO Architecture

How RETRO trains language models with retrieval from scratch, using chunked cross-attention to integrate a 2T-token database and cut parameter needs 25x. Read full notebook