The Optimal Token Baseline

Project Lead: Yingru Li Co-First Authors: Yingru Li and Jiawei Xu TL;DR The Problem : RL training for LLMs frequently suffers from &ldquo;training collapse&rdquo; due to exploding gradient variance in long-horizon tasks. Standard baselines (like Group Mean) fail because they treat all tokens and sequences as equally &ldquo;noisy.&rdquo; The Insight : Gradient noise is heterogeneous. We derive the…