Part 1: Why Off-Policy Breaks RL — An SGA Analysis Framework

Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch The Problem In reinforcement learning, we often cannot sample directly from the policy $\pi_\theta$ we are optimizing. Instead, we sample from a different behavior policy $\mu$. This off-policy setting ($\mu \neq \pi$) arises from multiple sources: Standard…