Corresponding Author: Yingru Li Co-First Authors: Yaxiang Zhang and Yingru Li TL;DR The Problem: Reinforcement Learning (RL) training for LLMs is notoriously unstable. While recent studies attribute this to “training-inference mismatch” (caused by hybrid engines), standard fixes like Importance Sampling might fail during longer training runs. The Insight: We analyze this instability…
Project Lead: Yingru Li Co-First Authors: Yingru Li and Jiawei Xu TL;DR The Problem : RL training for LLMs frequently suffers from “training collapse” due to exploding gradient variance in long-horizon tasks. Standard baselines (like Group Mean) fail because they treat all tokens and sequences as equally “noisy.” The Insight : Gradient noise is heterogeneous. We derive the…
Citation @online { li-2025-topk-stability , title = {The Stability Gap: Why Top-K Routing Breaks {RL} Optimization} , author = {Li, Yingru} , year = {2025} , month = dec , url = {https://richardli.xyz/post/topk-routing-stability-gap/} } The Problem Training Mixture of Experts (MoE) language models with Reinforcement Learning can be unstable. While dense LLMs have continuous and differentiable…
The Exploration Challenge in Deep RL Thompson Sampling is one of the most elegant algorithms for balancing exploration and exploitation in sequential decision-making. The idea is simple: maintain a posterior distribution over reward functions, sample from it, and act optimally with respect to the sample. This approach naturally trades off exploring uncertain regions against exploiting known good…
Citation @article { li2025languageagent , title = {Language as a Universal Interface for Reinforcement Learning Agents} , author = {Li, Yingru} , journal = {Richard Li's Blog} , year = {2025} , url = {https://richardli.xyz/post/language-rl-agent/} } Introduction Why do some agents succeed while others fail? While frameworks like ReAct (Yao et al., 2023) show that interleaving reasoning and acting…
Author: Yingru Li Abstract This document provides the definitive mathematical formulations for rollout correction methods in verl , following the natural progression from REINFORCE to PPO to Decoupled PPO . Rollout correction provides a unified framework to handle general off-policy problems in RL training - any scenario where the data collection distribution differs from the training…
Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Series Context Part 1 : We established the SGA (Stochastic Gradient Ascent) framework and identified two failure modes of off-policy mismatch: Bias (measured by $D_{TV}$) and Variance (measured by $\chi^2$-divergence). Part 2 : We analyzed gradient estimators and…
Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Citation @online { liu-li-2025-rl-collapse , title = {When Speed Kills Stability: Demystifying {RL} Collapse from the Training-Inference Mismatch} , author = {Liu, Jiacai and Li, Yingru and Fu, Yuqian and Wang, Jiawei and Liu, Qian and Shen, Yu} , year = {2025} ,…
Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch The Problem In reinforcement learning, we often cannot sample directly from the policy $\pi_\theta$ we are optimizing. Instead, we sample from a different behavior policy $\mu$. This off-policy setting ($\mu \neq \pi$) arises from multiple sources: Standard…
How Gradient Structure Determines Information Capacity When I first read the “ LoRA Without Regret ” blog post, one claim caught my attention: policy gradient algorithms learn roughly 1 bit of information per episode . This insight elegantly explains why LoRA—with its mere thousands of trainable parameters—works so remarkably well for RL fine-tuning of large language models. But what…
Co-First Authors: Jiacai Liu and Yingru Li Corresponding Authors: Yingru Li and Yu Shen TL;DR The relentless push for faster inference has created a dangerous “training-inference mismatch” that can silently kill reinforcement learning with LLMs. Our investigation reveals a vicious cycle that is particularly acute in modern reasoning and agentic RL: OOD Contexts Drive Low-Probability…