# liu (blogs) — RSS Amplifier Recent posts from the 2 feeds in the RSS Amplifier directory that cover liu. Page: Feed: --- ## [Beyond Precision: Why Training-Inference Mismatch is an Optimization Problem and How Simple LR Scheduling Fixes It](https://richardli.xyz/post/mismatch-lr-schedule/) _2025-12-20 · Yingru Li_ Corresponding Author: Yingru Li Co-First Authors: Yaxiang Zhang and Yingru Li TL;DR The Problem: Reinforcement Learning (RL) training for LLMs is notoriously unstable. While recent studies attribute this to “training-inference mismatch” (caused by hybrid engines), standard fixes like Importance Sampling might fail during longer training runs. The Insight: We analyze this instability… ## [The Optimal Token Baseline](https://richardli.xyz/post/optimal-token-baseline/) _2025-12-20 · Yingru Li_ Project Lead: Yingru Li Co-First Authors: Yingru Li and Jiawei Xu TL;DR The Problem : RL training for LLMs frequently suffers from “training collapse” due to exploding gradient variance in long-horizon tasks. Standard baselines (like Group Mean) fail because they treat all tokens and sequences as equally “noisy.” The Insight : Gradient noise is heterogeneous. We derive the… ## [Trust Region Masking for Long-Horizon LLM Reinforcement Learning](https://richardli.xyz/post/trust-region-masking/) _2025-12-20 · Yingru Li_ Authors: Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Baoxiang Wang 📄 arXiv: https://arxiv.org/abs/2512.23075 📊 Slides: trust\_region\_masking\_slides\_yingru.pdf 🔧 PR #4544: feat: trust region sequence masking Abstract Policy gradient methods for large language models optimize a surrogate objective computed from samples of a rollout policy $\\pi\_{\\mathrm{roll}}$. When… ## [The Stability Gap: Why Top-K Routing Breaks RL Optimization](https://richardli.xyz/post/topk-routing-stability-gap/) _2025-12-07 · Yingru Li_ Citation @online { li-2025-topk-stability , title = {The Stability Gap: Why Top-K Routing Breaks {RL} Optimization} , author = {Li, Yingru} , year = {2025} , month = dec , url = {https://richardli.xyz/post/topk-routing-stability-gap/} } The Problem Training Mixture of Experts (MoE) language models with Reinforcement Learning can be unstable. While dense LLMs have continuous and differentiable… ## [Scalable Exploration via Ensemble++](https://richardli.xyz/post/scalable-exploration/) _2025-11-29 · Yingru Li_ The Exploration Challenge in Deep RL Thompson Sampling is one of the most elegant algorithms for balancing exploration and exploitation in sequential decision-making. The idea is simple: maintain a posterior distribution over reward functions, sample from it, and act optimally with respect to the sample. This approach naturally trades off exploring uncertain regions against exploiting known good… ## [Language as a Universal Interface for Reinforcement Learning Agents](https://richardli.xyz/post/language-rl-agent/) _2025-11-07 · Yingru Li_ Citation @article { li2025languageagent , title = {Language as a Universal Interface for Reinforcement Learning Agents} , author = {Li, Yingru} , journal = {Richard Li's Blog} , year = {2025} , url = {https://richardli.xyz/post/language-rl-agent/} } Introduction Why do some agents succeed while others fail? While frameworks like ReAct (Yao et al., 2023) show that interleaving reasoning and acting… ## [Mathematical Formulations of Rollout Correction Methods](https://richardli.xyz/post/verl-rollout-correction/) _2025-11-04 · Yingru Li_ Author: Yingru Li Abstract This document provides the definitive mathematical formulations for rollout correction methods in verl , following the natural progression from REINFORCE to PPO to Decoupled PPO . Rollout correction provides a unified framework to handle general off-policy problems in RL training - any scenario where the data collection distribution differs from the training… ## [Part 3: Trust Region Optimization via Sequence Masking](https://richardli.xyz/post/rl-collapse-part3/) _2025-11-04 · Yingru Li_ Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Series Context Part 1 : We established the SGA (Stochastic Gradient Ascent) framework and identified two failure modes of off-policy mismatch: Bias (measured by $D\_{TV}$) and Variance (measured by $\\chi^2$-divergence). Part 2 : We analyzed gradient estimators and… ## [Part 2: Applying the SGA Framework — Token v.s. Sequence-level Correction](https://richardli.xyz/post/rl-collapse-part2/) _2025-10-31 · Yingru Li_ Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch Citation @online { liu-li-2025-rl-collapse , title = {When Speed Kills Stability: Demystifying {RL} Collapse from the Training-Inference Mismatch} , author = {Liu, Jiacai and Li, Yingru and Fu, Yuqian and Wang, Jiawei and Liu, Qian and Shen, Yu} , year = {2025} ,… ## [Part 1: Why Off-Policy Breaks RL — An SGA Analysis Framework](https://richardli.xyz/post/rl-collapse-part1/) _2025-10-30 · Yingru Li_ Authors: Yingru Li , Jiacai Liu Original Blog: When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch The Problem In reinforcement learning, we often cannot sample directly from the policy $\\pi\_\\theta$ we are optimizing. Instead, we sample from a different behavior policy $\\mu$. This off-policy setting ($\\mu \\neq \\pi$) arises from multiple sources: Standard… ## [Trade Smarter With IBKR (Sponsored)](https://crawlproof.com/a/1nqu4eoGAjcH) _2025-10-29 · **Sponsored**_ Join Interactive Brokers through a referral and start investing with low-cost global access. ## [Information Bandwidth in Reinforcement Learning](https://richardli.xyz/post/information-bandwidth-rl/) _2025-10-01 · Yingru Li_ How Gradient Structure Determines Information Capacity When I first read the “ LoRA Without Regret ” blog post, one claim caught my attention: policy gradient algorithms learn roughly 1 bit of information per episode . This insight elegantly explains why LoRA—with its mere thousands of trainable parameters—works so remarkably well for RL fine-tuning of large language models. But what… ## [When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch](https://richardli.xyz/post/rl-collapse-training-inference/) _2025-09-17 · Yingru Li_ Co-First Authors: Jiacai Liu and Yingru Li Corresponding Authors: Yingru Li and Yu Shen TL;DR The relentless push for faster inference has created a dangerous “training-inference mismatch” that can silently kill reinforcement learning with LLMs. Our investigation reveals a vicious cycle that is particularly acute in modern reasoning and agentic RL: OOD Contexts Drive Low-Probability… ## [MBTA-Assisted Wachusett Region Bike Overnight: Wachusett Station-Barre-Worcester](/posts/2025-05-07-wachussets-barre-worcester/) _2025-05-07 · About on Chris Woodall_ Origin My fiancée Emily and I have been riding our bikes more frequently this year. I’ve had a great start to my season with plenty of early days on my hardtail (Banshee Paradox) and a fun “race/ride” at the Mix Tape event in Freetown, MA. As Emily built up her own riding strength, we wanted to embark on a multi-day trip together. I’ve been interested in bikepacking since…