Mathematical Formulations of Rollout Correction Methods

Author: Yingru Li Abstract This document provides the definitive mathematical formulations for rollout correction methods in verl , following the natural progression from REINFORCE to PPO to Decoupled PPO . Rollout correction provides a unified framework to handle general off-policy problems in RL training - any scenario where the data collection distribution differs from the training…