RL Bite: Monotonic Policy Improvement and Deriving Proximal Policy Optimization (PPO)

Abstract Link to heading A while ago we looked into Policy Gradient and Reinforce. Policy gradient is versatile and under mild conditions it is guaranteed to converge to a local minimum (if we choose the correct policy and step size). This is already a huge step up when compared to Q Learning, which may just diverge. However, we may still want stronger guarantees like monotonic improvement at each…