Cheaply Approximating KL Against an EMA: An Async RL Hack

KL regularization against a reference policy can stabilize and improve RL training of LLMs, but computing the KL term requires an additional forward pass through the reference model. We show that asynchronous RL settings that avoid this extra cost by defining the reference policy to be the inference policy (the trainer policy from ∆ steps [ ]