GRPO

Main idea Key point it to understand the below pictures Iteration steps for each input, generator G outputs for each output, calculate logits_prob for each token in current, old, reference model calcualte objective value as loss update old model in each step update reference model in each epoch Objective function G is amount of outputs in each group for each input O_i is i-th output in current…