Comment on The Little Book of Reinforcement LearningparentComments−porridgeraisin1moGRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.−janalsncm1moIt is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.−porridgeraisin1moYes, but monte carlo estimating the critic model is not new.
Comments
GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.
It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.
Yes, but monte carlo estimating the critic model is not new.