Comment on The Little Book of Reinforcement LearningComments−janalsncm1moI wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.−porridgeraisin1moGRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.−janalsncm1moIt is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.−porridgeraisin1moYes, but monte carlo estimating the critic model is not new.
Comments
I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.
GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.
It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.
Yes, but monte carlo estimating the critic model is not new.