LLM Architecture Refresh [8]: Learning from Rewards
What happens when we train a language model using rewards instead of correct answers? In this post, I implement Group Relative Policy Optimization (GRPO) from scratch using Qwen2.5-0.5B-Instruct and LoRA. The model generates multiple solutions to multiplication problems, receives rewards from a programmatic verifier, and learns by comparing its attempts. I train it from three random seeds, and three more times with each group's rewards shuffled, to separate what the reward taught from what the training loop does on its own, and look at how RLHF runs the same loop with a learned judge in place of a program.