Vitor Sousa

Search

Try a topic

↑ ↓ Navigate Enter OpenEsc Close

All series

Series · 5 parts

Policy Optimization for LLMs: From Fundamentals to Production

From the mathematical foundations of reinforcement learning to the algorithms used to align large language models: why each method exists, what problem it addresses, and how the field moved from PPO to GRPO to GDPO — then a map of the GRPO variants.

5 parts · ~138 min total reading time · Last updated Oct 2026

All five articles are published. Read in order, or use the prerequisites below to enter midway.

Prerequisites: probability, calculus, and transformer fundamentals. Related code: Learning with Verifiable Rewards — model and training foundations complete; the study is paused before its SFT run, with GRPO and ablations planned.

  1. Part 1: Reinforcement Learning Foundations for LLM Alignment

    MDPs, policy gradients, value functions, and actor-critic methods: the building blocks that LLM alignment methods rely on.

    ~35 min MDP · policy gradient · GAE · actor-critic

  2. Part 2: PPO for Language Models: The RLHF Workhorse

    The clipped objective, trust regions, and why PPO's four-model architecture creates problems at scale.

    ~28 min PPO · clipping · KL penalty · trust region

  3. Part 3: GRPO: Eliminating the Value Network

    Replace the value network with group-relative advantages: about 44% less model memory than PPO, with competitive performance.

    ~32 min GRPO · group sampling · DeepSeek-R1

  4. Part 4: GDPO: Multi-Reward RL Done Right

    When multiple rewards collapse advantages, GDPO normalizes each independently for stable multi-objective optimization.

    ~25 min GDPO · multi-reward · advantage collapse

  5. Part 5: After GDPO: A Map of the GRPO Family

    GRPO did not fragment into a zoo; it exposed a handful of design axes. Dr. GRPO, DAPO, GSPO, and GDPO mapped onto them, with a guide to choosing between them.

    ~18 min DAPO · GSPO · Dr. GRPO · decision guide

Start with part 1