Articles tagged with llm-alignment | Vitor Sousa

Vitor Sousa

Articles tagged llm-alignment

Back to blog
After GDPO: A Map of the GRPO Family Part 5 GRPO didn't fragment into a zoo. It exposed a handful of design axes — token vs. sequence granularity, clip width, length and std normalization, KL, advantage normalization, sampling — and every 'new' algorithm is a different point in that space. A practitioner's map of Dr. GRPO, DAPO, GSPO, GDPO, and the frontier, with a decision guide for when to reach for each. ~18 min
GDPO: Multi-Reward RL Done Right Part 4 When GRPO meets multiple rewards, advantages collapse. GDPO fixes this by normalizing each reward independently before combining. Learn why this matters for tool calling, math reasoning, and any multi-objective LLM alignment. ~25 min
GRPO: Eliminating the Value Network Part 3 Group Relative Policy Optimization replaces PPO's learned value function with a simple insight: sample multiple outputs and use their relative rewards as advantages. 33% memory savings, simpler implementation, and the algorithm powering DeepSeek-R1. ~32 min
PPO for Language Models: The RLHF Workhorse Part 2 Deep dive into Proximal Policy Optimization—the algorithm behind most LLM alignment. Understand trust regions, the clipped objective, GAE, and why PPO's four-model architecture creates problems at scale. ~28 min
Reinforcement Learning Foundations for LLM Alignment Part 1 Master the RL fundamentals powering modern LLM training: from MDPs and policy gradients through value functions and actor-critic methods. The mathematical foundations you need before diving into PPO, GRPO, and beyond. ~35 min