GRPO: Eliminating the Value Network

Part 3 of 5: The Memory-Efficient Alternative

Updated 31 Oct 2026: Fixed the prerequisite link, which pointed off-site. The std-normalization paragraph had the effect backwards: dividing by the group std up-weights prompts that are nearly always or nearly never solved. The training code never exercised the clip, because the ratio was always 1; it now has a $\mu$ loop that reuses each batch. Corrected the memory numbers: about 44% less model state than PPO, not 33%. Also: the std convention, the RLOO view of the baseline, the zero-variance-group limitation, a hedge on the KL estimator, the DPO coefficient, group-size advice, the R1 “aha moment” excerpt and R1’s stage-3 data, and several formulas that rendered wrong.

TL;DR: GRPO (Group Relative Policy Optimization) replaces PPO’s learned value function with a simple insight: sample $G$ outputs per prompt and use their normalized relative rewards as advantages. Dropping the critic removes a whole trainable model — about 44% of PPO’s model state for a 7B model — while staying competitive with PPO, especially with verifiable rewards. The algorithm powers DeepSeekMath’s 51.7% MATH benchmark score and later became central to DeepSeek-R1’s remarkable reasoning capabilities.

Reading time: ~32 minutes

Prerequisites: Part 2: PPO for Language Models covers the clipped objective and GAE that GRPO builds upon. Symbols follow the series notation.

The Memory Problem Revisited

In Part 2, we saw PPO’s four-model architecture. With mixed-precision Adam, a trainable model costs about 16 bytes per parameter (weights, gradients, fp32 master weights, and Adam’s two moments) and a frozen one about 2:

The value network is the most expensive kind of model in the setup — a trainable one — and it’s there for a single purpose: estimating advantages via GAE.

But what if we didn’t need to learn $V(s)$? What if we could estimate advantages directly from samples?

This is GRPO’s insight: instead of learning “how good is this state?”, just sample multiple completions for the same prompt and compare them. The mean reward becomes your baseline. No neural network required.

Table of Contents

1. The Core Insight: Group-Relative Advantages
2. Mathematical Formulation
3. Why This Works
4. Outcome vs. Process Supervision
5. Decoupled KL Regularization
6. Iterative GRPO
7. PyTorch Implementation
8. DeepSeek-R1: GRPO at Scale
9. A Unified View: GRPO, PPO, DPO, and RFT
10. Limitations and When to Use Alternatives
11. Key Takeaways

The Core Insight: Group-Relative Advantages

The Algorithm

For each prompt $q$, GRPO:

1. Sample a group of $G$ outputs $\{o_1, o_2, \ldots, o_G\}$ from the current policy
2. Score each output with the reward model or verifier: $\{r_1, r_2, \ldots, r_G\}$
3. Normalize rewards using group statistics
4. Use normalized rewards as advantages for all tokens in each output

The advantage for output $o_i$ is:

where $\mathbf{r} = \{r_1, \ldots, r_G\}$ are rewards for all outputs in the group.

That’s it. No learned value function. No GAE computation. Just sample, score, normalize.

Visual Comparison

Memory Savings

About 44% less model state for a 7B model. That still doesn’t fit on one 80 GB GPU, so the saving shows up as fewer GPUs, or room for bigger batches and longer generations.

These totals exclude activations, the KV cache and the generation engine. ZeRO/FSDP sharding and offloading change the per-GPU numbers, and a PPO critic can be smaller than the policy, which shrinks the gap.

Key Insight: GRPO trades compute (sampling $G$ outputs per prompt) for memory (no value network). For LLMs where memory is the bottleneck, this is an excellent trade.

Mathematical Formulation

Let’s formalize GRPO precisely.

The GRPO Objective

GRPO maximizes:

The Per-Token Loss

Like PPO, GRPO uses the clipped surrogate objective:

where the importance ratio is:

The clipping mechanism is identical to PPO—what changes is how we compute $\hat{A}_{i,t}$.

Group-Relative Advantage Estimation

For outcome supervision (the standard case), all tokens in output $o_i$ share the same advantage:

where:

• $r_i = r_\phi(q, o_i)$ is the reward for output $o_i$
• $\mathbf{r} = \{r_1, r_2, \ldots, r_G\}$ are rewards for all $G$ outputs
• $\text{mean}(\mathbf{r}) = \frac{1}{G}\sum_{i=1}^G r_i$
• $\text{std}(\mathbf{r}) = \sqrt{\frac{1}{G}\sum_{i=1}^G (r_i - \text{mean}(\mathbf{r}))^2}$ — the population std. PyTorch’s rewards.std() defaults to the sample std (divide by $G-1$); rewards.std(unbiased=False) matches this formula, and the code below uses it.

Properties of Group-Relative Advantages

1. Zero-mean: Within each group, advantages sum to zero — exactly, up to the small $\epsilon_{\text{std}}$ added to the denominator: $\sum_{i=1}^G \tilde{r}_i = 0$

This provides balanced training signal: reinforcing good outputs requires suppressing worse ones.

2. Unit variance: By dividing by std, advantages are normalized: $\text{Var}(\tilde{r}) = 1$ This holds exactly only with the population std, and only when the group’s rewards aren’t all equal (then every advantage is 0).

This makes gradients comparable across prompts with different reward scales.

3. Relative, not absolute: The advantage measures performance relative to the group, not absolute quality. An output with reward 0.5 could have positive advantage (if others scored 0.3) or negative (if others scored 0.7).

Why This Works

The group-relative approach succeeds for several reasons:

1. Natural Baseline

The mean reward $\text{mean}(\mathbf{r})$ estimates the expected reward under the current policy for this prompt:

With $G=64$ samples (as in DeepSeekMath), this estimate is reasonably accurate. This is exactly what we want from a baseline.

One subtlety: the mean includes $r_i$ itself, and a baseline that depends on the sample it’s applied to can in principle bias the gradient. Here it doesn’t change the gradient’s direction. The group-mean advantage is the leave-one-out (RLOO) advantage, scaled by a constant:

The leave-one-out baseline uses only the other samples, so it is a valid, unbiased baseline, and the factor $(G-1)/G$ only rescales the step. The real biases in GRPO come from elsewhere: dividing by the std and by the response length $|o_i|$ (Part 5 covers both).

2. Matched to Reward Model Training

Reward models are typically trained on comparisons between outputs for the same prompt, with a Bradley–Terry model: $P(o_A \succ o_B \mid q) = \sigma\big(r(q, o_A) - r(q, o_B)\big)$. Only differences within a prompt enter that loss, so adding any constant $c(q)$ to every reward for prompt $q$ leaves the model unchanged. A Bradley–Terry reward is defined only up to a per-prompt constant.

Subtracting the group mean removes exactly that constant. GRPO’s advantage uses only the part of the reward the reward model was actually trained to get right.

3. Std Normalization Is a Weighting

Dividing by the group std is not a neutral rescale; it changes how much each prompt counts. Take binary rewards and $G = 16$. A prompt solved half the time has std 0.5, so its advantages are $\pm 1$. A prompt solved once in 16 tries has std about 0.24, and its one success gets an advantage of about $+3.9$.

So dividing by the group std up-weights prompts whose rewards barely vary — those nearly always or nearly never solved. Sometimes that’s what you want (a rare success on a hard prompt gets a loud signal); often it’s a bias toward prompts that carry little information. Dr. GRPO (Part 5) removes it for this reason.

4. No Bootstrapping Bias

GAE uses $V(s_{t+1})$ estimates, introducing bias if the value function is inaccurate. GRPO uses only observed rewards—no bootstrapping, no bias from learned estimates.

5. Simplicity

The entire advantage computation is:

advantages = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + eps)

No recursive GAE computation, no value function forward passes, no TD errors.

Key Insight: The group mean is a Monte Carlo estimate of $V(s_0)$ for this prompt. By sampling enough outputs, we get a good baseline without learning anything.

Outcome vs. Process Supervision

GRPO supports two supervision modes.

Outcome Supervision (Default)

A single reward at sequence end:

All tokens receive the same advantage—the normalized final reward.

Pros:

• Simple: Works with any output-level reward model
• Stable: Single reward signal, no step-level noise

Cons:

• Coarse credit assignment: Every token gets equal “blame” or “credit”

Process Supervision

Rewards at each reasoning step (e.g., separated by newlines):

Let $\text{idx}(j)$ be the token index at the end of step $j$, and $K_i$ the total steps in output $o_i$.

Step 1: Normalize per-step rewards

where $\mathbf{R}$ contains all step rewards across all outputs in the group.

Step 2: Compute return-to-go

The advantage at token $t$ is the sum of normalized rewards from all subsequent steps.

Which to Choose?

DeepSeekMath experiments show process supervision slightly outperforms outcome supervision for math, but the difference is modest. Unless you have a high-quality Process Reward Model (PRM), outcome supervision is the simpler default.

Decoupled KL Regularization

GRPO handles KL penalty differently from standard PPO.

PPO: KL in the Reward

Standard PPO-RLHF adds KL to the per-token reward:

This KL penalty flows into GAE computation, coupling reward signal with regularization.

GRPO: KL in the Loss

GRPO adds KL directly to the loss, decoupled from advantages:

The Schulman KL Estimator

GRPO uses the “k3” KL estimator from Schulman’s note on approximating KL:

Properties:

• Non-negative: Always $\geq 0$
• Zero at equality: Exactly 0 when $\pi_\theta = \pi_{\text{ref}}$
• Unbiased, with caveats: For tokens sampled from $\pi_\theta$, its expected value equals $\mathbb{D}_{\text{KL}}[\pi_\theta \,\|\, \pi_{\text{ref}}]$. In GRPO the tokens come from $\pi_{\theta_{\text{old}}}$, not $\pi_\theta$, and the gradient of this estimator is not the gradient of the KL. Treat it as a well-behaved penalty that tracks the KL, not as an exact KL gradient.

Why Decouple?

1. Cleaner advantage signal: Advantages reflect only task performance, not regularization.

2. Independent tuning: KL coefficient $\beta$ doesn’t interact with advantage normalization.

3. Simpler implementation: No need to track per-token KL during advantage computation.

Key Insight: Decoupling KL from rewards simplifies the algorithm. The policy gradient focuses on task performance; the KL term separately ensures we don’t drift too far from the reference.

Iterative GRPO

As training progresses, the reward model may become a poor discriminator for the improved policy’s outputs. Iterative GRPO addresses this.

The Staleness Problem

1. Initially: Policy outputs vary in quality; reward model discriminates well
2. After training: Policy consistently produces good outputs; reward model’s training distribution no longer matches
3. Result: Degraded training signal

The Solution: Co-evolve Policy and Reward Model

Algorithm: Iterative GRPO

For iteration = 1, ..., I:
    
    ┌─────────────────────────────────────────────────────┐
    │ Phase 1: Reset Reference                            │
    └─────────────────────────────────────────────────────┘
    π_ref ← π_θ   (current policy becomes new reference)
    
    ┌─────────────────────────────────────────────────────┐
    │ Phase 2: Policy Training                            │
    └─────────────────────────────────────────────────────┘
    For step = 1, ..., M:
        Sample batch of prompts
        Sample G outputs per prompt from π_θ
        Compute rewards using r_φ
        Compute group-relative advantages
        Update π_θ using GRPO objective
    
    ┌─────────────────────────────────────────────────────┐
    │ Phase 3: Reward Model Update                        │
    └─────────────────────────────────────────────────────┘
    Generate comparison data from improved π_θ
    Continually train r_φ with:
        - 90% new data (from current policy)
        - 10% historical data (replay buffer)

Key Design Choices

Reference reset: Each iteration starts fresh, allowing larger cumulative policy drift while constraining per-iteration change.

Replay buffer: 10% historical data prevents the reward model from forgetting how to score earlier, easier outputs.

Single GRPO update: DeepSeekMath uses one gradient step per sample batch ($\mu = 1$), prioritizing fresh samples over reuse.

PyTorch Implementation

Here’s a complete, educational implementation of GRPO:

"""
Group Relative Policy Optimization (GRPO)
=========================================

Memory-efficient alternative to PPO that eliminates the value network
by computing advantages from groups of sampled outputs.

Reference: DeepSeekMath (Shao et al., 2024)
"""

import torch
import torch.nn.functional as F
from torch import Tensor
from dataclasses import dataclass
from typing import Tuple, List


@dataclass
class GRPOConfig:
    """GRPO hyperparameters."""
    
    # Core settings
    group_size: int = 16           # G: outputs per prompt (DeepSeekMath used 64)
    clip_epsilon: float = 0.2      # ε: PPO clipping
    kl_coef: float = 0.04          # β: KL penalty coefficient
    mu: int = 1                    # μ: gradient steps per sampled batch. DeepSeekMath
                                   # used 1; then the ratio is exactly 1 and the clip
                                   # never fires. Set μ > 1 to reuse each batch.
    
    # Generation
    max_length: int = 1024
    temperature: float = 1.0
    
    # Training
    learning_rate: float = 1e-6
    max_grad_norm: float = 1.0
    
    # Numerical stability
    eps: float = 1e-8


def compute_group_advantages(rewards: Tensor, eps: float = 1e-8) -> Tensor:
    """
    Compute group-relative advantages.
    
    This is GRPO's key innovation: use group statistics
    instead of a learned value function.
    
    Args:
        rewards: [G] rewards for each output in the group
        eps: Small constant for numerical stability
        
    Returns:
        advantages: [G] normalized advantages
    """
    mean_r = rewards.mean()
    std_r = rewards.std(unbiased=False) + eps   # population std, as in the formula
    advantages = (rewards - mean_r) / std_r
    return advantages


def compute_schulman_kl(
    policy_log_probs: Tensor,
    reference_log_probs: Tensor,
) -> Tensor:
    """
    Compute Schulman's k3 KL estimator.
    
    D_KL = π_ref/π_θ - log(π_ref/π_θ) - 1
    
    This estimator is:
    - Non-negative (always >= 0)
    - Unbiased in value for tokens sampled from π_θ (here they come
      from π_old, and its gradient is not the KL's gradient)
    - Zero when π_θ = π_ref
    
    Args:
        policy_log_probs: [batch, seq_len] from π_θ
        reference_log_probs: [batch, seq_len] from π_ref
        
    Returns:
        kl: [batch, seq_len] per-token KL
    """
    log_ratio = reference_log_probs - policy_log_probs  # log(π_ref/π_θ)
    kl = torch.exp(log_ratio) - log_ratio - 1
    return kl


def compute_grpo_loss(
    policy_log_probs: Tensor,
    old_log_probs: Tensor,
    reference_log_probs: Tensor,
    advantages: Tensor,
    response_mask: Tensor,
    clip_epsilon: float = 0.2,
    kl_coef: float = 0.04,
) -> Tuple[Tensor, dict]:
    """
    Compute GRPO loss with clipping and KL penalty.
    
    Args:
        policy_log_probs: [G, seq_len] current policy
        old_log_probs: [G, seq_len] policy at sampling time
        reference_log_probs: [G, seq_len] reference policy
        advantages: [G] per-output advantages (will broadcast)
        response_mask: [G, seq_len] valid token mask
        clip_epsilon: PPO clipping parameter
        kl_coef: KL penalty coefficient
        
    Returns:
        loss: Scalar loss to minimize
        metrics: Dictionary of training metrics
    """
    G, seq_len = policy_log_probs.shape
    
    # =========================================
    # 1. Importance ratio: ρ = π_θ / π_old
    # =========================================
    log_ratio = policy_log_probs - old_log_probs
    ratio = torch.exp(log_ratio)  # [G, seq_len]
    
    # =========================================
    # 2. Expand advantages to token level
    # =========================================
    # [G] -> [G, seq_len]
    token_advantages = advantages.unsqueeze(1).expand(-1, seq_len)
    
    # =========================================
    # 3. PPO clipped objective
    # =========================================
    clipped_ratio = torch.clamp(ratio, 1 - clip_epsilon, 1 + clip_epsilon)
    
    policy_loss_1 = ratio * token_advantages
    policy_loss_2 = clipped_ratio * token_advantages
    policy_loss = torch.min(policy_loss_1, policy_loss_2)
    
    # =========================================
    # 4. KL penalty (decoupled from reward)
    # =========================================
    kl_penalty = compute_schulman_kl(policy_log_probs, reference_log_probs)
    
    # =========================================
    # 5. Combine losses
    # =========================================
    # Maximize policy objective, minimize KL
    token_loss = -policy_loss + kl_coef * kl_penalty
    
    # Mask and average
    masked_loss = token_loss * response_mask
    per_output_loss = masked_loss.sum(dim=1) / response_mask.sum(dim=1).clamp(min=1)
    loss = per_output_loss.mean()
    
    # =========================================
    # 6. Metrics
    # =========================================
    with torch.no_grad():
        clip_fraction = ((ratio - 1).abs() > clip_epsilon).float()
        clip_fraction = (clip_fraction * response_mask).sum() / response_mask.sum()
        
        mean_kl = (kl_penalty * response_mask).sum() / response_mask.sum()
    
    metrics = {
        "loss": loss.item(),
        "clip_fraction": clip_fraction.item(),
        "mean_kl": mean_kl.item(),
        "mean_advantage": advantages.mean().item(),
        "advantage_std": advantages.std().item(),
    }
    
    return loss, metrics


class GRPOTrainer:
    """
    GRPO trainer for LLM fine-tuning.
    
    Unlike PPO, requires only THREE models (no value network):
    - policy_model: The LLM being trained
    - reference_model: Frozen copy for KL penalty
    - reward_model: Scores complete responses
    
    Memory: ~44% less model state than PPO with a learned reward model
    """
    
    def __init__(
        self,
        policy_model: torch.nn.Module,
        reference_model: torch.nn.Module,
        reward_model,  # Can be a function or model
        tokenizer,
        config: GRPOConfig,
    ):
        self.policy = policy_model
        self.reference = reference_model
        self.reward_fn = reward_model
        self.tokenizer = tokenizer
        self.config = config
        
        # Freeze reference model
        for param in self.reference.parameters():
            param.requires_grad = False
        
        # Single optimizer (no value network!)
        self.optimizer = torch.optim.AdamW(
            self.policy.parameters(),
            lr=config.learning_rate,
            betas=(0.9, 0.95),
            weight_decay=0.1
        )
    
    @torch.no_grad()
    def sample_group(
        self,
        prompt_ids: Tensor,
        attention_mask: Tensor,
    ) -> Tuple[Tensor, Tensor]:
        """
        Sample G outputs for a single prompt.
        
        Args:
            prompt_ids: [1, prompt_len] token ids
            attention_mask: [1, prompt_len]
            
        Returns:
            output_ids: [G, total_len] generated sequences
            output_masks: [G, total_len] attention masks
        """
        G = self.config.group_size
        device = prompt_ids.device
        
        # Expand prompt for batch generation
        prompt_ids = prompt_ids.expand(G, -1)
        attention_mask = attention_mask.expand(G, -1)
        
        # Sample from policy
        self.policy.eval()
        outputs = self.policy.generate(
            input_ids=prompt_ids,
            attention_mask=attention_mask,
            max_new_tokens=self.config.max_length,
            do_sample=True,
            temperature=self.config.temperature,
            pad_token_id=self.tokenizer.pad_token_id,
            return_dict_in_generate=True,
        )
        
        output_ids = outputs.sequences
        output_masks = (output_ids != self.tokenizer.pad_token_id).long()
        
        return output_ids, output_masks
    
    @torch.no_grad()
    def compute_rewards(
        self,
        prompt_text: str,
        output_ids: Tensor,
        prompt_len: int,
    ) -> Tensor:
        """
        Compute rewards for each output in the group.
        
        Args:
            prompt_text: The original prompt string
            output_ids: [G, total_len] generated sequences
            prompt_len: Length of prompt in tokens
            
        Returns:
            rewards: [G] scalar reward per output
        """
        G = output_ids.shape[0]
        rewards = []
        
        for i in range(G):
            # Extract response tokens
            response_ids = output_ids[i, prompt_len:]
            response_text = self.tokenizer.decode(
                response_ids, skip_special_tokens=True
            )
            
            # Score with reward model
            reward = self.reward_fn(prompt_text, response_text)
            rewards.append(reward)
        
        return torch.tensor(rewards, device=output_ids.device)
    
    def compute_log_probs(
        self,
        model: torch.nn.Module,
        input_ids: Tensor,
        attention_mask: Tensor,
        response_start_idx: int,
    ) -> Tensor:
        """
        Compute per-token log probabilities for response tokens.
        
        Args:
            model: Language model
            input_ids: [batch, seq_len] full sequences
            attention_mask: [batch, seq_len]
            response_start_idx: Where response tokens begin
            
        Returns:
            log_probs: [batch, response_len] per-token log probs
        """
        outputs = model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            return_dict=True,
        )
        logits = outputs.logits  # [batch, seq_len, vocab]
        
        # Shift for next-token prediction
        shift_logits = logits[:, :-1, :]
        shift_labels = input_ids[:, 1:]
        
        # Log probabilities
        log_probs = F.log_softmax(shift_logits, dim=-1)
        
        # Gather log probs for actual tokens
        token_log_probs = log_probs.gather(
            dim=-1,
            index=shift_labels.unsqueeze(-1)
        ).squeeze(-1)
        
        # Extract response tokens only
        response_log_probs = token_log_probs[:, response_start_idx - 1:]
        
        return response_log_probs
    
    def train_step(
        self,
        prompt_text: str,
        prompt_ids: Tensor,
        attention_mask: Tensor,
    ) -> dict:
        """
        Single GRPO training step.
        
        Args:
            prompt_text: Original prompt string
            prompt_ids: [1, prompt_len] tokenized prompt
            attention_mask: [1, prompt_len]
            
        Returns:
            metrics: Training metrics dictionary
        """
        prompt_len = prompt_ids.shape[1]
        device = prompt_ids.device
        
        # =============================================
        # Step 1: Sample group of outputs
        # =============================================
        output_ids, output_masks = self.sample_group(prompt_ids, attention_mask)
        G = output_ids.shape[0]
        
        # =============================================
        # Step 2: Compute rewards
        # =============================================
        rewards = self.compute_rewards(prompt_text, output_ids, prompt_len)
        
        # =============================================
        # Step 3: Compute group-relative advantages
        # =============================================
        advantages = compute_group_advantages(rewards, self.config.eps)
        
        # =============================================
        # Step 4: Freeze π_old and π_ref log probs for this batch
        # =============================================
        with torch.no_grad():
            old_log_probs = self.compute_log_probs(
                self.policy, output_ids, output_masks, prompt_len
            )
            reference_log_probs = self.compute_log_probs(
                self.reference, output_ids, output_masks, prompt_len
            )
        
        # Response mask
        response_mask = output_masks[:, prompt_len:]
        
        # =============================================
        # Step 5: μ gradient steps on the same batch
        # =============================================
        # Advantages and old_log_probs stay fixed. From the second step on,
        # π_θ has moved, so the ratio leaves 1 and the clip starts to matter.
        self.policy.train()
        for _ in range(self.config.mu):
            policy_log_probs = self.compute_log_probs(
                self.policy, output_ids, output_masks, prompt_len
            )
            loss, metrics = compute_grpo_loss(
                policy_log_probs,
                old_log_probs,
                reference_log_probs,
                advantages,
                response_mask,
                self.config.clip_epsilon,
                self.config.kl_coef,
            )
            self.optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(
                self.policy.parameters(), 
                self.config.max_grad_norm
            )
            self.optimizer.step()
        
        # Add reward metrics
        metrics["mean_reward"] = rewards.mean().item()
        metrics["reward_std"] = rewards.std().item()
        
        return metrics

Implementation Notes

Group size selection: DeepSeekMath used $G=64$; $G = 8$–$16$ is common. Larger groups give a better baseline per prompt, but small groups work when each batch has enough prompts.

Reusing a batch: With $\mu = 1$ (DeepSeekMath’s setting) the policy hasn’t moved when the loss is computed, so the ratio is exactly 1 and the clip never fires; GRPO is then a plain policy gradient with a KL penalty. With $\mu > 1$, advantages stay fixed for the batch; the importance ratio and clip correct for the policy drifting between steps — that is what they are for.

Dropout: Compute old_log_probs and the training-step log probs with the same dropout setting (most RLHF setups disable dropout), or the ratio won’t be 1 even on the first step.

KL coefficient: Start with $\beta = 0.04$. Lower values allow more divergence; higher values constrain updates.

DeepSeek-R1: GRPO at Scale

In January 2025, DeepSeek released DeepSeek-R1, a reasoning model matching OpenAI’s o1 on many benchmarks. GRPO is central to its training.

R1’s Training Pipeline

Why GRPO for R1?

1. Long reasoning traces: R1 produces outputs with thousands of tokens. A value network would need to predict final reward from any point in this long sequence—extremely difficult.

2. Verifiable rewards: For math and code, rewards are rule-based (correct/incorrect). No reward model uncertainty.

3. Memory efficiency: At R1’s scale, GRPO’s memory savings are crucial.

4. Training stability: Simpler architecture means fewer things can go wrong.

The “Aha Moment” Phenomenon

The R1 paper reports an “aha moment” in an intermediate checkpoint of R1-Zero (the RL-only model, below): mid-solution, the model stops, flags its own approach, and re-evaluates. The paper’s example, abridged (the ellipses are the paper’s):

Question: If a > 1, then the sum of the real solutions of
√(a − √(a + x)) = x is equal to

Response: <think>
To solve the equation √(a − √(a + x)) = x, let's start by squaring both ···
(√(a − √(a + x)))² = x² ⟹ a − √(a + x) = x².
Rearrange to isolate the inner square root term:
(a − x²)² = a + x ⟹ a² − 2ax² + (x²)² = a + x ⟹ x⁴ − 2ax² − x + (a² − a) = 0
···
Wait, wait. Wait. That's an aha moment I can flag here.
Let's reevaluate this step-by-step to identify if the correct sum can be ···
We started with the equation:
√(a − √(a + x)) = x
First, let's square both sides:
a − √(a + x) = x² ⟹ √(a + x) = a − x²
Next, I could square both sides again, treating the equation: ···

No one supervised this behaviour directly; only the correctness reward did. How much of it RL creates is contested: the Dr. GRPO paper reports that self-reflection of this kind already appears in base models before RL, so RL may amplify it more than invent it.

R1-Zero: Pure RL Without SFT

DeepSeek also released R1-Zero, trained with GRPO directly from the base model without any SFT stage. Remarkably, it still develops coherent reasoning, though with some readability issues that the full R1 pipeline addresses.

This demonstrates GRPO’s power: given only a correctness signal, the algorithm can induce sophisticated reasoning behavior.

Key Insight: GRPO’s simplicity makes it stable enough for extreme-scale training. DeepSeek-R1 shows that you can train world-class reasoning models without the complexity of PPO’s value network.

A Unified View: GRPO, PPO, DPO, and RFT

The DeepSeekMath paper provides a unified framework for understanding different alignment methods (its Section 5.2).

The General Policy Gradient

All methods can be viewed through:

Methods differ in:

1. Data source $\mathcal{D}$: Where do samples come from?
2. Gradient coefficient $w(q, o)$: How are outputs weighted?

Method Comparison

For DPO, $\ell^\pm = \log \pi_\theta(o^\pm \mid q) / \pi_{\text{ref}}(o^\pm \mid q)$ is the log-ratio of the chosen ($o^+$) or rejected ($o^-$) response. The chosen response is pushed up, and the rejected one down, by the same weight: large when the model still prefers the rejected one, near zero once it clearly prefers the chosen one.

Key Insights

Online vs. Offline: GRPO, PPO, and Online RFT sample from the current policy and can adapt. DPO and RFT use fixed datasets.

Weighted vs. Filtered: RFT keeps only correct outputs (binary filter). GRPO keeps all outputs but weights by normalized reward (continuous weighting).

Advantage estimation: PPO learns $V(s)$; GRPO uses group statistics; DPO implicitly computes advantages through preferences.

Limitations and When to Use Alternatives

GRPO Limitations

1. Sampling cost: $G$ outputs per prompt (commonly 8–16; DeepSeekMath used 64). More generation per prompt than PPO’s single sample, though no critic forward and backward passes.

2. Advantage variance: With small groups, the mean is a noisy baseline. In practice $G = 8$–$16$ is common, and small groups work with enough prompts per batch: the GDPO paper trained with 4 rollouts (tool calling) and 16 (maths), and 2-GRPO, with just two completions per prompt, was reported to match 16-GRPO at about 1/8 the cost (overview).

3. Zero-variance groups: If every completion in a group is right, or every one is wrong, every advantage is zero and the group contributes no gradient. With binary rewards this is the biggest practical issue: as the model improves, more prompts become all-correct and the useful batch shrinks. DAPO’s dynamic sampling (Part 5) filters these groups out and samples more.

4. Uniform token credit: All tokens get the same advantage. No fine-grained credit assignment without process supervision.

5. Reward model dependency: Like all RL methods, susceptible to reward hacking if the reward model has exploits.

6. No value function benefits: Can’t use value-guided search or intrinsic motivation bonuses.

When to Use Each Method

Key Takeaways

The Core Innovation

Sample multiple outputs, normalize rewards, use as advantages. No value network.

Memory Savings

• Eliminates a trainable value network: ~112 GB of model state for a 7B model with Adam
• About 44% less model state than PPO (252 GB → 140 GB with a learned reward model; 126 GB with a verifier)
• Excludes activations, KV cache and the generation engine; sharding changes per-GPU numbers

Design Choices

• Group size: $G = 8$–$16$ is common (DeepSeekMath used 64)
• Std: population std in the formula; std(unbiased=False) in code
• Clip epsilon: $\varepsilon = 0.2$ (standard PPO)
• KL coefficient: $\beta = 0.04$
• Decoupled KL: In loss, not reward

When GRPO Excels

• Memory-constrained training
• Outcome-based rewards
• Long generations (where value prediction is hard)
• Verifiable rewards (math, code)

DeepSeek-R1 Connection

GRPO powers R1’s remarkable reasoning through:

• Rule-based correctness rewards
• Self-reflection that RL amplified (“aha moments”), which the Dr. GRPO paper finds already present in base models
• Stable training at scale

What’s Next

In Part 4: GDPO, we’ll see what happens when GRPO meets multi-reward settings:

• Why combined reward normalization causes “advantage collapse”
• GDPO’s solution: decouple normalization per reward
• Production patterns for tool calling, math, and coding

Further Reading

Original Papers:

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning (Shao et al., 2024)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025)

Related Work:

• Proximal Policy Optimization (Schulman et al., 2017)
• Direct Preference Optimization (Rafailov et al., 2023)

Implementations:

• TRL: Transformer Reinforcement Learning
• VERL
• OpenRLHF

@misc{sousa2026grpo,
  title={{GRPO: Eliminating the Value Network}},
  author={Sousa, Vitor},
  year={2026},
  howpublished={\url{https://www.vitorsousa.com/blog/grpo/}},
  url={https://www.vitorsousa.com/blog/grpo/}
}