GDPO: Multi-Reward RL Done Right

Part 4 of 5: The Multi-Reward Frontier

Updated 31 Oct 2026: Reworked. The old four-output example didn’t show what GDPO changes: in that group both methods give the same advantages. It is replaced with the paper’s two-rollout example and a “reweighting” view of what GDPO computes. Removed the claim that GDPO gives each reward its own gradient — it doesn’t. The results tables now use the paper’s actual setups and numbers. Added the std-ablation result, the paper’s advice to condition rewards rather than weight them, and a note that the final normalisation is batch-wide. Fixed the prerequisite link and several formulas.

TL;DR: GRPO works well for single rewards, but with multiple rewards it sees only their sum. Normalising the sum maps groups with different reward profiles to identical advantages: a rollout that satisfies both rewards can get the same signal as one that satisfies only one. GDPO (Group reward-Decoupled normalization Policy Optimization) normalizes each reward within the group first, then combines. In effect it reweights each reward by how little it varies, so a rarely-met objective isn’t drowned out. It’s a small change that gained a few points on tool calling, maths and coding in the paper.

Reading time: ~25 minutes

Prerequisites: Part 3: GRPO covers group-relative advantage estimation. Symbols follow the series notation; this post uses the sample std, to match the paper’s numbers.

The Multi-Reward Reality

Modern LLM training rarely optimizes a single reward. Consider these common scenarios:

Tool Calling:

• Format reward: Does the output follow the required JSON schema?
• Correctness reward: Are the tool calls semantically correct?

Math Reasoning:

• Correctness reward: Is the final answer mathematically correct?
• Length reward: Does the response stay within the token budget?

Coding:

• Pass-rate reward: What fraction of the test cases pass?
• Length reward: Does the response stay within the token budget?
• Bug reward: How few bugs does the code have?

These three are the settings the GDPO paper trains on.

The naive approach: sum the rewards and apply GRPO. This doesn’t work well.

Table of Contents

1. The Advantage Collapse Problem
2. Mathematical Analysis
3. GDPO: The Solution
4. Implementation
5. Experimental Results
6. When to Use GDPO vs GRPO
7. Production Patterns
8. Key Takeaways

The Advantage Collapse Problem

GRPO with Multiple Rewards: The Naive Approach

Suppose we have $K$ rewards, say format and correctness. The naive approach:

1. Combine rewards: $r_i = \sum_k w_k \, r_i^{(k)}$
2. Apply GRPO normalization: $\hat{A}_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$

This is exactly what happens if you use standard GRPO with a multi-reward setup.

The Collapse

The GDPO paper’s own example is the clearest. Take two rollouts per prompt and two binary rewards, so each rollout’s total is 0, 1 or 2. Normalise within the group, using the sample std as the paper does, and before any batch-level normalisation:

Compare the first two rows. In (0, 1) the better rollout satisfies one reward; in (0, 2) it satisfies both. GRPO gives them the same signal, ±0.71: with only the totals to go on, normalisation erases how far apart the two rollouts are. That is advantage collapse — groups with different reward profiles producing identical advantages.

GDPO normalises each reward separately before combining (next section), and gives the rollout that satisfies both rewards twice the signal. The GDPO column assumes the natural split of each total: in (0, 2), the second rollout satisfies both rewards and the first neither.

Why This Matters

1. Lost resolution: The advantage stops reflecting how many objectives an output satisfies. A rollout that gets everything right is treated like one that gets half of it right.
2. Weaker signal for the hard objective: When one reward is rarely satisfied, its successes are folded into a total dominated by the others, and the policy gets little push toward it.

Key Insight: Summing rewards before normalising throws away how many objectives each output satisfied. GRPO sees only the totals, and normalisation then maps very different groups to the same advantages.

Mathematical Analysis

Let’s formalize why advantage collapse happens.

GRPO’s Combined Normalization

Given $K$ reward functions with weights $\{w_k\}$, GRPO computes:

where $\mu_r = \text{mean}(\mathbf{r})$ and $\sigma_r = \text{std}(\mathbf{r})$.

The Collapse Condition

GRPO’s advantages depend on the rewards only through the totals $r_i$, and after normalisation only through their standardised pattern. Any two groups whose totals standardise to the same pattern get identical advantages, whatever the individual rewards were. With two rollouts this is extreme: every group with two different totals gives $(-0.71, 0.71)$.

The paper makes this concrete by counting the distinct advantage values each method can produce as the number of rewards and rollouts grows. GDPO’s count grows much faster than GRPO’s (its Fig. 3). That is a statement about how finely the advantage can resolve different outcomes.

Information Loss

Consider the space of possible reward vectors in $\mathbb{R}^K$ (K rewards). GRPO projects this to $\mathbb{R}^1$ (the weighted sum), then normalizes.

Information lost: All reward vectors mapping to the same sum become indistinguishable.

For $K=2$ rewards with equal weights, the level sets are lines in 2D:

• $(0, 2)$ and $(2, 0)$ and $(1, 1)$ all map to sum $= 2$
• These represent very different behaviors, but get identical treatment

GDPO doesn’t escape the projection: it still ends with one scalar per output. What it changes is which projection, chosen per group, as the next section shows.

A Constant Reward Carries No Signal — in Any Method

If a reward is constant within a group (say every output follows the format), it contributes nothing to that group’s advantages. GRPO subtracts it along with the mean; GDPO’s per-reward normalisation sends it to zero. This is true of every group-relative method, not a GRPO failure that GDPO fixes. If a reward is constant everywhere, the fix is in the data or the reward, not the normalisation.

GDPO: The Solution

Group reward-Decoupled normalization Policy Optimization (GDPO) normalizes each reward independently before combining. The policy update is GRPO’s clipped surrogate, unchanged, with the importance ratio $\rho_{i,t}$ (the paper writes it $s_{i,t}$; this series uses $\rho$ throughout).

The GDPO Algorithm

For each reward $k \in \{1, \ldots, K\}$:

Step 1: Normalize each reward independently, within each group

where $\mu^{(k)} = \text{mean}(\mathbf{r}^{(k)})$ and $\sigma^{(k)} = \text{std}(\mathbf{r}^{(k)})$ over the group’s $G$ rollouts.

Step 2: Combine normalized advantages

Step 3: Final batch normalization

Here $\mu_{\text{pre}}$ and $\sigma_{\text{pre}}$ are taken across the whole batch — every rollout of every prompt — not within one group. This keeps the advantage scale stable as $K$ grows; the paper reports that removing this step occasionally caused convergence failures.

What GDPO Actually Computes

Within a group, step 1 is an affine map of each reward, and step 3 is one more affine map shared by every output in the batch. So, up to that shared final scaling, GDPO is GRPO run on a reweighted sum, where each reward’s weight is divided by its own group std:

with $\mu_k$, $\sigma_k$ reward $k$‘s mean and std in this group. GRPO, by contrast, uses the raw weights $w_k$ and divides the whole sum by one std. Two consequences follow:

• Rewards that rarely vary in a group are amplified. A reward that one rollout in 16 satisfies has a small $\sigma_k$, so in that group it gets a large effective weight. A rarely-met objective isn’t drowned out by an easy one.
• Satisfying more objectives at once earns more signal. Each reward contributes its own standardised term, and they add: in the (0, 2) group, two terms of ±0.71 make ±1.41.

What GDPO does not do is tell the model which objective failed. Each output still gets a single scalar advantage, applied to every token. Take equal weights and complementary binary rewards, $r^{(2)} = 1 - r^{(1)}$ — a format-only output against a correctness-only one. GRPO sees equal totals and gives zero. GDPO normalises the two rewards to exact opposites, they cancel, and it gives zero too.

The two methods also coincide on symmetric groups. In a four-output group containing every combination — (1, 0), (0, 1), (1, 1), (0, 0) — both rewards have the same mean and std, and GDPO’s advantages equal GRPO’s: (0, 0, 1.41, −1.41). GDPO differs where the rewards vary by different amounts, which is the usual case in training.

Visual Comparison

Key Insight: GDPO doesn’t keep the rewards separate in the gradient. It reweights them, per group, by how little each one varies. That is enough to stop a rarely-satisfied objective from being drowned out, and to give more signal to outputs that satisfy several objectives at once.

Implementation

GDPO is a straightforward modification to GRPO. Here’s the key difference:

GRPO (Original)

def grpo_advantages(rewards_per_func, reward_weights, num_generations):
    """
    GRPO: Normalize combined rewards.
    
    Args:
        rewards_per_func: [batch, K] rewards from K reward functions
        reward_weights: [K] weights for each reward
        num_generations: G, outputs per prompt
    """
    # Combine rewards first
    rewards = (rewards_per_func * reward_weights).sum(dim=1)  # [batch]
    
    # Then normalize
    mean_grouped = rewards.view(-1, num_generations).mean(dim=1)
    std_grouped = rewards.view(-1, num_generations).std(dim=1)
    
    mean_grouped = mean_grouped.repeat_interleave(num_generations)
    std_grouped = std_grouped.repeat_interleave(num_generations)
    
    advantages = (rewards - mean_grouped) / (std_grouped + 1e-4)
    
    return advantages

GDPO (Decoupled)

def gdpo_advantages(rewards_per_func, reward_weights, num_generations):
    """
    GDPO: Normalize each reward independently, then combine.
    
    Args:
        rewards_per_func: [batch, K] rewards from K reward functions
        reward_weights: [K] weights for each reward
        num_generations: G, outputs per prompt
    """
    K = len(reward_weights)
    device = rewards_per_func.device
    
    # Handle NaN values
    rewards_per_func = torch.nan_to_num(rewards_per_func)
    
    all_reward_advantages = []
    
    # Step 1: Normalize each reward independently
    for k in range(K):
        reward_k = rewards_per_func[:, k]  # [batch]
        
        # Group-wise normalization for this reward
        mean_k = reward_k.view(-1, num_generations).mean(dim=1)
        std_k = reward_k.view(-1, num_generations).std(dim=1)
        
        mean_k = mean_k.repeat_interleave(num_generations)
        std_k = std_k.repeat_interleave(num_generations)
        
        advantage_k = (reward_k - mean_k) / (std_k + 1e-4)
        all_reward_advantages.append(advantage_k)
    
    # Step 2: Combine normalized advantages
    combined_advantages = torch.stack(all_reward_advantages, dim=1)  # [batch, K]
    pre_bn_advantages = (combined_advantages * reward_weights).sum(dim=1)  # [batch]
    
    # Step 3: Final batch normalization
    bn_mean = pre_bn_advantages.mean()
    bn_std = pre_bn_advantages.std()
    
    advantages = (pre_bn_advantages - bn_mean) / (bn_std + 1e-4)
    
    return advantages

Complete GDPO Trainer

"""
Group reward-Decoupled normalization Policy Optimization (GDPO)
===============================================================

Extends GRPO for multi-reward settings by normalizing each
reward independently before combining.

Reference: NVIDIA (Liu et al., 2026)
"""

import torch
import torch.nn.functional as F
from torch import Tensor
from dataclasses import dataclass
from typing import List, Tuple, Callable


@dataclass
class GDPOConfig:
    """GDPO hyperparameters."""
    group_size: int = 16
    clip_epsilon: float = 0.2
    kl_coef: float = 0.04
    learning_rate: float = 1e-6
    max_grad_norm: float = 1.0
    eps: float = 1e-4


def compute_gdpo_advantages(
    rewards_list: List[Tensor],
    reward_weights: List[float],
    eps: float = 1e-4,
) -> Tuple[Tensor, List[Tensor]]:
    """
    Compute GDPO advantages with decoupled normalization.
    
    Args:
        rewards_list: List of [G] tensors, one per reward function
        reward_weights: Weights for each reward
        eps: Numerical stability constant
        
    Returns:
        advantages: [G] final normalized advantages
        per_reward_advantages: List of [G] tensors for logging
    """
    K = len(rewards_list)
    G = rewards_list[0].shape[0]
    device = rewards_list[0].device
    
    # Step 1: Normalize each reward independently
    normalized_rewards = []
    for k in range(K):
        r_k = rewards_list[k]
        mean_k = r_k.mean()
        std_k = r_k.std() + eps
        normalized_k = (r_k - mean_k) / std_k
        normalized_rewards.append(normalized_k)
    
    # Step 2: Combine with weights
    combined = torch.zeros(G, device=device)
    for k in range(K):
        combined += reward_weights[k] * normalized_rewards[k]
    
    # Step 3: Final normalization. The paper normalizes across the whole
    # batch, every rollout of every prompt. This trainer sees one group per
    # step, so here it is within-group. To match the paper, collect several
    # prompts per step and use gdpo_advantages() above.
    final_mean = combined.mean()
    final_std = combined.std() + eps
    advantages = (combined - final_mean) / final_std
    
    return advantages, normalized_rewards


class GDPOTrainer:
    """
    GDPO trainer for multi-reward LLM alignment.
    
    Key difference from GRPO: each reward is normalized
    independently before combining, preserving signal resolution.
    """
    
    def __init__(
        self,
        policy_model: torch.nn.Module,
        reference_model: torch.nn.Module,
        reward_functions: List[Callable],
        reward_weights: List[float],
        tokenizer,
        config: GDPOConfig,
    ):
        self.policy = policy_model
        self.reference = reference_model
        self.reward_fns = reward_functions
        self.reward_weights = torch.tensor(reward_weights)
        self.tokenizer = tokenizer
        self.config = config
        
        assert len(reward_functions) == len(reward_weights), \
            "Must have same number of reward functions and weights"
        
        # Freeze reference
        for param in self.reference.parameters():
            param.requires_grad = False
        
        self.optimizer = torch.optim.AdamW(
            self.policy.parameters(),
            lr=config.learning_rate,
        )
    
    @torch.no_grad()
    def sample_and_score(
        self,
        prompt_text: str,
        prompt_ids: Tensor,
        attention_mask: Tensor,
    ) -> Tuple[Tensor, List[Tensor], Tensor]:
        """
        Sample G outputs and compute all rewards.
        
        Returns:
            output_ids: [G, seq_len] generated sequences
            rewards_list: List of [G] tensors, one per reward function
            output_masks: [G, seq_len] attention masks
        """
        G = self.config.group_size
        prompt_len = prompt_ids.shape[1]
        
        # Expand for batch generation
        prompt_ids = prompt_ids.expand(G, -1)
        attention_mask = attention_mask.expand(G, -1)
        
        # Sample
        self.policy.eval()
        outputs = self.policy.generate(
            input_ids=prompt_ids,
            attention_mask=attention_mask,
            max_new_tokens=512,
            do_sample=True,
            temperature=1.0,
            pad_token_id=self.tokenizer.pad_token_id,
        )
        
        output_ids = outputs
        output_masks = (output_ids != self.tokenizer.pad_token_id).long()
        
        # Score with each reward function
        rewards_list = []
        for reward_fn in self.reward_fns:
            rewards_k = []
            for i in range(G):
                response_ids = output_ids[i, prompt_len:]
                response_text = self.tokenizer.decode(
                    response_ids, skip_special_tokens=True
                )
                r = reward_fn(prompt_text, response_text)
                rewards_k.append(r)
            rewards_list.append(
                torch.tensor(rewards_k, device=output_ids.device)
            )
        
        return output_ids, rewards_list, output_masks
    
    def compute_log_probs(
        self,
        model: torch.nn.Module,
        input_ids: Tensor,
        attention_mask: Tensor,
        response_start: int,
    ) -> Tensor:
        """Compute per-token log probs for response."""
        outputs = model(input_ids, attention_mask=attention_mask)
        logits = outputs.logits
        
        shift_logits = logits[:, :-1, :]
        shift_labels = input_ids[:, 1:]
        
        log_probs = F.log_softmax(shift_logits, dim=-1)
        token_log_probs = log_probs.gather(
            dim=-1, index=shift_labels.unsqueeze(-1)
        ).squeeze(-1)
        
        return token_log_probs[:, response_start - 1:]
    
    def train_step(
        self,
        prompt_text: str,
        prompt_ids: Tensor,
        attention_mask: Tensor,
    ) -> dict:
        """
        Single GDPO training step.
        """
        prompt_len = prompt_ids.shape[1]
        
        # Sample and score
        output_ids, rewards_list, output_masks = self.sample_and_score(
            prompt_text, prompt_ids, attention_mask
        )
        
        # Compute GDPO advantages
        advantages, per_reward_adv = compute_gdpo_advantages(
            rewards_list,
            self.reward_weights.tolist(),
            self.config.eps,
        )
        
        # Get log probs
        with torch.no_grad():
            old_log_probs = self.compute_log_probs(
                self.policy, output_ids, output_masks, prompt_len
            )
            ref_log_probs = self.compute_log_probs(
                self.reference, output_ids, output_masks, prompt_len
            )
        
        # Update: one step per batch (μ = 1), so the ratio is exactly 1 and
        # the clip is inactive. See Part 3's μ loop to reuse the batch.
        self.policy.train()
        policy_log_probs = self.compute_log_probs(
            self.policy, output_ids, output_masks, prompt_len
        )
        
        response_mask = output_masks[:, prompt_len:]
        
        # Compute loss (same as GRPO)
        loss, metrics = self._compute_loss(
            policy_log_probs,
            old_log_probs,
            ref_log_probs,
            advantages,
            response_mask,
        )
        
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(
            self.policy.parameters(), 
            self.config.max_grad_norm
        )
        self.optimizer.step()
        
        # Add per-reward metrics
        for k, rewards_k in enumerate(rewards_list):
            metrics[f"reward_{k}_mean"] = rewards_k.mean().item()
            metrics[f"reward_{k}_std"] = rewards_k.std().item()
            metrics[f"adv_{k}_mean"] = per_reward_adv[k].mean().item()
        
        return metrics
    
    def _compute_loss(
        self,
        policy_log_probs: Tensor,
        old_log_probs: Tensor,
        ref_log_probs: Tensor,
        advantages: Tensor,
        mask: Tensor,
    ) -> Tuple[Tensor, dict]:
        """Compute GDPO loss with clipping and KL."""
        G, seq_len = policy_log_probs.shape
        
        # Importance ratio
        ratio = torch.exp(policy_log_probs - old_log_probs)
        
        # Expand advantages
        adv_expanded = advantages.unsqueeze(1).expand(-1, seq_len)
        
        # Clipped objective
        clipped = torch.clamp(
            ratio,
            1 - self.config.clip_epsilon,
            1 + self.config.clip_epsilon
        )
        
        policy_loss = -torch.min(
            ratio * adv_expanded,
            clipped * adv_expanded
        )
        
        # KL penalty
        log_ratio = ref_log_probs - policy_log_probs
        kl = torch.exp(log_ratio) - log_ratio - 1
        
        # Combine
        token_loss = policy_loss + self.config.kl_coef * kl
        
        # Mask and average
        masked_loss = (token_loss * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1)
        loss = masked_loss.mean()
        
        with torch.no_grad():
            clip_frac = ((ratio - 1).abs() > self.config.clip_epsilon).float()
            clip_frac = (clip_frac * mask).sum() / mask.sum()
        
        metrics = {
            "loss": loss.item(),
            "mean_kl": (kl * mask).sum().item() / mask.sum().item(),
            "clip_fraction": clip_frac.item(),
            "mean_advantage": advantages.mean().item(),
        }
        
        return loss, metrics

Experimental Results

The GDPO paper compares GRPO and GDPO on three tasks. The numbers below are the paper’s (arXiv:2601.05242).

Tool Calling (2 Rewards)

Format and correctness rewards, evaluated on BFCL-v3. Averages of 5 runs, 4 rollouts per prompt, 100 training steps (paper, Table 1):

GDPO is ahead on both metrics at both sizes, by 0.6 to 4.3 points.

Math Reasoning (2 Rewards)

The rewards are correctness and a length limit — stay under 4,000 tokens — trained on DeepScaleR-Preview with 16 rollouts per prompt. Both methods use DAPO’s dynamic sampling, clip-higher and token-mean loss. AIME pass@1 and the share of responses over the length limit (paper, Table 3):

Two caveats. Not every cell favours GDPO: on MATH, DeepSeek-R1-7B scores 94.1% with GRPO and 93.9% with GDPO. And both methods trail the base models on AIME, because the length limit cuts off long reasoning (DeepSeek-R1-7B base: 55.4%). The comparison is between two ways of trading accuracy for length, not a free gain.

Code Generation (3 Rewards)

Three rewards — pass rate, a length limit, and bug ratio — on DeepSeek-R1-7B. The paper’s Table 5 reports all three metrics for GRPO and GDPO on each coding benchmark.

Ablation: Is It Just the Std?

Part 5 describes Dr. GRPO, which drops std normalisation altogether. Could that alone fix multi-reward collapse? The paper tested it: GRPO without std normalisation matched GDPO on correctness, but reached 0% correct format on BFCL-v3. Without any std, the rarely-varying format reward lost its share of the signal entirely. Per-reward std is the part doing the work.

Key Observations

1. More rewards, more distinct advantages: The number of distinct advantage values GDPO can produce grows with the number of rewards and rollouts much faster than GRPO’s (paper, Fig. 3). That is about signal resolution; the paper doesn’t show accuracy gains growing with the number of rewards.
2. Better on most metrics, not all: Gains are a few points, and some cells go the other way.
3. Complementary rewards: GDPO matters most when the rewards vary by different amounts within a group — typically one easy objective next to a hard one.

Key Insight: With a single reward, GDPO reduces to GRPO plus a batch-level normalisation. With several, it is a cheap change that helped in most settings the paper tried, by a few points.

When to Use GDPO vs GRPO

Use GRPO When:

• Single reward: No multi-reward setting
• Highly correlated rewards: Rewards that move together don’t benefit from decoupling
• Simplicity preferred: GRPO is simpler to implement

Use GDPO When:

• Multiple rewards: Two or more distinct reward functions
• Complementary rewards: Rewards measure different aspects (format vs. correctness)
• Training instability with GRPO: If GRPO shows oscillation or failure, try GDPO
• Production multi-objective alignment: Default for real-world multi-reward scenarios

Decision Tree

Production Patterns

Pattern 1: Conditioned Rewards (the default when priorities matter)

When one objective is much easier than another — format versus correctness, or length versus correctness — weights are a weak way to set priority. The GDPO paper found that weights don’t reliably make the model prioritise the hard objective, because the easy one is learned first and keeps paying out. What worked better was conditioning the easy reward on the hard one: grant the easy reward only when the hard one is satisfied.

def conditioned_length_reward(prompt: str, response: str) -> float:
    """Easy reward (stay under the length limit), granted only for a correct answer."""
    if correctness_reward(prompt, response) < 1.0:
        return 0.0
    return length_reward(prompt, response)

reward_fns = [correctness_reward, conditioned_length_reward]

The model can no longer collect the easy reward while ignoring the hard one. Use GDPO on top: the two rewards still vary by different amounts within a group.

Pattern 2: Format + Correctness

The most common multi-reward setup:

def format_reward(prompt: str, response: str) -> float:
    """Binary: follows required structure?"""
    has_think = "<think>" in response and "</think>" in response
    has_answer = "<answer>" in response and "</answer>" in response
    return 1.0 if (has_think and has_answer) else 0.0

def correctness_reward(prompt: str, response: str) -> float:
    """Continuous: semantically correct?"""
    # Extract answer and check against ground truth
    # Returns value in [-1, 1]
    ...

reward_fns = [format_reward, correctness_reward]
reward_weights = [1.0, 1.0]  # Equal weighting

Use GDPO: format is usually learned quickly and then barely varies, which is exactly when per-reward std matters. If the model satisfies the format but neglects correctness, switch to Pattern 1.

Pattern 3: Weighted Multi-Objective

Different importance for different rewards:

reward_fns = [safety_reward, helpfulness_reward, conciseness_reward]
reward_weights = [2.0, 1.0, 0.5]  # Safety prioritized

Weights scale each reward’s normalised contribution, but GDPO’s effective weight is $w_k / \sigma_k$, which shifts from group to group as each reward’s spread changes. Treat weights as a soft nudge, not a guarantee of priority; for a hard priority, condition (Pattern 1).

Pattern 4: Dynamic Weights

Adjusting weights during training:

def get_weights(step: int) -> List[float]:
    """Start with format emphasis, shift to correctness."""
    format_w = max(0.5, 1.0 - step / 10000)
    correct_w = min(1.5, 0.5 + step / 10000)
    return [format_w, correct_w]

GDPO handles this without extra work, since normalization is per-reward and the weights apply afterwards.

Key Takeaways

The Problem: Advantage Collapse

When GRPO normalizes combined rewards, groups with different reward profiles can collapse to identical advantages: satisfying two objectives can earn no more signal than satisfying one.

The Solution: GDPO

Normalize each reward independently within the group, then combine — in effect, GRPO on a sum where each reward is weighted by $w_k / \sigma_k$:

# GRPO (problematic)
combined = sum(w * r for w, r in zip(weights, rewards))
advantage = (combined - mean) / std

# GDPO (preserves signal)
normalized = [(r - mean(r)) / std(r) for r in rewards]
combined = sum(w * n for w, n in zip(weights, normalized))
advantage = (combined - mean) / std  # Final norm, across the whole batch

When to Use GDPO

• Multiple rewards measuring different aspects
• Tool calling, math reasoning, code generation
• Any multi-objective LLM alignment

Implementation

GDPO is a drop-in replacement for GRPO’s advantage computation. A minimal code change, worth a few points in the paper’s experiments. For hard priorities between rewards, condition the easy reward on the hard one rather than relying on weights.

The Broader Lesson

As LLM training becomes more sophisticated, we’ll see more multi-reward settings. Understanding how normalization affects training signal is crucial for designing effective alignment pipelines.

Series Summary

Over five articles, we’ve built a complete understanding of policy optimization for LLMs:

Each algorithm addresses specific limitations of its predecessor:

• PPO addresses policy gradient variance
• GRPO addresses PPO’s memory overhead
• GDPO addresses GRPO’s multi-reward limitations
• Part 5 maps the rest of the family — Dr. GRPO, DAPO, GSPO, CISPO — onto a handful of design axes

Understanding this progression helps you choose the right tool for your alignment task.

Further Reading

GDPO:

• GDPO: Group reward-Decoupled Normalization Policy Optimization (Liu et al., 2026)
• GitHub: NVlabs/GDPO

Foundations:

• DeepSeekMath — GRPO origin
• PPO — The baseline

Multi-Objective RL:

• Multi-Objective Reinforcement Learning: A Survey
• Reward Hacking in Reinforcement Learning

Implementations:

• TRL with GDPO
• VERL with GDPO
• NeMo-RL with GDPO

@misc{sousa2026gdpo,
  title={{GDPO: Multi-Reward RL Done Right}},
  author={Sousa, Vitor},
  year={2026},
  howpublished={\url{https://www.vitorsousa.com/blog/gdpo/}},
  url={https://www.vitorsousa.com/blog/gdpo/}
}