Part 5 of 5: The Map

TL;DR: DAPO, Dr. GRPO, GSPO, GDPO, CISPO, VAPO, GiGPO, GMPO, REINFORCE++… it looks like GRPO fragmented into a zoo. It didn’t. Every post-GRPO method keeps the same skeleton — drop the critic, compute a group-relative advantage — and varies along a small set of design axes: token vs. sequence granularity, clip width, length and standard-deviation normalization, KL on or off, how advantages are normalized, and how groups are sampled. Hold the axes and the zoo collapses into a handful of deliberate trade-offs. This post maps the four variants worth treating in depth, gives you a decision guide for when to reach for each, and equips you to reason about the next variant you meet — not just the ones listed here.

Reading time: ~18 minutes

Prerequisites: This is the capstone of the series. It assumes you already know GRPO. If not, start with Part 3: GRPO — and ideally Part 2: PPO and Part 4: GDPO — then come back. This post sits on top of those; it does not re-derive them.

The Zoo Is Real

Open any RL-for-LLMs reading list from the last year and you will drown in acronyms. DAPO. Dr. GRPO. GSPO. GDPO. CISPO. GMPO. VAPO. GiGPO. REINFORCE++. There is even a curated index whose entire job is to keep up with the post-R1 GRPO family. The natural reaction is the one everyone has: the field fragmented, and I now have ten algorithms to learn instead of one.

That reaction is wrong, and it is wrong in a useful way. These are not ten algorithms. They are one algorithm — GRPO — being paid down along a handful of engineering debts that GRPO left on the table. Once you can name those debts, every “new” method announces itself as which debt it pays and how. You stop memorizing the zoo and start reading coordinates on a map.

This post is that map. It is deliberately a synthesis, not a listicle: the value is in the axes and the judgement, not in summaries you could get from each paper’s abstract.

The Shared Skeleton (What Nobody Changes)

Start with what every member of the family keeps. Strip away the branding and they all make GRPO’s two core moves:

1. Drop the critic. No learned value network $V_\psi$. PPO’s four-model architecture loses a model, and with it roughly a third of the memory.
2. Compute a group-relative advantage. For a prompt $q$, sample a group of $G$ completions $\{o_1, \ldots, o_G\}$ from the old policy, score them, and use the group statistics as the baseline:

$\hat{A}_{i,t} = \tilde{r}_i = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}, \qquad \mathbf{r} = \{r_1, \ldots, r_G\}.$

That is the whole skeleton. At heart it is REINFORCE with a baseline — the mean reward of the group is the baseline — which is exactly the lineage Ahmadian et al. (2024) argued we should return to for RLHF. Everything is then plugged into a PPO-style clipped surrogate:

$\mathcal{L}_{i,t} = \min\left[\rho_{i,t}\, \hat{A}_{i,t}, \; \text{clip}(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon)\, \hat{A}_{i,t}\right], \qquad \rho_{i,t} = \frac{\pi_\theta(o_{i,t}\,|\,q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\,|\,q, o_{i,<t})}.$

Every variant in this post inherits these two equations. What they argue about is the fine print — and the fine print turns out to be a short list.

The Axes of Variation

This is the intellectual core of the post. Spend time here; the variants below are just coordinates in the space these axes define.

Granularity. Is the importance ratio $\rho$ and its clipping computed per token or per sequence? GRPO is token-level. A single rare token can produce a wildly large or small ratio, and that noise compounds — badly in mixture-of-experts models, where the active experts can change between the sampling policy and the training policy. Sequence-level methods average the signal over the whole completion so the unit of optimization matches the unit of reward.

Clip width. PPO/GRPO clip symmetrically, $[1-\varepsilon, 1+\varepsilon]$. But the clip ceiling quietly caps upside: it limits how much probability mass a good-but-currently-unlikely token can gain in one step. Those low-probability tokens are often the interesting ones — the “wait,” “let me reconsider,” fork-and-reflect tokens that long chain-of-thought reasoning depends on. Widening or de-symmetrizing the clip (“clip-higher”) controls the explore/exploit edge and is the main lever against entropy collapse, where the policy sharpens into a narrow, repetitive mode and stops learning.

Length and std normalization. GRPO divides the per-response loss by response length and divides the advantage by the group’s standard deviation. Both are “harmless-looking” conveniences that bias the gradient. Length normalization makes the optimizer prefer longer wrong answers (more tokens over which to spread a negative advantage means a smaller per-token penalty); std normalization over-weights easy prompts where rewards barely vary. These biases are why responses balloon during training without getting better.

KL term. Keep the KL penalty to the reference model, or drop it? KL anchors the policy and prevents drift, but it costs a reference model in memory and can hold the policy back when you want it to move far from the base model (as in R1-Zero-style training from a base checkpoint). Many recent recipes drop it entirely.

Advantage normalization. Per-group versus global (batch-wide) normalization, and — crucially in multi-objective settings — single-reward versus multi-reward. When several rewards are summed and then normalized together, distinct reward combinations can collapse to identical advantages, blinding the policy to one objective. This is the failure Part 4 is built around.

Sampling. Static groups versus dynamic sampling. A group where every completion is correct (or every one is wrong) has zero advantage variance and therefore contributes no gradient — it is wasted compute. Dynamically filtering or resampling degenerate groups keeps every batch informative.

Six axes. That is the entire design space the family lives in. Here is the same idea as a picture — each axis, and the variant that pushes hardest along it:

Two patterns jump out of this picture. First, most variants move along one axis — they are surgical. Second, DAPO moves along several at once; it is less a single idea than a curated bundle. Keep both observations in mind as we walk the variants.

What do these axes look like in a training run? The two biases that motivate Dr. GRPO and DAPO show up as characteristic curve shapes:

The Variants: Problem → Change → When

With the axes in hand, here are the four worth knowing in depth, plus the frontier cluster. Read each as a triple: the problem it noticed, the change it made, and the situation that should make you reach for it.

Dr. GRPO — debias the gradient

Understanding R1-Zero-Like Training: A Critical Perspective (Liu et al., 2025) noticed that GRPO’s length and standard-deviation normalization are not neutral conveniences — they are biases. The length term systematically rewards longer wrong answers, which is exactly the runaway-verbosity pathology practitioners kept seeing. The fix is almost embarrassingly small: delete both normalizers. No division by response length, no division by group std. The gradient becomes unbiased, responses stop inflating, and the model is markedly more token-efficient. Built on the lightweight Oat framework, with a clean reference implementation. Reach for it the moment you see length climbing without accuracy following.

DAPO — the production bundle

DAPO: An Open-Source LLM Reinforcement Learning System at Scale (Yu et al., 2025) is the one that is really four things at once — Decoupled Clip and Dynamic sAmpling Policy Optimization. It ships clip-higher (de-symmetrize the clip ceiling to keep low-probability exploratory tokens alive and fight entropy collapse), dynamic sampling (drop degenerate all-correct/all-wrong groups so every batch carries gradient), token-level policy-gradient loss (so long completions are not under-weighted in long-CoT training), and overlong reward shaping (soften the penalty for responses that run past the length budget, reducing reward noise). Together they reach 50 points on AIME 2024 with a Qwen2.5-32B base, with code and data fully open on top of verl (repo). DAPO is what you reach for when you are scaling up real long-CoT training and instability — diverging loss, collapsing entropy — is the thing standing between you and a good model.

GSPO — match the unit of optimization to the unit of reward

Group Sequence Policy Optimization (Zheng et al., 2025, the Qwen team) made a single sharp observation: the reward is assigned to a sequence, so why is the importance ratio computed per token? Token-level ratios are noisy, and that noise is catastrophic for mixture-of-experts models, where the set of active experts can differ between the sampling and training passes. GSPO defines the importance ratio on the sequence likelihood and clips at the sequence level. The result stabilizes MoE RL training — enough that it underpins the latest Qwen3 models. Reach for it when you are training MoE models or chasing large-scale stability, and when token-level ratios are giving you grief. (Implementations live in verl and TRL; the Qwen write-up is the readable intro.)

GDPO — keep multiple rewards from colliding

This is Part 4, so I will be brief. GDPO: Group reward-Decoupled Normalization Policy Optimization (Liu et al., 2026, NVIDIA) attacks the multi-reward case. Sum several rewards and normalize the total, and different reward combinations can map to the same advantage — the policy goes blind to one objective. GDPO normalizes each reward independently before combining, preserving the resolution of every signal. Reach for it whenever you are optimizing more than one reward at once — tool calling with format-plus-correctness, math with format-plus-answer-plus-integer, coding with execution-plus-tests-plus-style. (If you suspect this is already happening to you, the advantage-collapse checklist is a quick diagnostic.)

The frontier cluster

Beyond the four, a frontier worth knowing as coordinates, not as separate disciplines:

• CISPO, from the MiniMax-M1 technical report (2025), clips the importance-sampling weights rather than the token updates. The motivation is precisely the clip-width axis above: standard clipping silently discards the low-probability fork-and-reflect tokens, so CISPO keeps every token contributing and reports a 2× speedup over DAPO in their setting. Reach for it when you suspect your clip is eating your model’s “aha” moments.
• SPO (Single-stream Policy Optimization, Xu & Ding, 2025) goes group-free: it replaces the on-the-fly group baseline with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch. It is a glimpse of where the family heads once you decide the group itself is the bottleneck.

You can now place the rest of the zoo — VAPO, GMPO, GiGPO, REINFORCE++ — without my help: each one moves along one or two of the six axes. That is the entire point of the map.

A Decision Guide

The variants reduce to a flowchart. Start from the symptom, not the acronym.

The honest version of this flowchart has a caveat: the moves compose. You can run Dr. GRPO’s debiasing and clip-higher and drop KL at the same time; that is roughly what the minimal from-scratch repos do by default. The flowchart tells you which lever a symptom points to, not that you may pull only one.

”Is This Just PPO With Extra Steps?”

A fair skeptic will push back: if everything is REINFORCE-with-a-baseline wearing a PPO surrogate, did we really invent anything?

Two uncomfortable findings say mostly, the skeptic is right — and that is worth internalizing.

First, the on-policy story is a myth. Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm (Yao et al., 2025) derives group-relative REINFORCE from first principles and shows it admits a native off-policy interpretation — the within-group mean baseline does not require the on-policy assumption everyone attaches to it. Where this lands is sobering: the convergence point of the whole family is something close to “PPO with a global advantage and no critic.” The variants are different routes to the same neighborhood, not different destinations.

Second, and more deflating for algorithm enthusiasts: The Art of Scaling RL Compute for LLMs (Khatri, Madaan et al., 2025) — a 400,000+ GPU-hour study — finds that loss aggregation, normalization, curriculum, and the choice of off-policy algorithm primarily move compute efficiency, not the asymptotic performance ceiling. The ceiling is set elsewhere: by the data and the reward. Algorithms get you to the ceiling faster; they rarely raise it.

That is the judgement this map is really teaching. The axes are real and worth tuning — they decide whether your run is stable, efficient, and reaches the ceiling at all. But if you are choosing a variant hoping it will out-level your data, you have the wrong lever in your hand.

The Production Lens

Which points straight at the thing that actually decides outcomes: every variant on this map optimizes against a reward, and every reward is a proxy.

In math and code you get lucky — a verifier exists. The answer is right or it is not; the unit tests pass or they do not. Reinforcement learning with verifiable rewards (RLVR) works there precisely because the proxy is nearly the real thing. Step outside that world — helpfulness, faithfulness, tone, “did the agent actually accomplish the task” — and there is no clean verifier. You build one (a reward model, an LLM judge, a heuristic), and the optimizer’s job becomes finding its seams. Whatever gap exists between your proxy and what you actually want, a good optimizer will widen it. That is reward hacking, and a better optimizer hacks a flawed reward faster.

So the practical ranking is the opposite of the one the zoo implies. Outside math and code, the bottleneck is almost never the optimizer; it is the verifier. Picking GSPO over DAPO is a second-order decision. Building a reward your optimizer cannot game is the first-order one — and it is the genuinely hard, under-glamorous engineering that decides whether any of these algorithms help you. Hold this as a principle: spend your scrutiny on the reward, then let the axes tune the run.

Feeling the Differences on One GPU

You do not need a cluster to build intuition for the axes — you need a small model and an afternoon. The deltas between these methods are legible at tiny scale, and a handful of repos are built to make them legible:

• GRPO-Zero implements GRPO from scratch with DAPO’s improvements (token-level loss, no KL, overlong filtering) on the Countdown task. It is the single best repo to dissect if you want to see the axis-moves in code rather than read about them.
• simple_GRPO is ~200 lines across two files and trains in under an hour on a single A800 — teaching-oriented, nothing hidden.
• TinyZero reproduces R1-Zero-style training on verl for under $30, a clean baseline to fork.
• Unsloth’s GRPO notebooks get the R1-Zero “aha” on 5–7 GB of VRAM via QLoRA — about as close to a laptop as this gets.

For the research framing of why small-model RL is even worth your time, Reinforcement Learning for Reasoning in Small LLMs (Dang & Ngo, 2025) trained a 1.5B model to 46.7% on AIME 2024 for roughly $42 on 4×A40 in 24 hours — and was candid about what broke. If you would rather run the full family under one roof, verl implements almost all of it (PPO, GRPO, GSPO, DAPO, Dr. GRPO, RLOO, REINFORCE++, PRIME) with verifiable rewards; TRL is the gentlest on-ramp via its GRPOTrainer. This is exactly the territory my own verifiable-reward-lab project lives in.

Where the Family Is Heading

The axes will keep being the right lens, even as the methods drift past the ones named here. Three directions are already visible. Group-free methods like SPO ask whether the group baseline — the move that defined GRPO — is itself the next bottleneck, replacing it with a persistent value tracker. Agentic and hierarchical advantages push group-relative credit assignment down to steps and sub-trajectories for tool-using, multi-turn agents. And verifier-free RL tries to escape the proxy problem from the other side — learning a reward signal without a hand-built verifier at all, which, if it works, attacks the actual ceiling rather than the path to it.

None of these will look like a clean break. Each will arrive as another coordinate: a different granularity, a different clip, a different normalization, a different way to sample. GRPO never fragmented into a zoo. It exposed a design space — and once you can read the axes, you are not learning the next algorithm so much as recognizing where on the map it sits.

That is the whole series in one sentence: from PPO’s critic, to GRPO’s group baseline, to GDPO’s decoupled rewards, the field has been walking a small set of axes the entire time. You can see the full arc on the series page.

References

Foundations (linked, not re-derived here):

• Schulman et al. (2017), Proximal Policy Optimization Algorithms — arXiv:1707.06347
• Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs — arXiv:2402.14740
• Shao et al. (2024), DeepSeekMath (GRPO origin) — arXiv:2402.03300
• DeepSeek-AI / Guo et al. (2025), DeepSeek-R1 — arXiv:2501.12948

The four core variants:

• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO) — arXiv:2503.20783
• Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale — arXiv:2503.14476
• Zheng et al. (2025), Group Sequence Policy Optimization (GSPO) — arXiv:2507.18071
• Liu et al. (2026), GDPO: Group reward-Decoupled Normalization Policy Optimization — arXiv:2601.05242

Frontier and critical perspective:

• MiniMax (2025), MiniMax-M1 (introduces CISPO) — arXiv:2506.13585
• Yao et al. (2025), Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm — arXiv:2509.24203
• Khatri, Madaan et al. (2025), The Art of Scaling RL Compute for LLMs — arXiv:2510.13786
• Xu & Ding (2025), Single-stream Policy Optimization (SPO) — arXiv:2509.13232

Small-model / single-GPU:

• Dang & Ngo (2025), Reinforcement Learning for Reasoning in Small LLMs (Open-RS) — arXiv:2503.16219
• Frameworks and minimal repos: verl, TRL, Oat, GRPO-Zero, simple_GRPO, TinyZero, Unsloth, and the curated rl-llm-nlp index.

Introduction

OpenELM is a state-of-the-art open language model launched by Apple.
The researchers introduced OpenELM, a family of Open-source Efficient Language Models. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. The OpenELM models were pretrained using the CoreNet library. They release both pretrained and instruction-tuned models with 270M, 450M, 1.1B and 3B parameters.

They also release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs, to facilitate open research.

OpenELM outperforms existing open LLMs that are pre-trained using publicly available datasets. OpenELM with 1.1 billion parameters outperforms OLMo, which has 1.2 billion parameters, by 2.36% while requiring 2× fewer pre-training tokens.

Innovations in Architecture

OpenELM distinguishes itself with a layer-wise scaling strategy within its transformer architecture, enabling more efficient parameter allocation across layers. Traditional transformers allocate parameters uniformly across all layers, the models are isotropic. In contrast, OpenELM adjusts the configuration of each layer to optimize parameter usage effectively. This approach not only enhances the model’s learning efficiency but also significantly boosts its performance with less training data.

Key architectural points:

• They don’t use learnable bias parameters in any fully connected(linear) layers.
• Apply pre-normalization using RMSNorm
• Use Rotatory Positional Embedding (ROPE): Enhances the model’s understanding of word order, critical for generating coherent and contextually aware language outputs.
• Use Grouped Query Attention (GQA): An alternative to the typical multi-head attention that allows more nuanced attention mechanisms, improving the model’s interpretative capabilities.
• Use SwiGLU Activation: A novel activation function that provides a more dynamic range for model training, leading to faster convergence and improved accuracy (replacing the FFN).
• Use Flash Attention to compute the dot-product attention
• Use the same tokenizer as LLama.
  • They filter and tokenize data on the fly. This facilitates experimentation with different tokenizers.

Layer-wise scaling

For the non-uniform allocation of parameters in the transformer layer, they adjust the number of attention heads and the FFN multiplier in each transformer layer.

Assuming that the standard transformer model with uniform parameter allocation has $N$ transformer layers and the dimensionality of the input of each layer is $d_{model}$. The MHA has $n_h$​ heads and a dimension o

Where:

and

The $a_{min}, a_{max}$ are the hyperparameters that allow them to scal teh attention heads. Similarly $b_{min},b_{max}$ allow them to vary the with of the FFN layers. Note, setting $a_{min}= a_{max}=1$ and $m_i = m$ produces the standard uniform transformer model.

Pretraining

The used pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totalling approximately 1.8 trillion tokens.

They filter and tokenize data on the fly. This facilitates experimentation with different tokenizers.

To filter out low-length sequences, they apply two filtering methods. The first method operates at the characterlevel, checking if the number of characters in the sequence is below a specified threshold. The second method operates at the token-level, where it examines whether the sequence contains fewer tokens than a specified threshold. Sequences that are shorter than either of these thresholds are skipped. In experiments, were used 200 characters and 256 tokens as character and token-level filtering thresholds.

Performance and Benchmarking

OpenELM’s performance is outstanding, especially when benchmarked against similar-sized models like OLMo. With 1.1 billion parameters, OpenELM achieves a 2.36% higher accuracy while requiring only half the pre-training tokens. This efficiency is illustrated through various benchmarks on tasks such as ARC-c, BoolQ, and HellaSwag, where OpenELM consistently outperforms its peers.

Evaluation

They evaluate the performance across different tasks using LM Evaluation Harness:

• Standard zero-shot tasks(7):

  • ARC easy and challenge,
  • BoolQ,
  • HellaSwag,
  • PIQA,
  • SciQ, and
  • WinoGrande

  Model Size	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	SciQ	WinoGrande	Average
  OpenELM-270M	26.45	45.08	53.98	46.71	69.75	84.70	53.91	54.37
  OpenELM-270M-Instruct	30.55	46.68	48.56	52.07	70.78	84.40	52.72	55.11
  OpenELM-450M	27.56	48.06	55.78	53.97	72.31	87.20	58.01	57.56
  OpenELM-450M-Instruct	30.38	50.00	60.37	59.34	72.63	88.00	58.96	59.95
  OpenELM-1_1B	32.34	55.43	63.58	64.81	75.57	90.60	61.72	63.44
  OpenELM-1_1B-Instruct	37.97	52.23	70.00	71.20	75.03	89.30	62.75	65.50
  OpenELM-3B	35.58	59.89	67.40	72.44	78.24	92.70	65.51	67.39
  OpenELM-3B-Instruct	39.42	61.74	68.17	76.36	79.00	92.50	66.85	69.15

• OpenLLM (5)

  • ARC challenge,
  • HellaSwag,
  • MMLU,
  • TruthfulQA, and
  • WinoGrande.

  Model Size	ARC-c	HellaSwag	MMLU	TruthfulQA	WinoGrande	Average
  OpenELM-270M	27.65	47.15	25.72	39.24	53.83	38.72
  OpenELM-270M-Instruct	32.51	51.58	26.70	38.72	53.20	40.54
  OpenELM-450M	30.20	53.86	26.01	40.18	57.22	41.50
  OpenELM-450M-Instruct	33.53	59.31	25.41	40.48	58.33	43.41
  OpenELM-1_1B	36.69	65.71	27.05	36.98	63.22	45.93
  OpenELM-1_1B-Instruct	41.55	71.83	25.65	45.95	64.72	49.94
  OpenELM-3B	42.24	73.28	26.76	34.98	67.25	48.90
  OpenELM-3B-Instruct	47.70	76.87	24.80	38.76	67.96	51.22

• LLM360 leaderboard tasks (7):

  • ARC challenge,
  • CrowS-Pairs (English version),
  • HellaSwag,
  • WinoGrande,
  • MMLU,
  • PIQA, and
  • RACE

  Model Size	ARC-c	CrowS-Pairs	HellaSwag	MMLU	PIQA	RACE	TruthfulQA	WinoGrande	Average
  OpenELM-270M	27.65	66.79	47.15	25.72	69.75	30.91	39.24	53.83	45.13
  OpenELM-270M-Instruct	32.51	66.01	51.58	26.70	70.78	33.78	38.72	53.20	46.66
  OpenELM-450M	30.20	68.63	53.86	26.01	72.31	33.11	40.18	57.22	47.69
  OpenELM-450M-Instruct	33.53	67.44	59.31	25.41	72.63	36.84	40.48	58.33	49.25
  OpenELM-1_1B	36.69	71.74	65.71	27.05	75.57	36.46	36.98	63.22	51.68
  OpenELM-1_1B-Instruct	41.55	71.02	71.83	25.65	75.03	39.43	45.95	64.72	54.40
  OpenELM-3B	42.24	73.29	73.28	26.76	78.24	38.76	34.98	67.25	54.35
  OpenELM-3B-Instruct	47.70	72.33	76.87	24.80	79.00	38.47	38.76	67.96	55.73

OpenELM outperforms existing open LLMs that are pre-trained using publicly available datasets. OpenELM with 1.1 billion parameters outperforms OLMo, which has 1.2 billion parameters, by 2.36% while requiring 2× fewer pre-training tokens.

Instruction tuning results

They also check the impact of inctruct tuning, that consistently improves OpenELM’s average accuracy by 1-2% across different evaluation frameworks.

Parameter-efficient fine-tuning (PEFT) results

They used CommonSense reasoning training and evaluation setup that provides 170k training samples across 8 multiple-choice datasets for PEFT studies with different methods, including LoRA and DoRA

Decrease on throughput

In one of the lastest section they show an analysis that reveals tha OpenELM is a bit slower thatn OLMo. They attribute significant portion of OpenELM’s processing time to the naive implementation of RMSNorm. Naive RMSNorm implementation results in many individual kernel launches each of which processes a small input, rather than a launch of a single, fused kernel, as would be the case with e.g. LayerNorm. By replacing the naive RMSNorm with Apex’s RMSNorm we observe a notable increase in OpenELM’s throughput.

Conclusion

The major point was the usage of a layer-wise scaling method for efficient parameter allocation within the transformer model, resulting in improved accuracy compared to existing models.

OpenELM is more than just another entry in the growing list of language models. Its innovative architecture, superior performance, and unprecedented commitment to openness set a new standard in the field. It’s really interesting see a company like Apple to release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained checkpoints and training logs, this could really help in further developments in research.

In future work, one of the point identified to improve the inference efficiency of OpenELM.

Explore More

For those interested in diving deeper into OpenELM or even experimenting with the model, check out the resources provided by the researchers .

Part 1 of 4: The Mathematical Foundations

TL;DR: Before understanding PPO, GRPO, or any modern LLM alignment algorithm, you need solid RL foundations. This article covers MDPs, policy gradients, the variance problem, value functions, Bellman equations, and advantage estimation—all through the lens of language model fine-tuning. By the end, you’ll understand why these algorithms are designed the way they are.

Reading time: ~35 minutes

Why RL for Language Models?

You’ve trained a language model on terabytes of text. It can complete sentences, answer questions, even write code. But it also happily generates harmful content, confidently states falsehoods, and ignores user preferences. Supervised fine-tuning on curated examples helps, but you can’t enumerate every possible good response.

What you need is a way to optimize for “goodness” directly—to tell the model “responses like this are better than responses like that” and have it learn the pattern. This is exactly what reinforcement learning provides.

The insight behind RLHF (Reinforcement Learning from Human Feedback) is powerful: instead of showing the model correct outputs, we show it a reward signal indicating how good its outputs are. The model then learns to maximize this reward.

But RL for LLMs isn’t straightforward. The action space is enormous (vocabulary size per token), episodes are variable length, rewards are sparse (often only at generation end), and we need stable training for billion-parameter models.

Understanding the foundations deeply will help you see why algorithms like PPO and GRPO make specific design choices—and when those choices might not be right for your problem.

Table of Contents

1. The MDP Framework
2. Policies and the Objective
3. Policy Gradients: Learning Without a Model
4. The Variance Problem
5. Value Functions and Bellman Equations
6. Advantage Functions: The Key Insight
7. Actor-Critic Methods
8. From General RL to LLM Fine-Tuning
9. Key Takeaways

The MDP Framework

Reinforcement learning formalizes sequential decision-making through Markov Decision Processes (MDPs). Understanding this framework precisely will help you see how LLM generation maps onto RL concepts.

Definition

An MDP is defined by the tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$:

The Markov Property

The “Markov” in MDP refers to the Markov property: the future depends only on the current state, not on how we got there.

$P(s_{t+1} | s_t, a_t, s_{t-1}, a_{t-1}, \ldots) = P(s_{t+1} | s_t, a_t)$

This is crucial for tractability—we don’t need to remember the entire history, just the current state.

The Agent-Environment Loop

At each timestep $t$:

1. Agent observes state $s_t$
2. Agent selects action $a_t$ according to its policy
3. Environment transitions to $s_{t+1} \sim P(\cdot | s_t, a_t)$
4. Agent receives reward $r_t = R(s_t, a_t, s_{t+1})$
5. Repeat until terminal state or horizon

LLM Generation as an MDP

For language model fine-tuning, we can map concepts as follows:

The state space is enormous (all possible token sequences up to context length), and the action space is the vocabulary size (typically 32K-128K tokens). This scale motivates many of the algorithmic choices we’ll see.

Key Insight: The Markov property technically holds for LLM generation—the next token distribution depends only on the current sequence, not on how we generated it. However, the state itself encodes the full history, so we’re not actually discarding information.

Policies and the Objective

What is a Policy?

A policy $\pi$ maps states to actions. It can be:

• Deterministic: $a = \pi(s)$ — always the same action in state $s$
• Stochastic: $a \sim \pi(a|s)$ — a probability distribution over actions

For LLMs, the policy is inherently stochastic: $\pi_\theta(o_t | s_t) = \pi_\theta(o_t | q, o_{<t})$ is the probability of token $o_t$ given the context. The parameters $\theta$ are the model weights.

The Return

The return $G_t$ is the cumulative (discounted) reward from time $t$ onwards:

$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k} = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \cdots$

The discount factor $\gamma$ serves multiple purposes:

• Mathematical: Ensures the sum converges for infinite horizons
• Practical: Encodes preference for immediate vs. delayed rewards
• LLM setting: Often $\gamma = 1$ since episodes are finite

The RL Objective

The goal of RL is to find a policy that maximizes expected return:

$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^{T} \gamma^t r_t\right] = \mathbb{E}_{\tau \sim \pi_\theta}[G_0]$

where $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \ldots)$ is a trajectory sampled by following policy $\pi_\theta$.

The expectation is over:

• Initial state distribution $s_0 \sim \rho_0$
• Actions from policy $a_t \sim \pi_\theta(a|s_t)$
• Transitions from dynamics $s_{t+1} \sim P(s'|s_t, a_t)$

For LLMs Specifically

In the LLM setting, the objective simplifies to:

$J(\theta) = \mathbb{E}_{q \sim P(Q), \, o \sim \pi_\theta(O|q)}[R(q, o)]$

where:

• $q$ is a prompt from the prompt distribution $P(Q)$
• $o = (o_1, \ldots, o_T)$ is a complete response sampled from the model
• $R(q, o)$ is the reward (typically from a reward model)

With sparse rewards (only at episode end), there’s no discounting to worry about within an episode.

Policy Gradients: Learning Without a Model

How do we optimize $J(\theta)$? We can’t compute it analytically—it requires integrating over all possible trajectories. But we can estimate its gradient and use gradient ascent.

The Policy Gradient Theorem

The policy gradient theorem (Sutton et al., 1999) provides a remarkable result: we can estimate $\nabla_\theta J(\theta)$ using only samples, without knowing the transition dynamics $P$.

$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t\right]$

Let’s unpack this:

• $\nabla_\theta \log \pi_\theta(a_t|s_t)$ is the score function—the direction that increases the probability of action $a_t$
• $G_t$ is the return from time $t$—how good the outcome was
• The product says: “increase probability of actions that led to high returns”

Intuition: Credit Assignment

The gradient has an elegant interpretation:

1. Sample a trajectory by running the policy
2. For each action taken, compute how much total reward followed
3. If reward was high, increase that action’s probability
4. If reward was low, decrease that action’s probability

This is the core of policy gradient methods: reinforce good actions, discourage bad ones.

The REINFORCE Algorithm

The simplest policy gradient algorithm is REINFORCE (Williams, 1992):

Algorithm: REINFORCE

For each episode:
    1. Sample trajectory τ = (s₀, a₀, r₀, ..., s_T, a_T, r_T) using π_θ
    
    2. For t = 0 to T:
        Compute return: G_t = Σₖ γᵏ r_{t+k}
    
    3. Compute gradient estimate:
        ∇̂ = Σₜ ∇_θ log π_θ(aₜ|sₜ) · Gₜ
    
    4. Update: θ ← θ + α∇̂

Why This Works: The Log-Derivative Trick

The derivation uses the log-derivative trick:

$\nabla_\theta \pi_\theta(a|s) = \pi_\theta(a|s) \cdot \nabla_\theta \log \pi_\theta(a|s)$

This converts a gradient of a probability into an expectation we can sample:

$\nabla_\theta \mathbb{E}_{a \sim \pi_\theta}[f(a)] = \mathbb{E}_{a \sim \pi_\theta}[f(a) \cdot \nabla_\theta \log \pi_\theta(a|s)]$

The full derivation for trajectories is more involved but follows the same principle.

Key Insight: Policy gradients are model-free—we don’t need to know $P(s'|s,a)$. We only need to sample from the environment and observe rewards. This is crucial for LLMs where the “environment” is just string concatenation.

The Variance Problem

REINFORCE is elegant but has a critical flaw: high variance. The gradient estimates are so noisy that learning is impractically slow.

Sources of Variance

Consider what affects the gradient estimate $\nabla_\theta \log \pi_\theta(a_t|s_t) \cdot G_t$:

1. Stochastic policy: Different action samples give different gradients
2. Stochastic transitions: Same action can lead to different states
3. Long horizons: $G_t$ includes all future rewards, accumulating randomness
4. Sparse rewards: Most of the signal comes from rare events

A Concrete Example

Suppose you’re training an LLM to solve math problems. The reward is +1 for correct, 0 for incorrect.

Episode 1: Model generates a correct solution. Every token gets gradient proportional to +1.

Episode 2: Model generates an incorrect solution. Every token gets gradient proportional to 0.

The problem: In episode 1, all tokens get positive reinforcement—including lucky guesses, unnecessary steps, and tokens that happened to work but weren’t essential. The gradient doesn’t distinguish “this token was crucial” from “this token was irrelevant.”

Variance Reduction with Baselines

A key insight: we can subtract any baseline $b(s_t)$ that doesn’t depend on actions without changing the expected gradient:

$\nabla_\theta J(\theta) = \mathbb{E}\left[\sum_{t} \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (G_t - b(s_t))\right]$

Why doesn’t this change the expectation?

$\mathbb{E}_{a \sim \pi}\left[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)\right] = b(s) \cdot \mathbb{E}_{a \sim \pi}\left[\nabla_\theta \log \pi_\theta(a|s)\right] = b(s) \cdot \nabla_\theta \sum_a \pi_\theta(a|s) = b(s) \cdot \nabla_\theta 1 = 0$

The baseline can be anything that doesn’t depend on the action—but choosing it well dramatically reduces variance.

Optimal Baseline

The variance-minimizing baseline is approximately the expected return from state $s_t$:

$b^*(s_t) \approx \mathbb{E}[G_t | s_t]$

This is the value function $V(s_t)$, which we’ll explore next.

Intuition: Relative Performance

With a good baseline, the gradient update becomes:

$\nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (G_t - V(s_t))$

This says:

• If $G_t > V(s_t)$: Action was better than expected → increase probability
• If $G_t < V(s_t)$: Action was worse than expected → decrease probability
• If $G_t = V(s_t)$: Action was exactly as expected → no change

This relative signal is far more informative than absolute returns.

Key Insight: The baseline transforms absolute returns into relative performance measures. This is the foundation of advantage-based methods—and explains why GRPO’s group normalization works.

Value Functions and Bellman Equations

To use $V(s)$ as a baseline, we need to estimate it. Value functions are central to RL and worth understanding deeply.

State Value Function

The state value function $V^\pi(s)$ is the expected return starting from state $s$ and following policy $\pi$:

$V^\pi(s) = \mathbb{E}_\pi\left[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \,\Big|\, s_t = s\right]$

This answers: “How good is it to be in state $s$?”

Action Value Function

The action value function (or Q-function) $Q^\pi(s, a)$ is the expected return starting from state $s$, taking action $a$, then following policy $\pi$:

$Q^\pi(s, a) = \mathbb{E}_\pi\left[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \,\Big|\, s_t = s, a_t = a\right]$

This answers: “How good is it to take action $a$ in state $s$?”

Relationship Between V and Q

The value function is the expected Q-value under the policy:

$V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] = \sum_a \pi(a|s) Q^\pi(s, a)$

The Bellman Equation

Value functions satisfy a recursive relationship called the Bellman equation:

$V^\pi(s) = \mathbb{E}_{a \sim \pi, s' \sim P}\left[r(s, a, s') + \gamma V^\pi(s')\right]$

This says: “The value of a state equals the expected immediate reward plus the discounted value of the next state.”

For Q-functions:

$Q^\pi(s, a) = \mathbb{E}_{s' \sim P}\left[r(s, a, s') + \gamma \sum_{a'} \pi(a'|s') Q^\pi(s', a')\right]$

Why Bellman Equations Matter

The Bellman equation enables bootstrapping: we can estimate $V(s)$ using our current estimate of $V(s')$, without waiting for the episode to end.

Monte Carlo: Wait for episode end, compute $G_t = \sum_k \gamma^k r_{t+k}$, update $V(s_t) \leftarrow V(s_t) + \alpha(G_t - V(s_t))$

Temporal Difference (TD): After one step, update using $V(s_t) \leftarrow V(s_t) + \alpha(r_t + \gamma V(s_{t+1}) - V(s_t))$

TD learning has lower variance (one-step update vs. full episode) but introduces bias (using an estimate $V(s_{t+1})$ instead of the true expected return).

TD Error

The TD error is the difference between our estimate and the bootstrapped target:

$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$

If our value function is accurate, $\mathbb{E}[\delta_t] = 0$. The TD error measures how “surprised” we are by what happened.

Key Insight: Bellman equations let us learn value functions incrementally, updating after each step rather than waiting for episodes to complete. This is especially valuable for long episodes like LLM generation.

Advantage Functions: The Key Insight

The advantage function combines V and Q to measure how much better an action is than average:

$A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$

Interpretation

• $A(s, a) > 0$: Action $a$ is better than the average action in state $s$
• $A(s, a) < 0$: Action $a$ is worse than average
• $A(s, a) = 0$: Action $a$ is exactly average

The advantage answers the crucial question: “Is this action better or worse than what I’d typically do?”

Why Advantages for Policy Gradients?

Using advantages in the policy gradient gives:

$\nabla_\theta J(\theta) = \mathbb{E}\left[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot A^\pi(s_t, a_t)\right]$

This is optimal for variance reduction because:

1. $\mathbb{E}_a[A(s, a)] = 0$ by construction (advantages are centered)
2. The signal focuses on relative action quality
3. It separates “was the action good?” from “was the state good?”

Estimating Advantages

We rarely know $A^\pi$ exactly. Common estimators:

1. Monte Carlo Advantage: $\hat{A}_t = G_t - V(s_t)$ Use actual returns minus estimated value. Unbiased but high variance.

2. TD Advantage: $\hat{A}_t = r_t + \gamma V(s_{t+1}) - V(s_t) = \delta_t$ One-step TD error. Low variance but biased.

3. Generalized Advantage Estimation (GAE): $\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{k=0}^{\infty} (\gamma\lambda)^k \delta_{t+k}$ Interpolates between MC ($\lambda=1$) and TD ($\lambda=0$). We’ll explore GAE deeply in Part 2.

Properties of Good Advantage Estimators

There’s typically a bias-variance tradeoff: lower variance estimators introduce more bias (by relying on learned value functions).

Key Insight: The advantage function is the “right” quantity for policy gradients. It tells us exactly what we want to know: was this action better or worse than average? All modern policy optimization methods (PPO, GRPO, etc.) are fundamentally about estimating advantages well.

Actor-Critic Methods

Actor-critic methods combine policy gradients (the “actor”) with learned value functions (the “critic”). This architecture underlies PPO and most modern RL algorithms.

The Two Components

Actor: The policy $\pi_\theta(a|s)$ that selects actions

• Parametrized by $\theta$
• Optimized via policy gradients
• Goal: maximize expected return

Critic: The value function $V_\psi(s)$ or $Q_\psi(s, a)$

• Parametrized by $\psi$
• Optimized via Bellman equation (TD learning)
• Goal: accurately estimate expected returns

Why Two Networks?

Each component benefits the other:

1. Critic helps actor: Provides baseline/advantage estimates for lower-variance policy gradients
2. Actor helps critic: Generates on-policy data for value function training

The A2C Algorithm

Advantage Actor-Critic (A2C) is a foundational actor-critic algorithm:

Algorithm: A2C

For each batch of experience:
    1. Collect trajectories using current policy π_θ
    
    2. Compute advantages:
        For each timestep t:
            δ_t = r_t + γV_ψ(s_{t+1}) - V_ψ(s_t)  # TD error
            (Or use GAE for multi-step advantages)
    
    3. Update critic (minimize value loss):
        L_critic = Σ_t (V_ψ(s_t) - G_t)²  # Or TD targets
        ψ ← ψ - α_critic ∇_ψ L_critic
    
    4. Update actor (maximize policy objective):
        L_actor = Σ_t log π_θ(a_t|s_t) · Â_t
        θ ← θ + α_actor ∇_θ L_actor

On-Policy vs. Off-Policy

On-policy methods (A2C, PPO, GRPO) require data from the current policy:

• Samples from $\pi_\theta$ used to update $\pi_\theta$
• Must discard data after each update
• Simpler theory but less sample efficient

Off-policy methods (DQN, SAC) can reuse old data:

• Samples from any policy can update $\pi_\theta$
• Requires importance sampling or replay buffers
• More sample efficient but more complex

LLM fine-tuning typically uses on-policy methods because:

1. The state space (text) is too large for replay buffers
2. On-policy methods are more stable for large models
3. Sample efficiency matters less when each “sample” is a full text generation

Challenges in Actor-Critic

1. Training instability: Actor and critic must improve together; if one diverges, both fail
2. Sample efficiency: On-policy methods need fresh samples after each update
3. Hyperparameter sensitivity: Learning rates, advantage estimation, entropy bonuses all interact
4. Scale: For LLMs, the critic is another massive network to train

Key Insight: Actor-critic methods are powerful but complex. The critic provides essential variance reduction, but at the cost of doubled model capacity and potential instability. This motivates GRPO’s approach of eliminating the critic entirely.

From General RL to LLM Fine-Tuning

Let’s map everything we’ve learned to the specific setting of language model alignment.

The LLM-RL Correspondence

Unique Challenges for LLMs

1. Enormous Action Space

Vocabulary sizes of 32K-128K tokens mean:

• Can’t enumerate all actions
• Must use function approximation (the LLM itself)
• Exploration is implicit in stochastic sampling

2. Sparse Rewards

Typically, reward comes only at generation end:

• $r_t = 0$ for $t < T$
• $r_T = R(q, o)$ — the reward model score

This makes credit assignment hard: which tokens were responsible for the reward?

3. Variable-Length Episodes

Generations can be 10 tokens or 1000 tokens:

• Can’t use fixed-horizon methods
• Must handle EOS token properly
• Padding and masking complexities

4. The Reference Model Constraint

Unlike standard RL, we don’t want to maximize reward unconditionally. We want to improve while staying close to a reference model:

$\max_\theta \mathbb{E}[R(q, o)] - \beta \cdot \text{KL}(\pi_\theta || \pi_{\text{ref}})$

This prevents:

• Reward hacking: Finding exploits in the reward model
• Mode collapse: Generating the same high-reward response always
• Capability loss: Forgetting useful behaviors from pretraining

5. Scale

Training billion-parameter models requires:

• Massive memory for model weights, gradients, optimizer states
• Distributed training across many GPUs
• Careful numerical stability

The RLHF Pipeline

What’s Coming Next

With these foundations, we’re ready to understand PPO (Part 2):

• How clipping provides stable updates
• How GAE estimates advantages
• Why it requires four models
• And why that’s a problem

Then GRPO (Part 3) will show how to eliminate the critic by clever use of group statistics.

Finally, GDPO (Part 4) will address multi-reward settings where GRPO falls short.

Key Takeaways

The MDP Framework

• RL formalizes sequential decisions as states, actions, rewards
• Markov property: future depends only on current state
• LLM generation maps naturally to this framework

Policy Gradients

• We can optimize expected reward using gradient ascent
• The policy gradient theorem enables learning without knowing dynamics
• REINFORCE is simple but high-variance

The Variance Problem

• Raw policy gradients are too noisy for practical use
• Baselines reduce variance without changing expected gradient
• The optimal baseline is approximately the value function

Value Functions

• $V(s)$ = expected return from state $s$
• $Q(s,a)$ = expected return from state $s$ taking action $a$
• Bellman equations enable bootstrapped learning

Advantage Functions

• $A(s,a) = Q(s,a) - V(s)$ measures relative action quality
• Advantages are centered: $\mathbb{E}_a[A(s,a)] = 0$
• This is the key quantity for policy optimization

Actor-Critic

• Actor (policy) + Critic (value function) work together
• Critic provides variance reduction for actor updates
• Doubles the model capacity required

LLM-Specific Considerations

• Sparse rewards make credit assignment hard
• Reference model constraint prevents reward hacking
• Scale demands memory-efficient algorithms

What’s Next

In Part 2: PPO for Language Models, we’ll see how Proximal Policy Optimization addresses many challenges:

• Trust regions for stable updates
• GAE for flexible advantage estimation
• Clipping for simplicity

But we’ll also see PPO’s costs: the four-model architecture that strains GPU memory, and the complexity that makes implementation tricky. This sets the stage for GRPO’s elegant simplification.

Further Reading

Foundational Papers:

• Policy Gradient Methods for Reinforcement Learning with Function Approximation (Sutton et al., 1999)
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (Williams, 1992 — REINFORCE)

Textbooks:

• Reinforcement Learning: An Introduction (Sutton & Barto, 2018) — The definitive reference
• Deep Reinforcement Learning (Foundations and Trends survey)

LLM-Specific:

• Training language models to follow instructions with human feedback (InstructGPT, Ouyang et al., 2022)
• Learning to summarize from human feedback (Stiennon et al., 2020)

Part 2 of 5: Theory That Informs Practice

TL;DR: Contextual bandits minimize regret—cumulative opportunity cost versus the optimal policy. Optimal algorithms achieve O(√(dT log T)) regret for d-dimensional contexts. The exploration-exploitation tradeoff is fundamental: confidence bounds (UCB) and posterior sampling (Thompson) provide principled solutions. This math directly guides algorithm selection and hyperparameter tuning.

Reading time: ~18 minutes

Introduction: Why Math Matters for Practice

You don’t need to derive regret bounds to deploy bandits in production. But understanding the mathematics directly informs practical decisions:

• Algorithm selection: Why LinUCB for linear rewards? Why Thompson Sampling often wins empirically?
• Hyperparameter tuning: What does α control in LinUCB? How to set it?
• Debugging: Why is my bandit not converging? (Check if regret is sublinear)
• Feature engineering: Why does dimensionality hurt? (√d penalty in regret bounds)

This post builds intuition for the theory that makes contextual bandits work.

The Contextual Bandit Problem (Formal)

Setup

At each round t = 1, 2, …, T:

1. Environment reveals context xₜ ∈ X (context space)
2. Agent chooses action aₜ ∈ A from available action set
3. Environment generates reward rₜ ~ P(r | xₜ, aₜ) from unknown distribution
4. Agent observes reward rₜ for chosen action (but not rewards for unchosen actions)

Goal

Learn policy π: X → A that maximizes expected cumulative reward:

$V(\pi) = \mathbb{E}\left[\sum_{t=1}^{T} r_t \mid \pi \right]$

Key Insight: Partial Feedback

We only observe rewards for actions we take. This is the partial feedback problem—we never know what would have happened if we’d chosen differently (the counterfactual outcome).

Example:

Context: User is 25-year-old in San Francisco
Action chosen: Recommend Product A
Reward observed: Click (reward = 1)

Unknown counterfactuals:
- What if we'd shown Product B? Maybe reward = 0 (worse)
- What if we'd shown Product C? Maybe reward = 1 (equally good)

We'll never know unless we try those actions in similar contexts.

This partial observability is why exploration is necessary—we must try actions to learn their value.

Regret: The Fundamental Performance Metric

Since we don’t know the true reward distributions, we can’t compute V(π) directly. Instead, we measure regret—how much worse we do compared to the optimal policy.

Definition

Let π* be the optimal policy that always chooses the best action for each context. The regret after T rounds is:

$R(T) = \sum_{t=1}^{T} r_t^* - \sum_{t=1}^{T} r_t = T \cdot V(\pi^*) - V(\pi)$

where rₜ* is the reward we would have gotten from the optimal action at time t.

Interpretation

Regret measures cumulative opportunity cost. If R(T) = 100 after 1000 rounds, we earned 100 less total reward than optimal.

Important: We want algorithms that minimize regret growth.

Sublinear Regret

An algorithm is good if regret grows sublinearly in T:

$R(T) = o(T)$

Equivalently, average regret goes to zero:

$\frac{R(T)}{T} \to 0 \text{ as } T \to \infty$

This means we’re converging to optimal performance. Linear regret R(T) = Θ(T) would mean we’re making a constant mistake forever—we never learn.

Typical Regret Bounds

The √T bound is essentially optimal—you can’t do better in the worst case. This comes from a fundamental information-theoretic tradeoff.

Visual Comparison of Regret Growth

Practical interpretation:

After 10,000 rounds:

• Linear O(T): ~10,000 cumulative regret (never learned)
• ε-greedy O(T^(2/3)): ~464 cumulative regret
• UCB O(√T log T): ~130 cumulative regret
• Thompson O(√T): ~100 cumulative regret

The difference compounds over time. In production with millions of decisions, UCB/Thompson save significant value vs ε-greedy.

Reward Models: Assumptions About Structure

Real-world reward functions are complex. We make modeling assumptions to make learning tractable.

Linear Reward Model

Assumption: Reward is a linear function of context features:

$r(x, a) = \theta_a^T x + \epsilon$

where:

• θₐ ∈ ℝᵈ is a weight vector for action a
• x ∈ ℝᵈ is the context vector
• ε ~ N(0, σ²) is noise

When this is reasonable:

✅ Features are well-engineered (embeddings, hand-crafted)
✅ Relationships are approximately linear in feature space
✅ Interpretability is important (coefficients show feature importance)

Limitations:

❌ Misses complex nonlinear interactions
❌ Requires good feature engineering upfront
❌ Can underfit when true function is highly nonlinear

Example: Content Recommendation

Context x: [user_age, user_tenure, content_freshness, 
            content_popularity, topic_match]
Action a: show_content (binary)

Linear model: 
r(x, show) = θ₁·age + θ₂·tenure + θ₃·freshness + 
             θ₄·popularity + θ₅·topic_match

Interpretation: If θ₃ = 0.8, each unit increase in freshness 
                adds 0.8 to expected reward

Generalized Linear Models (GLMs)

Assumption: Reward follows an exponential family distribution with linear predictor:

$\mathbb{E}[r \mid x, a] = g^{-1}(\theta_a^T x)$

where g is a link function.

Common cases:

Logistic regression (binary rewards):

$\mathbb{P}(r = 1 \mid x, a) = \sigma(\theta_a^T x) = \frac{1}{1 + e^{-\theta_a^T x}}$

Use when reward is click/no-click, conversion/no-conversion.

Poisson regression (count rewards):

$\mathbb{E}[r \mid x, a] = e^{\theta_a^T x}$

Use when reward is number of purchases, page views, etc.

Advantage: Captures bounded or discrete reward structures while maintaining interpretability.

Nonlinear Models (Neural Bandits)

Assumption: Reward is a complex nonlinear function:

$r(x, a) = f_{\theta_a}(x) + \epsilon$

where f is a neural network with parameters θₐ.

When this is necessary:

✅ Complex feature interactions that linear models can’t capture
✅ High-dimensional raw inputs (images, text)
✅ Sufficient data to train deep models

Challenges:

❌ Requires much more data
❌ Less interpretable (black box)
❌ Harder to quantify uncertainty (needed for exploration)
❌ Computationally expensive

When to use each:

The Exploration-Exploitation Tradeoff

This is the fundamental dilemma in bandit problems.

The Tension

Exploitation: Choose the action that has performed best so far. Maximize immediate reward based on current knowledge.

Exploration: Try actions with uncertain rewards. Gather information that might lead to better decisions later.

The conflict: Exploration sacrifices short-term reward for long-term learning. Exploitation maximizes current reward but risks missing better alternatives.

Toy Example

Two actions: A and B
Current estimates: 
  μ̂_A = 0.8 (based on 100 trials)
  μ̂_B = 0.6 (based on 10 trials)

Pure exploitation: Always choose A (highest estimate)
Problem: B's estimate is uncertain. True μ_B might be 0.9, 
         but we have sparse data.

Pure exploration: Choose randomly to gather data
Problem: While gathering data, we're not maximizing reward

Optimal balance: Choose A most of the time, occasionally try B 
                 until confident about its value

Mathematical Formalization

Define the exploitation gap as the difference between estimated best and true best:

$\Delta_t = \max_a \mathbb{E}[r \mid x_t, a] - \mathbb{E}[r \mid x_t, a_t]$

And the exploration benefit as information gain:

$I_t = \text{Reduction in uncertainty about } \theta$

Optimal policies balance these:

$a_t = \arg\max_a \left\{ \mathbb{E}[r \mid x_t, a] + \lambda \cdot I(a) \right\}$

where λ controls exploration intensity.

Visual Representation

Why This Matters in Practice

If you over-exploit (ε = 0.01):

• Converge quickly to locally optimal solution
• Miss potentially better alternatives
• Regret grows linearly if you lock onto suboptimal action

If you over-explore (ε = 0.5):

• Waste traffic on clearly inferior actions
• Slow convergence to optimal policy
• Users see more bad experiences

Optimal strategy: Exploration rate should decrease over time. Early on, uncertainty is high—explore more. Later, confidence grows—exploit more.

The Confidence-Bound Principle

Many algorithms use optimism under uncertainty: when unsure about an action’s value, assume it might be great.

Upper Confidence Bound (UCB)

Intuition: Add a confidence bonus to each action’s estimate. Choose the action with the highest upper confidence bound.

$a_t = \arg\max_a \left\{ \hat{\mu}_a + \sqrt{\frac{2 \log t}{n_a}} \right\}$

where:

• $\hat{\mu}_a$ = estimated mean reward for action a
• nₐ = number of times action a has been tried
• The √ term = confidence radius

Why This Works

• Well-explored actions (large nₐ): Confidence radius is small. Trust the estimate $\hat{\mu}_a$.
• Under-explored actions (small nₐ): Confidence radius is large. UCB is optimistic.
• This creates natural exploration: uncertain actions get inflated values and get tried.

The Math Behind the Confidence Radius

The √(2 log t / nₐ) comes from Hoeffding’s inequality:

$\mathbb{P}\left(|\hat{\mu}_a - \mu_a| > \sqrt{\frac{2 \log t}{n_a}}\right) \leq \frac{1}{t^4}$

With high probability (1 - 1/t⁴), the true mean is within the confidence radius. By using the upper bound, we ensure we don’t prematurely dismiss good actions.

Visual Intuition for UCB

Key insight: Action B has lower estimate (0.4 vs 0.5) but higher UCB (0.7 vs 0.6) due to uncertainty. UCB chooses B to reduce uncertainty—exactly what we want.

LinUCB: Extending to Contextual Bandits

For linear contextual bandits, UCB extends to:

$a_t = \arg\max_a \left\{ \theta_a^T x_t + \alpha \sqrt{x_t^T A_a^{-1} x_t} \right\}$

where:

• Aₐ = XₐᵀXₐ + λI (regularized Gram matrix)
• α = exploration parameter (typically 0.1-2.0)
• λ = regularization parameter (typically 1.0)

The confidence ellipsoid √(xᵀA⁻¹x) captures uncertainty in the parameter estimate θₐ.

Novel contexts (far from previously seen data) have large confidence radii, encouraging exploration in new regions of the feature space.

Thompson Sampling: Bayesian Perspective

Instead of confidence bounds, Thompson Sampling uses probability matching.

Algorithm

1. Maintain posterior distribution P(θₐ | data) for each action’s parameters
2. Sample θ̃ₐ ~ P(θₐ | data) for each action
3. Choose action with highest sampled reward: aₜ = argmaxₐ θ̃ₐᵀxₜ
4. Observe reward, update posteriors

Probability Matching

The probability of choosing action a equals the probability that a is optimal given current knowledge:

$\mathbb{P}(\text{choose } a) = \mathbb{P}(a \text{ is optimal} \mid \text{data})$

This naturally balances exploration and exploitation through Bayesian updating:

• Early: Posteriors are diffuse → high probability of sampling various actions → exploration
• Late: Posteriors concentrate → high probability of sampling best action → exploitation

No manual tuning of exploration rate needed. The uncertainty drives exploration organically.

Why It Works

Thompson Sampling achieves optimal O(√T) regret bounds for many problems while being simpler to implement than UCB variants. The Bayesian framework naturally handles uncertainty.

Example: Bernoulli Bandits

For binary rewards (click/no-click), use Beta-Bernoulli conjugacy:

Prior: θₐ ~ Beta(1, 1) (uniform)

Likelihood: r | θₐ ~ Bernoulli(θₐ)

Posterior after observing data:

$\theta_a \mid \text{data} \sim \text{Beta}(\alpha_a, \beta_a)$

where:

• αₐ = 1 + number of successes (clicks)
• βₐ = 1 + number of failures (no clicks)

To choose action: Sample θ̃ₐ ~ Beta(αₐ, βₐ) for each action, pick argmax θ̃ₐ.

For Linear Contextual Bandits

Assume Gaussian prior and noise:

Prior: θₐ ~ N(0, σ²I)
Noise: ε ~ N(0, σ²)

Posterior:

$\theta_a \mid D_a \sim \mathcal{N}(\hat{\theta}_a, \Sigma_a)$

where:

• $\hat{\theta}_a = (X^T X + \lambda I)^{-1} X^T r$ (ridge regression estimate)
• $\Sigma_a = \sigma^2(X^T X + \lambda I)^{-1}$ (posterior covariance)

To choose action: Sample $\tilde{\theta}_a \sim \mathcal{N}(\hat{\theta}_a, \Sigma_a)$, compute $\tilde{r}_a = \tilde{\theta}_a^T x_t$, pick $a_t = \arg\max_a \tilde{r}_a$.

The Fundamental Limits: Lower Bounds

Can we do better than O(√T) regret? No (in the worst case).

Theorem (Lai & Robbins, 1985)

For any algorithm, there exist problem instances where regret is at least:

$R(T) = \Omega(\log T)$

for stochastic bandits (MAB).

For contextual bandits with d-dimensional context and linear rewards:

$R(T) = \Omega(\sqrt{dT})$

Implication

Algorithms achieving O(√(dT log T)) are near-optimal. The log T factor is unavoidable slack, but the √(dT) matches the lower bound up to logarithmic factors.

Intuition for the √T Bound

You need to explore each action enough times to distinguish it from the optimal one. If the gap between rewards is Δ, you need roughly O(1/Δ²) samples per action. With K actions, total exploration cost is O(K/Δ²).

If you don’t know Δ in advance and must be robust to small gaps, worst-case regret is O(√(KT)).

This isn’t just theoretical—it explains why bandit algorithms can’t magically achieve zero regret. There’s a fundamental cost to learning under uncertainty.

Key Takeaways

Essential concepts:

Regret R(T) = cumulative opportunity cost vs optimal policy

• Goal: Achieve sublinear regret R(T) = o(T)
• Means average regret → 0 as T → ∞ (we’re learning)

Optimal algorithms achieve O(√(dT log T)) for d-dimensional contexts

• The √T term is unavoidable (information-theoretic limit)
• The √d factor is the cost of personalization
• Linear regret O(T) means algorithm never learns (failure)

Exploration-exploitation is fundamental—no free lunch

• Explore: Try uncertain actions to gather information
• Exploit: Choose known best action to maximize reward
• Tradeoff: Short-term reward vs long-term learning

Two principled approaches to balance:

• UCB: Optimism under uncertainty (confidence-based)
• Thompson: Posterior sampling (Bayesian)
• Both achieve near-optimal regret bounds

Reward model matters:

• Linear: Fast, interpretable, works for d < 500
• GLM: Handles bounded/discrete rewards
• Neural: Required for high-dim or complex nonlinear

Practical implications:

When to worry about math:

✅ Do worry: Algorithm selection, tuning α/λ, understanding regret curves
❌ Don’t over-optimize: Chasing last 1% theoretical gain—empirical performance matters more

Further Reading

Foundational papers:

• Finite-time Analysis of the Multiarmed Bandit Problem (Auer et al., 2002) - UCB1 algorithm and regret bounds
• Analysis of Thompson Sampling (Agrawal & Goyal, 2012) - Theoretical analysis
• Thompson Sampling for Contextual Bandits (Agrawal & Goyal, 2013) - Extension to contextual case

Textbooks:

• Bandit Algorithms (Lattimore & Szepesvári, 2020) - Comprehensive theoretical treatment (free online)
• Reinforcement Learning: An Introduction (Sutton & Barto) - Includes bandit chapters

Blog posts:

• Lil’Log: Multi-Armed Bandits - Excellent visual tutorial
• Jeremy Kun: Bandits Series - Mathematical blog series

Part 1 of 5: Understanding the Landscape

TL;DR: Contextual bandits bridge the gap between static A/B testing and full reinforcement learning. Use them when you need personalized, adaptive optimization with immediate feedback—but not for sequential decision-making or when you need clean causal inference. This post helps you decide if bandits are right for your problem.

Reading time: ~20 minutes

The Adaptive Optimization Gap

You’re running an A/B test to optimize your recommendation system. Variant A gets 50% of traffic, variant B gets 50%. You wait three weeks for statistical significance. Finally, results are in: B wins with 5.2% lift. You ship it to 100% of users.

But here’s what you missed: During those three weeks, 50% of your users got the inferior experience. And variant B might work great for power users but poorly for newcomers—your aggregate metric doesn’t capture this. You just left significant value on the table.

This is the adaptive optimization gap that contextual bandits solve.

The contextual bandit loop

Before diving deeper, understand the core interaction pattern:

The key difference from A/B testing: Step 4 happens immediately after each interaction. The policy learns and adapts continuously, not after weeks of fixed allocation.

Quick Start: Is This Guide For You?

✅ This series is for you if:

• You’re optimizing recommendations, rankings, or content selection
• You have >1000 decisions/day with measurable outcomes
• User preferences vary (personalization matters)
• You need faster learning than A/B testing provides

❌ Skip to alternatives if:

• Decisions are highly sequential (use RL instead)
• Feedback is delayed >1 week (use delayed RL)
• You have <100 decisions/day (use A/B testing)
• Context doesn’t matter (use multi-armed bandit)

What Are Contextual Bandits?

Starting Simple: The Casino Analogy

Multi-Armed Bandit (MAB): You walk into a casino with K slot machines. Each has an unknown payout rate. Your goal: maximize winnings over 1000 plays.

The challenge: explore (try machines to learn rates) vs. exploit (play the best one). Every player faces the same problem—machine 3 is either best for everyone or no one. No personalization.

The Real World Problem: The “best” choice depends on who you are and the situation.

News articles example: {Tech, Sports, Politics}

• MAB thinking: “Sports gets 15% CTR on average → show Sports to everyone”
• Reality: Tech enthusiasts click Tech 40% vs Sports 5%. Sports fans reverse this pattern.
• The insight: Optimal action changes based on context (user type).

Technical Definition

Contextual Bandit: An online learning algorithm for personalized decision-making under uncertainty.

Mathematical Formulation:

At each timestep t = 1, 2, …, T:

1. Observe context $x_t \in \mathbb{R}^d$ Context vector with d features (user demographics, item attributes, environment)
2. Choose action $a_t \in \mathcal{A} = {1, ..., K}$ Select from K available actions based on context
3. Receive reward $r_t \sim P(r \mid x_t, a_t)$ Observe stochastic reward from unknown distribution
4. Update model $\hat{f}(x, a) \approx \mathbb{E}[r \mid x, a]$ Learn to predict expected reward for any (context, action) pair
5. Repeat with continuous adaptation

Goal: Learn policy $\pi: \mathcal{X} \rightarrow \mathcal{A}$ that maximizes cumulative reward:

$V(\pi) = \mathbb{E}\left[\sum_{t=1}^{T} r_t \mid \pi\right]$

Key Differences from MAB:

Concrete Example: Content Recommendation

Setup:

Context x ∈ ℝ⁵: 
  x = [user_age, user_tenure_days, is_mobile, time_of_day, topic_affinity]
  Example: x₁ = [28, 450, 1, 14, 0.8]  
  (28yo, 450 days tenure, mobile, 2pm, 80% tech affinity)

Actions a ∈ {1, 2, 3}:
  a=1: article_A (tech content)
  a=2: article_B (sports content)  
  a=3: article_C (politics content)

Reward r ∈ {0, 1}:
  r = 1 if user clicks, r = 0 otherwise

How MAB Fails:

After 1000 uniform trials:
  article_A: 120/333 = 36% CTR
  article_B: 133/333 = 40% CTR ← Highest average
  article_C: 97/334 = 29% CTR

MAB decision: Always show article_B
Problem: Misses that tech users prefer A, sports fans prefer B

How Contextual Bandit Succeeds:

Learn function $P(\text{click} \mid x, a) = f(x, a)$:

Tech enthusiast: x₁ = [28, 450, 1, 14, 0.8]
  f(x₁, article_A) = 0.75 ← Optimal for this user
  f(x₁, article_B) = 0.15
  f(x₁, article_C) = 0.10
  → Recommend article_A

Sports fan: x₂ = [45, 1200, 0, 19, 0.2]
  f(x₂, article_A) = 0.12
  f(x₂, article_B) = 0.68 ← Optimal for this user
  f(x₂, article_C) = 0.20
  → Recommend article_B

Result: Personalized CTR ~55% vs MAB’s global 40%.

The Learning Challenge

What makes this hard:

1. Partial feedback: Only observe reward for chosen action
   • Chose A, saw reward 1 → What would B have given? Unknown (counterfactual)
   • Must explore alternatives to learn their value
2. Function approximation: Learning $f(x,a)$ across continuous context space
   • Can’t try every possible x (infinite contexts)
   • Must generalize from observed data to unseen contexts
3. Exploration-exploitation tradeoff:
   • Explore: Try uncertain actions to gather information (short-term cost)
   • Exploit: Choose known best action to maximize reward (long-term requires learning)
   • Balance: Neither pure exploration nor pure exploitation is optimal

Optimal algorithms (LinUCB, Thompson Sampling) achieve $O(\sqrt{dT \log T})$ regret—near-optimal convergence to best policy.

Where Contextual Bandits Fit

Critical distinction from RL:

Contextual bandits are stateless. Each decision is independent.

Contextual Bandit (memoryless):
  t=1: x₁ → a₁ → r₁
  t=2: x₂ → a₂ → r₂  (x₂ independent of a₁)

Full RL (stateful):
  t=1: s₁ → a₁ → r₁ → s₂  
  t=2: s₂ → a₂ → r₂ → s₃  (s₂ caused by a₁)

Example: Recommending an article doesn’t change tomorrow’s user → bandit. Playing chess (moves change board state) → RL.

Why Contextual Bandits Matter

1. Automatic Personalization

Traditional:

Run A/B test → Analyst finds "B wins for 18-25, A for 40+" 
→ Engineer 2-variant system → Repeat for each feature

Contextual bandit:

Deploy with features → Learns f(age, location, ...) → best action
→ Discovers "A wins for 18-25 + mobile + evening" automatically

2. Sample Efficiency

Over 10,000 decisions:

• A/B test (50/50 split): ~5,000 suboptimal → 2,500 lost conversions
• Contextual bandit: ~500 exploratory → 100 lost conversions
• Savings: 2,400 conversions (48% improvement)

3. Continuous Adaptation

Handles non-stationarity:

• User preferences shift (trends)
• Seasonality (December ≠ July)
• Competition (dynamic pricing)
• External shocks (COVID)

Bandit automatically detects changes and adapts. A/B test remains static until next experiment.

Real-world impact:

• Netflix: 20-30% engagement lift (personalized artwork)
• E-commerce: 5-15% revenue increase (dynamic pricing)
• Healthcare: 10-25% outcome improvement (treatment personalization)

Quick Decision Guide

✅ Use contextual bandits when:

1. Context predicts optimal action (heterogeneous users/situations)
2. Decisions are independent (stateless)
3. Feedback is immediate (seconds to days)
4. Scale justifies complexity (1000+ decisions/day)
5. Reward is measurable and well-defined

❌ Use alternatives when:

• No context → MAB
• Sequential dependencies → Full RL
• Delayed rewards (>1 week) → Delayed RL
• Need causal estimates → A/B testing
• Low scale (<100/day) → A/B testing

The Problem with Static Experimentation

A/B testing is the gold standard for causal inference, but has fundamental limitations for adaptive optimization:

Core Limitations

1. Fixed allocation wastes traffic 50/50 split for entire experiment. Bad variants get equal traffic until significance.

2. No personalization Aggregate metrics hide segment-level differences. Manual segmentation requires pre-defining groups and exponentially splits traffic.

3. Slow adaptation Wait weeks for statistical power. System doesn’t learn during experiment.

4. Sequential testing overhead Testing 5 variants requires either multiple rounds (slow) or 5-way split (massive sample size needed).

Reality: Production systems need faster, adaptive, personalized optimization.

Understanding the Decision-Making Spectrum

Contextual Bandit vs Multi-Armed Bandit

The distinction: Do you have side information about the decision context?

MAB: Learn global optimum. Same best action for everyone.

When MAB suffices:

• Homepage layout (users identical)
• Global search ranking (uniform population)
• Fixed-time promotions (time is only context)

Example MAB failure:

Notification timing for fitness app:

MAB: Learns 7 PM is optimal on average (60% prefer evening, 40% prefer morning)
→ Converges to 7 PM (majority wins)
→ 40% of users get suboptimal forever

Contextual bandit: Uses activity patterns
→ Morning people → 6 AM, evening people → 7 PM
→ Personalized optimal for all

When CB is necessary:

• Product recommendations (preferences vary by user)
• Ad selection (relevance is contextual)
• Content ranking (interests heterogeneous)

Critical question: Is personalization lift > complexity cost? If users homogeneous → MAB. If diverse → CB.

Contextual Bandit vs A/B Testing

The Fundamental Tradeoff

When to Use Each

A/B testing:

• Irreversible platform decisions (pricing, core UX)
• Need clean causal estimates (reporting/legal)
• Testing hypotheses, not optimizing
• Lack ML infrastructure
• Abundant sample size

Contextual bandits:

• Optimizing recommendations/rankings at scale
• High user heterogeneity
• Quick iteration on many variants
• Significant opportunity cost
• ML engineering resources available

Hybrid approach:

1. Week 1-2: A/B test (validate personalization beats baseline)
2. Week 3+: Deploy bandit (continuous optimization)
3. Periodic A/B audits (validate effectiveness)

Contextual Bandit vs Full Reinforcement Learning

The boundary: Do current actions affect future rewards beyond immediate impact?

Contextual Bandits

One-step decisions. Each round:

1. Observe context xₜ
2. Choose action aₜ
3. Receive reward rₜ
4. Episode ends (next context independent)

Valid when:

• Recommendations (showing A doesn’t prevent showing B later)
• Ad serving (A doesn’t change future ad inventory)
• Email timing (7 PM today doesn’t affect tomorrow’s options)

Full RL

Sequential decisions. Markov Decision Process:

• States sₜ, actions aₜ
• Transitions P(sₜ₊₁|sₜ, aₜ) ← Actions affect future states
• Learn policy π(s) maximizing discounted cumulative reward

Required when:

• Game playing (moves change board)
• Inventory management (purchases deplete stock)
• Dialogue systems (responses change conversation)

The MDP Structure Test

Ask: “If I choose action A, does that change future actions available or desirable?”

• No → Bandit (ad A doesn’t change ad inventory)
• Weakly → Borderline (recommendation A slightly shifts interests)
• Strongly → RL (chess move A fundamentally changes options)

Sample Complexity Comparison

Sample complexity answers: “How many interactions needed to learn a near-optimal policy?”

Different from regret (cumulative loss), it measures the learning time itself.

Where these come from:

• Δ² in denominator: From hypothesis testing (Hoeffding’s inequality). To distinguish options with gap Δ requires O(1/Δ²) samples. Smaller gaps need exponentially more data.

• d factor (contextual): Linear regression in d dimensions needs O(d/ε²) samples to achieve accuracy ε. With gap Δ, setting ε ~ Δ gives O(d/Δ²).

• |S|² (MDP): Must learn P(s’|s,a) for all state pairs. Need to visit each (s,a) multiple times and estimate |S| transition probabilities per pair = O(|S|²).

• (1-γ)³ (MDP): Discount γ determines effective horizon 1/(1-γ). Longer horizons need more samples to propagate value information. Cube comes from value error + mixing time + Bellman error propagation.

Sources:

• MAB: Auer et al. (2002), Lai & Robbins (1985)
• Contextual: Li et al. (2010), Abbasi-Yadkori et al. (2011)
• MDP: Kakade (2003), Strehl & Littman (2008)

Concrete example (Δ = 0.1 gap, 10K decisions):

MAB with K=10 arms:
  Sample complexity: 10 × log(10,000) / 0.1² ≈ 9,200 samples
  
Contextual with d=20 features:
  Sample complexity: 20 × log(10,000) / 0.1² ≈ 18,400 samples
  
Tabular MDP with |S|=100 states, |A|=10, γ=0.9:
  Sample complexity: 100² × 10 / (0.1)³ × 0.001 ≈ 10,000,000 samples

Why this matters: RL sample complexity is 100-1000× higher than contextual bandits for equivalent problems. This isn’t just theory—it explains why recommendation systems learn in days (bandits) while game AI needs millions of episodes (RL).

RL sample complexity explodes with state space size and planning horizon. For large state spaces or long horizons, you need millions of samples.

Real-World Application Domains

1. Recommendation Systems

Context: User demographics, history, session behavior, item attributes Actions: Which items to recommend Reward: Clicks, purchases, watch time

Example: Netflix artwork personalization

2. Online Advertising

Context: User profile, page content, ad attributes Actions: Which ad to display Reward: Clicks (CTR) or conversions (CPA)

Challenges: Delayed conversion, budget constraints, dynamic bids

3. Content Ranking

Context: User preferences, content recency, feed position Actions: Order of content items Reward: Engagement (clicks, dwell time, shares)

Example: LinkedIn feed ranking

4. Personalized Notifications

Context: User timezone, engagement history, content urgency Actions: Send time (discrete slots) Reward: Open rate, click-through

Challenges: Delayed feedback, notification fatigue

5. Clinical Trials

Context: Patient demographics, medical history, biomarkers Actions: Treatment assignment Reward: Health outcomes

Challenges: Safety constraints, small samples, delayed outcomes, ethics

Decision Framework: Is Your Problem a Contextual Bandit?

✅ Requirements Checklist

• Independent decisions: Choices don’t affect future options/rewards
• Immediate feedback: Observe reward within seconds to days
• Heterogeneous contexts: Different situations favor different actions
• Multiple options: At least 2-3 actions to choose between
• Sufficient scale: 1000+ decisions/day for worthwhile learning
• Measurable reward: Clear metric capturing value

❌ Red Flags

• Sequential dependencies → Full RL
• Delayed feedback (>1 week) → Delayed RL
• Homogeneous population → MAB
• Only 1-2 total decisions → A/B testing
• Ambiguous reward → Fix measurement first

Decision Tree

Key Takeaways

Use contextual bandits when:

• Decisions independent (today doesn’t constrain tomorrow)
• Feedback immediate (seconds to days)
• Personalization matters (users/contexts differ)
• Scale sufficient (1000+ decisions/day)

Choose alternatives when:

• A/B testing: Need causal inference, low traffic, risk-averse
• MAB: Homogeneous population
• Full RL: Sequential dependencies, long horizons

Fundamental tradeoffs:

• A/B: Clean inference ↔ Sample efficiency
• Bandits: Adaptation ↔ Biased data
• RL: Sequential planning ↔ Sample complexity

Quick validation: ✅ Independent? ✅ Immediate? ✅ Heterogeneous? ✅ Multiple options? ✅ Scale? ✅ Measurable?

All ✅ → Contextual bandit likely right choice

Further Reading

Alternatives:

• A/B Testing: Trustworthy Online Controlled Experiments (Kohavi et al.)
• MAB: Introduction to Multi-Armed Bandits (Lattimore & Szepesvári)
• RL: Sutton & Barto’s RL Book

Contextual bandits:

• LinUCB paper (Li et al., 2010)
• Lil’Log: Multi-Armed Bandits

Production:

• Netflix: Artwork Personalization

Part 3 of 5: From Theory to Code

TL;DR: This post provides complete, production-ready implementations of the essential contextual bandit algorithms: ε-greedy (baseline), UCB/LinUCB (confidence-based), and Thompson Sampling (Bayesian). Includes algorithm selection guide with default hyperparameters, when to use each method, and practical tuning strategies.

Reading time: ~25 minutes

Introduction: From Theory to Practice

Parts 1 and 2 covered when to use contextual bandits and the mathematical foundations. Now it’s time to implement.

This post provides complete, working code for the core algorithms you’ll actually deploy:

1. ε-greedy: Simplest baseline (start here)
2. UCB/LinUCB: Confidence-based exploration (production workhorse)
3. Thompson Sampling: Bayesian posterior sampling (often best empirically)

Each algorithm includes:

• Full Python implementation
• When to use it
• Hyperparameter defaults and tuning
• Strengths and limitations

By the end, you’ll know which algorithm fits your problem and have code ready to run.

Algorithm Selection: Quick Start Guide

Before diving into implementations, use this table to pick your starting point:

Start simple: Begin with ε-greedy to validate your setup. Upgrade to LinUCB or Thompson Sampling once you’ve confirmed the basics work.

ε-greedy: The Simplest Baseline

ε-greedy is the go-to baseline. It’s easy to implement, easy to explain to stakeholders, and works reasonably well.

Algorithm

At each round t:
1. With probability ε: Choose random action (explore)
2. With probability 1-ε: Choose action with highest estimated reward (exploit)
3. Observe reward, update estimates

Mathematical Formulation

Part 2 of 4: The Industry Standard

TL;DR: PPO (Proximal Policy Optimization) is the backbone of RLHF, powering ChatGPT, Claude, and most aligned LLMs. It provides stable training through clipped objectives and flexible advantage estimation via GAE. But this comes at a cost: four models (policy, reference, value, reward) straining GPU memory. Understanding PPO deeply reveals why simpler alternatives like GRPO emerged.

Reading time: ~28 minutes

Prerequisites: Part 1: RL Foundations covers policy gradients, value functions, and advantage estimation.

From REINFORCE to PPO

In Part 1, we saw that policy gradients are powerful but suffer from high variance. Actor-critic methods add a value function to reduce variance, but introduce their own challenges: training two networks that must improve together without destabilizing each other.

PPO (Proximal Policy Optimization) addresses stability through a deceptively simple idea: limit how much the policy can change in a single update. If updates are small enough, the value function doesn’t become stale, and training remains stable.

But “small updates” is vague. How do we formalize it? And how do we enforce it efficiently?

The Trust Region Idea

The Problem with Large Updates

Consider what happens when we take a policy gradient step:

1. Collect trajectories using current policy $\pi_{\theta_{\text{old}}}$
2. Compute advantages using value function $V_\psi$
3. Update: $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

The problem: our advantages were computed for $\pi_{\theta_{\text{old}}}$, but we’re evaluating actions under $\pi_\theta$. If $\theta$ changes too much, the advantages become stale—they no longer reflect the true quality of actions under the new policy.

This can cause:

• Overshooting: A good action gets reinforced too strongly
• Policy collapse: All probability mass shifts to one action
• Training instability: Performance oscillates wildly

Trust Regions: The Concept

A trust region is a constraint on how much the policy can change:

$\max_\theta J(\theta) \quad \text{subject to} \quad D(\pi_\theta, \pi_{\theta_{\text{old}}}) \leq \delta$

where $D$ is some distance measure between policies, and $\delta$ is a small threshold.

Within the trust region, our local approximation (using old advantages) is valid. Outside it, we can’t trust our gradient estimates.

TRPO: The Principled Approach

Trust Region Policy Optimization (TRPO) uses KL divergence as the distance:

$\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A^{\pi_{\text{old}}}(s, a)\right] \quad \text{s.t.} \quad \mathbb{E}\left[\text{KL}(\pi_{\theta_{\text{old}}} \| \pi_\theta)\right] \leq \delta$

TRPO solves this constrained optimization using second-order methods (conjugate gradient + line search). It’s theoretically elegant but:

• Computationally expensive
• Complex to implement
• Hard to parallelize

This motivated PPO: can we get trust region benefits with first-order methods?

From TRPO to PPO: The Clipped Objective

PPO’s insight: instead of enforcing a hard constraint, clip the objective so that updates beyond the trust region give no additional benefit.

The Importance Sampling Ratio

First, we rewrite the policy gradient using importance sampling:

$J(\theta) = \mathbb{E}_{s, a \sim \pi_{\theta_{\text{old}}}}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A^{\pi_{\text{old}}}(s, a)\right]$

Define the importance ratio:

$\rho_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$

When $\theta = \theta_{\text{old}}$, we have $\rho = 1$. As $\theta$ diverges from $\theta_{\text{old}}$, $\rho$ moves away from 1.

The Clipped Surrogate Objective

PPO’s objective clips the ratio to prevent large updates:

$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\left(\rho_t(\theta) A_t, \, \text{clip}(\rho_t(\theta), 1-\varepsilon, 1+\varepsilon) A_t\right)\right]$

Let’s unpack this carefully.

How Clipping Works

Case 1: Positive advantage ($A_t > 0$) — the action was good

We want to increase $\pi_\theta(a_t|s_t)$, which increases $\rho_t$.

• If $\rho_t < 1 + \varepsilon$: The objective is $\rho_t A_t$, so gradients push $\rho_t$ higher
• If $\rho_t > 1 + \varepsilon$: The clipped term becomes $(1+\varepsilon) A_t$, which is constant. No gradient — we’ve increased the action probability enough

Case 2: Negative advantage ($A_t < 0$) — the action was bad

We want to decrease $\pi_\theta(a_t|s_t)$, which decreases $\rho_t$.

• If $\rho_t > 1 - \varepsilon$: The objective is $\rho_t A_t$, so gradients push $\rho_t$ lower
• If $\rho_t < 1 - \varepsilon$: The clipped term becomes $(1-\varepsilon) A_t$, which is constant. No gradient — we’ve decreased the action probability enough

The “Pessimistic” Bound

The $\min$ in the objective creates a pessimistic bound: we take the worse of (unclipped, clipped) for each term.

• For good actions: Can’t get more than $(1+\varepsilon)A_t$ benefit
• For bad actions: Can’t get less than $(1-\varepsilon)A_t$ penalty

This is conservative by design—it prevents the optimizer from exploiting stale advantages.

Why $\varepsilon = 0.2$?

The canonical choice is $\varepsilon = 0.2$, allowing $\rho$ to range from 0.8 to 1.2. This means:

• Action probabilities can at most increase by 20% (relative)
• Action probabilities can at most decrease by 20% (relative)

per update step. Empirically, this balances stability and learning speed. Some implementations use 0.1 for more conservative updates.

Key Insight: PPO’s clipping creates a “soft trust region” using only first-order optimization. The policy can change, but changes beyond $\varepsilon$ provide no additional gradient signal. It’s not a hard constraint, but it works remarkably well in practice.

Generalized Advantage Estimation (GAE)

PPO pairs the clipped objective with Generalized Advantage Estimation (GAE), a flexible way to estimate advantages.

The Bias-Variance Tradeoff

Recall from Part 1 that we can estimate advantages in different ways:

Monte Carlo (MC): $\hat{A}_t^{\text{MC}} = G_t - V(s_t) = \sum_{k=0}^{T-t} \gamma^k r_{t+k} - V(s_t)$

• Unbiased: Uses actual returns
• High variance: Includes all future randomness

Temporal Difference (TD): $\hat{A}_t^{\text{TD}} = r_t + \gamma V(s_{t+1}) - V(s_t) = \delta_t$

• Biased: Uses estimated $V(s_{t+1})$ instead of true expected return
• Low variance: Only one step of randomness

GAE: Interpolating Between MC and TD

GAE introduces parameter $\lambda \in [0, 1]$ to interpolate:

$\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{k=0}^{\infty} (\gamma\lambda)^k \delta_{t+k}$

where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the TD error.

Special cases:

• $\lambda = 0$: $\hat{A}_t = \delta_t$ (TD, low variance, high bias)
• $\lambda = 1$: $\hat{A}_t = \sum_k \gamma^k \delta_{t+k} = G_t - V(s_t)$ (MC, high variance, low bias)

Practical Computation

GAE can be computed efficiently backwards through the trajectory:

$\hat{A}_t = \delta_t + \gamma\lambda \hat{A}_{t+1}$

Starting from the end of the episode ($\hat{A}_T = 0$ or $\hat{A}_T = \delta_T$), we recurse backwards.

def compute_gae(rewards, values, gamma=0.99, lam=0.95):
    """
    Compute GAE advantages.
    
    Args:
        rewards: [T] rewards at each timestep
        values: [T+1] value estimates (includes bootstrap V(s_T+1))
        gamma: Discount factor
        lam: GAE lambda parameter
    
    Returns:
        advantages: [T] GAE advantage estimates
    """
    T = len(rewards)
    advantages = torch.zeros(T)
    last_adv = 0
    
    for t in reversed(range(T)):
        # TD error: δ_t = r_t + γV(s_{t+1}) - V(s_t)
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        # GAE recursion: A_t = δ_t + γλ A_{t+1}
        advantages[t] = delta + gamma * lam * last_adv
        last_adv = advantages[t]
    
    return advantages