An empirical research engineering project exploring reinforcement learning with verifiable rewards. The work combines model and training foundations with a reproducible SFT baseline, GRPO, controlled ablations, uncertainty analysis, and systematic failure analysis.

Structure

Model foundations

• decoder-only transformer
• attention
• positional encodings
• normalization
• feed-forward layers
• KV cache
• sampling
• correctness and numerical tests

Reference explainers: attention · positional encoding · transformer architecture. Close-out article: A Transformer from Raw Tensors — What 112 Tests Taught Me.

Training foundation

• configuration
• data pipeline
• training loop
• tiny reproducible training run
• metrics
• evaluation
• CLI

A training substrate that can be re-run, seed-swept, and compared — not a large-scale pretraining project.

RLVR study

• Qwen 2.5-0.5B
• GSM8K
• SFT baseline
• verifier
• GRPO
• evaluation

Rollout generation, advantage estimation, and the GRPO objective implemented explicitly so each moving part can be inspected in isolation.

Empirical analysis

• controlled ablations
• group size
• KL regularization
• format reward
• per-seed variability
• uncertainty
• failure analysis

The output is evidence about how the system behaves and where it breaks, not a leaderboard number.

Follow along

• Foundations — reference explainers on the model foundations.
• Blog posts — empirical write-ups as each part lands: what was measured, what varied across seeds, and where the method failed.
• Bits — short observations published as the work progresses.

Historical work on PPO, GRPO, and GDPO — the Reinforcement Learning for LLMs series — remains published and accessible under Writing.