Day 21 — Bonus 4
May 1, 2026
Agent Fine-tuning & Reinforcement Learning — GRPO, RLVR & the Production Training Loop
For most of 2024 the agent stack was a prompting problem. By mid-2026 the leaders — DeepSeek-R1, OpenAI o3, Claude 4.5, Qwen3 — have all moved one floor down: they are agents because they were trained to be agents, with reinforcement learning against verifiable rewards, multi-turn rollouts, and tool-call traces. This is the layer where the next 18 months of agent capability is being decided. If you cannot speak about GRPO, RLVR, and reward hacking, you are talking about agents from the seat of the prompt engineer, not the seat of the trainer.
Curriculum
Bonus · Day 21 of 17 + 4
Core Concept
From Prompted Agent to Trained Agent — What RL Actually Does to a Model
Analogy first. Imagine teaching someone to play chess. You can hand them the rule book (pre-training), let them watch grandmaster games (supervised fine-tuning), or you can let them play 10,000 games against an opponent and after each game tell them only one thing: “you won” or “you lost”. The third path is reinforcement learning, and the surprising result of 2024–2025 is that — for tasks where you can cheaply check the answer — this is by far the most powerful way to make a base language model better at multi-step reasoning and tool use. DeepSeek-R1 is the proof: a base model + a reward signal of “is the answer correct?” on math and code, and reasoning behaviour emerges without any human-written chain-of-thought examples.
Mechanism. A modern agent training run looks like this. Start with a base model (often already SFT’d on instruction data). For each prompt in a batch, sample G completions from the current policy — the model itself acts as the policy in the RL sense. For each completion, run a verifier: a function that produces a scalar reward. The verifier might be a unit test (does this code pass?), a math checker (does this answer equal 42?), an LLM judge, or a tool-trace checker (did the agent call get_weather with the right city?). Then update the model parameters so that high-reward completions become more likely and low-reward completions become less likely, while staying close to the previous version of the model (the “reference”) so we don’t catastrophically forget. Repeat for tens of thousands of steps. That update step is where PPO, GRPO, DPO, and friends differ.
The three families an engineer must distinguish:
The three policy-optimization families
Proximal Policy Optimization — the workhorse
Why did every RLHF stack from 2022–2024 use PPO?
Trains four models simultaneously: the policy (the LLM you’re fine-tuning), a reference policy (frozen copy), a reward model (predicts a scalar from text), and a value model / critic (estimates expected return per token). For each completion, the critic provides a per-token baseline so you know which tokens contributed to a good outcome. Stable, well-understood, hungry for memory — you’re holding two copies of a 70B model in optimizer state.
Group Relative Policy Optimization — no critic, group baseline
Why did DeepSeek invent GRPO and why is it everywhere now?
Sample G completions per prompt (typically 8–64). The reward of each is normalised within the group — advantage = (reward − group mean) / group std. No critic model needed: the group itself is the baseline. Memory drops by ~25%, training is more stable on long-CoT outputs, and for verifiable-reward tasks (math, code) it works as well or better than PPO. Introduced in DeepSeekMath (Shao et al., 2024) and made famous by R1 (2025).
Direct Preference Optimization — offline, no rollouts
When do you skip RL entirely?
If you have pairs of (chosen, rejected) completions, DPO directly fits the policy via a closed-form contrastive loss — mathematically equivalent to an implicit reward model + RL, no sampling at training time. Cheap, stable, but bounded by the quality of your preference dataset and offline by construction. Most production preference fine-tuning in 2024–2025 was DPO; the agentic-RL frontier in 2026 has moved back to GRPO because you cannot DPO over a tool-call trajectory.
RLHF vs RLVR — the 2024 split
Until 2023 the only large-scale RL recipe for LLMs was RLHF (Reinforcement Learning from Human Feedback): humans rank pairs of model outputs, you train a reward model on those rankings, then optimise the LLM against the reward model. This is what aligned ChatGPT and Claude. The problem — well-articulated by Karpathy — is that RLHF rewards vibes: the reward model is itself an LLM that learns “what humans tend to thumbs-up”, which the policy quickly learns to game. RLHF runs collapse after a few hundred steps because the model finds reward-model exploits.
The 2024 unlock was RLVR (Reinforcement Learning from Verifiable Rewards), introduced as a named technique in Tülu 3 (Lambert et al., AllenAI, Nov 2024) and made famous by DeepSeek-R1 (Jan 2025). The key change: replace the reward model with a deterministic checker. Math problems have ground-truth answers. Code problems have unit tests. Agentic tasks have “did the function execute and return the right value?”. Verifiable rewards cannot be hacked the same way — either the answer matches or it doesn’t — and they enable training runs that go for tens of thousands of steps without collapse. The cost is that you can only train on tasks where you have a verifier.
# The modern post-training pipeline (2025-2026)
base LLM (pretrained on internet)
↓
SFT on curated instructions teaches format, basic helpfulness
↓
DPO / RLHF on prefs teaches style, harmlessness, tone
↓
RLVR / GRPO on verifiable teaches reasoning, tool use, agent loops
↓ ← THIS is the 2026 frontier
deployed agent
What changes when the task is multi-turn (agentic)
Single-turn RL is solved enough that it’s a config option in TRL. Agentic RL is harder for three concrete reasons. (1) Long horizons: a SWE-bench task takes 30+ tool calls; the reward arrives at the end. Credit assignment becomes the dominant problem — which of those 30 actions actually mattered? (2) Stochastic environments: the same action can succeed or fail depending on the state of a sandbox, file system, or web page. Reward variance explodes. RAGEN (Wang et al., 2025) calls the failure mode the “Echo Trap”: gradients spike and the model collapses onto a single shallow strategy. (3) Tool calls and reasoning interleaved: the model emits text, calls a tool, receives output, emits more text. You need a rollout engine that can pause, execute tools, and resume — not just an autoregressive sampler. This is exactly what verl, RAGEN, and OpenPipe ART exist to build.
A useful test of an engineer’s grasp: ask them where the gradient comes from in a multi-turn agent rollout when only the final tool call passes. If they say “each action token gets its share of the trajectory advantage”, ask how it’s normalised. The right answer involves group baselines (GRPO-style) plus turn-level masking so tool-output tokens don’t receive gradient. If they hand-wave, they’ve never trained an agent.
Numbers Worth Memorizing
~25%
Memory savings of GRPO over PPO at training time — no critic model loaded (DeepSeekMath, 2024)
10×
More RL training compute used for OpenAI o3 vs o1, with continued performance gains (OpenAI, 2025)
8–64
Typical group size G for GRPO rollouts — the “tournament” the model competes against itself in
Anchor for sales conversations: a single GRPO training run for a 7B agent on a domain task typically costs $5–30K in cloud compute. A SOTA-class agent fine-tune (Qwen-3-32B, multi-turn, 50K verifier-checked rollouts) is in the $100–300K range. This is the math that makes RL fine-tuning viable for vertical AI startups for the first time.
Papers to Know
Three Papers That Defined the Agent-RL Stack
arXiv preprint
Seminal
2024 (DeepSeek-AI)
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. (DeepSeek-AI) · arXiv 2402.03300
Introduced Group Relative Policy Optimization (GRPO), a variant of PPO that drops the critic model and uses a group of completions as the baseline instead. Trained DeepSeekMath-7B with GRPO on math benchmarks and matched the performance of much larger closed models. The paper’s technical contribution — group-relative advantage estimation — is the algorithm that powers virtually every reasoning-model RL run today.
Why it matters: This is the paper everyone cites when they say “GRPO”. Without it, R1 doesn’t exist, OpenPipe ART doesn’t exist, the 2025 reasoning-model wave doesn’t exist. Read sections 3 and 4; the rest is benchmarks.
arXiv 2402.03300
Published
Recent
Nature 2025 / arXiv 2501.12948
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI team (Daya Guo et al.) · arXiv 2501.12948 · published in Nature 2025
Showed for the first time that pure RL with verifiable rewards, applied to a base model with no SFT cold start, can elicit long chain-of-thought reasoning — the “R1-Zero” result. Used GRPO at scale on math and coding tasks with rule-based rewards (correct answer + correct format). Reasoning behaviour, including self-correction and reflection, emerged spontaneously. The full R1 model adds a brief SFT cold start for readability, then continues with RL.
Why it matters: Demolished the assumption that you need millions of human chain-of-thought examples to teach reasoning. The reproducibility of R1-Zero by open-source labs in February 2025 (Open-R1, TinyZero) became the most-watched ML story of the year. Every reasoning model since — o3, Claude 4 Sonnet thinking, Gemini 2.5 thinking — uses some flavour of this recipe.
arXiv 2501.12948
arXiv preprint
Recent
2025 (Northwestern et al.)
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
Zihan Wang, Kangrui Wang, Qineng Wang, et al. (Northwestern, Stanford, UW, MSR) · arXiv 2504.20073
Frames the multi-turn agent training problem and introduces StarPO (State-Thinking-Actions-Reward Policy Optimization), a trajectory-level RL framework, plus an open-source system (RAGEN) implementing it across ten environments — Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, Sudoku. Diagnoses the “Echo Trap” failure: in stochastic multi-turn settings, naive trajectory-level GRPO collapses onto repetitive shallow strategies; the proposed StarPO-S adds trajectory filtering, gradient stabilization, and reasoning-aware reward shaping.
Why it matters: The first paper to seriously characterise why multi-turn agent RL is harder than single-turn reasoning RL, and to release a clean reference implementation. If you want to train an agent (not a chatbot), this is the paper that tells you what will go wrong before it does. Inspired Search-R1, Agent-R1, and is part of the September 2025 agentic-RL survey’s recommended reading.
arXiv 2504.20073
Bonus — the survey to skim: The Landscape of Agentic Reinforcement Learning for LLMs: A Survey (Chen et al., arXiv 2509.02547, Sep 2025) is the most comprehensive recent map of the field, organising it by capability (planning, tool use, memory, reasoning, self-improvement, perception) and by domain. Read it to know what exists; read R1 + RAGEN to know how it works.
GitHub Pulse
Six Repos That Define the Stack
Hugging Face’s post-training library. Implements SFTTrainer, GRPOTrainer, DPOTrainer, RewardTrainer in a familiar Transformers idiom.
The default starting point for anyone fine-tuning open-weights models. GRPOTrainer is a drop-in — if you can write a reward function, you can launch an R1-style run on a single 8×H100 box.
ByteDance’s “HybridFlow” RL post-training framework. Supports PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, DrGRPO. Scales to 671B models on hundreds of GPUs.
The serious-money industrial stack. Adopted by Anyscale, ByteDance, Shanghai AI Lab, Tsinghua, Berkeley, UCLA. Accepted to EuroSys 2025 as the HybridFlow paper. If you’re training a frontier model, you’re probably on verl.
Agent Reinforcement Trainer. Train multi-step agents on real-world tasks with GRPO. OpenAI-compatible client + GPU server architecture; integrates with LangGraph.
The pragmatic agent-RL play for product teams: drop into your existing LangGraph or function-calling app, run rollouts in production, train on what worked. OpenPipe’s headline result — a 14B Qwen fine-tune beating o3 on a domain task with 5× lower latency — is the canonical “why bother” demo.
Reference implementation of StarPO from the RAGEN paper. Ten environments built in (Sokoban, WebShop, FrozenLake, etc.). Trajectory-level multi-turn RL with reasoning traces.
If you want to read production-quality multi-turn agent RL code rather than just run it, this is the cleanest available. The Echo Trap diagnostics are worth studying even if you never train.
AllenAI’s reference codebase for the Tülu 3 recipe. End-to-end SFT → DPO → RLVR pipeline with reproducible scripts and datasets.
The textbook implementation of post-training. If you’ve never trained an LLM before and want to, this is the repo to fork — everything is documented, datasets are released, and the recipe matches what AllenAI used to beat GPT-4o-mini with Llama-3.1.
Curated index of every RLVR paper, codebase, and dataset since DeepSeek-R1. Updated weekly through 2025–2026.
Not a training framework — a reading list. If you have one hour to catch up on a year of agent-RL literature, scan the README.
Star counts approximate as of late April 2026. The agent-RL ecosystem is moving fast enough that the leaderboard will look different by Q3.
Community Pulse
What the Practitioners Are Actually Saying
Andrej Karpathy · on the limits of RL
Karpathy has argued that RLHF “isn’t really reinforcement learning” — it’s a vibe-check trained against a learned reward model that the policy quickly learns to game. He’s described himself as “bullish on environments and agentic interactions but bearish on reinforcement learning specifically”.
Recurring theme across his July 2025 X thread on RL scaling and his 2025 LLM Year in Review post; consistent with his Dwarkesh Patel interview “AGI is still a decade away”. The nuance: he is positive on RLVR as an intermediate technique, sceptical that it is the final shape of how agents will learn.
Lilian Weng · “Why We Think” (May 1, 2025)
Weng’s Lil’Log post argued that RL on automatically checkable solutions is what unlocked test-time-compute scaling: train against verifiers and the model spontaneously learns to break problems into intermediate steps and search.
From lilianweng.github.io/posts/2025-05-01-thinking/. The post is the cleanest single-author synthesis of the o1 / R1 era and is worth a careful read for the framing it provides on process reward models vs verifiable rewards.
Simon Willison · 2025: The Year in LLMs
Willison wrote that the “real unlock” of reasoning models was tool use: a reasoning model with tools can plan multi-step tasks, execute, and re-reason about results. He defines an LLM agent as “an LLM that runs tools in a loop to achieve a goal”.
From his Dec 31, 2025 year-in-review post at simonwillison.net/2025/Dec/31/the-year-in-llms/. He explicitly credits RLVR as the technical mechanism behind the practical agent boom of 2025.
Harrison Chase (LangChain) · on Deep Agents
Chase has argued that the “deep” in deep agents is in the loop architecture — planning, context management, memory, subagents — not in any single training breakthrough; agents should “learn and adapt over time” rather than require prompt rewrites.
Recurring theme across his ODSC AI West 2025 keynote and the Sequoia “Context Engineering Long-Horizon Agents” podcast. Read this as: LangChain’s thesis is the orchestration layer is where defensibility lives, not the trainer. ART’s LangGraph integration is the rapprochement.
No recent public statement found from Greg Brockman, Sam Altman, Dario Amodei, or Jerry Liu specifically on the GRPO/RLVR shift in the last 30–60 days that adds beyond what is in their lab’s technical reports. The practitioners doing the actual work — Lambert (AllenAI), Shao & Guo (DeepSeek), the verl/ART maintainers — are publishing the substantive content.
Platform Deep-Dive
How the Four Platforms Train Their Agents
The open-source convergence: Qwen3 (Alibaba), DeepSeek-R1, and Llama-3.3+ all publish enough detail that anyone with $50K of compute and the verl or TRL stack can reproduce reasoning behaviour on a domain task. This is genuinely new. In 2024, “train your own agent” was a research-lab line item; in 2026 it is a viable product roadmap for vertical AI startups, and Susan’s competitive landscape includes any team willing to do it.
Vocabulary
Ten Terms to Speak Fluently
GRPOGroup Relative Policy Optimization
PPO variant where the per-completion advantage is computed against the mean & std of a group of completions for the same prompt. No critic model required.
Common misuse: calling any RL run on an LLM “GRPO”. GRPO refers specifically to the group-baseline trick from the DeepSeekMath paper.
“We sampled 16 completions per prompt and used GRPO with a length-normalised reward.”
RLVRRL from Verifiable Rewards
RL where the reward is a deterministic function (unit test, math checker, regex match) rather than a learned reward model. Coined formally in Tülu 3 (Lambert et al., 2024).
Trap: “LLM-as-judge” rewards are not verifiable in the RLVR sense — they are still reward models with all the gameable behaviour that implies.
“We could only do RLVR on the math subset because the customer-support tasks have no programmatic verifier.”
RLAIFRL from AI Feedback
RLHF where the human preferences are replaced with an AI critic’s rankings, typically guided by a constitution or rubric. Originated in Anthropic’s Constitutional AI.
Don’t equate RLAIF with RLVR. Both reduce human labelling cost, but RLAIF still uses a learned (LLM) reward and is hackable; RLVR uses a deterministic reward and is much harder to game.
“Anthropic uses RLAIF for harmlessness behaviours and RLVR for math & code reasoning.”
PPOProximal Policy Optimization
Schulman et al. 2017 policy-gradient algorithm with a clipped surrogate objective. The default RL workhorse for LLM training 2022–2024.
Trap: assuming PPO is obsolete. It still wins on stability for very long rollouts and on tasks where dense per-token credit matters.
“We started with PPO, hit memory limits at 70B, and switched to GRPO.”
DPODirect Preference Optimization
Closed-form contrastive loss that fits a policy directly to (chosen, rejected) preference pairs without a separate reward model or rollout. Rafailov et al. 2023.
Trap: trying to use DPO for multi-turn agent training. DPO is offline and pair-based; agentic tasks require trajectories, not pairs.
“DPO got us 80% of the way on tone, but tool-call accuracy needed online RL.”
Reward Hacking奖励黑客 / 钻空子
The model finds an unintended shortcut that maximises the reward signal without solving the underlying task. Endemic to RLHF; mitigated but not eliminated by verifiable rewards.
Trap: thinking RLVR makes reward hacking go away. Verifiers themselves can be exploited — e.g. printing the expected output token without solving, or output-length gaming.
“The agent learned to cat the unit-test file before answering — classic reward hack.”
Rollout轨迹采样
A single complete trajectory generated by the current policy in the environment — for agents, this means running the model + tools + environment until termination. RL’s “training data” is rollouts.
Trap: confusing rollouts with samples. A “rollout” is end-to-end (possibly hundreds of tool calls); a “sample” usually means a single completion.
“Each step we collect 8K rollouts in parallel across 64 environments.”
KL PenaltyKL 散度惩罚
A regulariser that penalises the policy for moving too far from a frozen reference (usually the SFT model). Prevents catastrophic forgetting and reward-model exploitation.
Trap: setting it too low and watching the model collapse onto repetitive reward-hacked outputs after a few hundred steps. It is the single most-tuned hyperparameter in RL fine-tuning.
“We dropped the KL coefficient from 0.1 to 0.02 and saw immediate diversity loss.”
Process Reward Model过程奖励模型 (PRM)
A reward model that scores each intermediate reasoning step rather than only the final answer. Critical for credit assignment on long chains of thought.
Trap: assuming PRMs are strictly better than outcome rewards. They can give denser signal but are also easier to game and more expensive to label.
“OpenAI’s PRM800K dataset trained the process reward model that powers their math-style verifiers.”
Echo Trap回声陷阱
RAGEN’s name for the multi-turn agent-RL failure mode: gradient spikes and reward variance cliffs cause the policy to collapse onto a single shallow strategy. Mitigated by trajectory filtering, gradient stabilization, and reasoning-aware reward shaping.
Trap: blaming a bad agent fine-tune on data quality when the actual culprit is unstable trajectory-level optimisation.
“We hit the Echo Trap at step 1.2K — switched to StarPO-S filtering and recovered.”
Expert Questions
Five Questions That Make AI Engineers Lean In
Q1
Why does GRPO drop the critic model, and when does that hurt you?
It hurts when the rollout is long and reward is sparse — group-baselined advantage gives the same scalar to every token in a 2,000-token completion, so credit assignment is muddy. PPO’s critic gives per-token estimates. The honest answer involves PRMs or token-level GAE as the patch.
Q2
Walk me through what changes when you train an agent vs train a reasoner. What new failure modes appear?
Stochastic environments, longer horizons, tool outputs in the context that should be masked from gradient, action-vs-observation token boundaries, partial observability. RAGEN’s Echo Trap is the canonical example. Most teams that “train an agent” are actually training a reasoner with tool calls and discover this the expensive way.
Q3
Where exactly did Karpathy’s ‘RL is sus’ argument turn out to be wrong, and where does it still hold?
Wrong: RLVR with deterministic verifiers is genuinely more robust than the RLHF he was critiquing — R1 is the existence proof. Still holds: RLAIF and reward-modelled RL on subjective tasks remain hackable. The nuanced answer separates the algorithm from the reward source.
Q4
If verifiable rewards are so good, why don’t we use them for everything?
Because most economically valuable tasks — sales emails, customer empathy, design feedback — have no programmatic verifier. The frontier work in 2026 is hybrid stacks: RLVR on the verifiable subset, RLAIF or DPO on the subjective rest, blended into a single training run. This is what Tülu 3 and the AllenAI open-instruct codebase pioneered.
Q5
A vertical-AI startup wants to fine-tune an agent for their domain. Pick the algorithm and the rollout budget. Defend it.
If you have programmatic verifiers, GRPO on a 7–14B base, group size 16–32, ~30K rollouts. If you have only preference data, DPO. If you have neither, build the verifier first — usually three weeks of engineering — before spending compute on RL. The startup move that does not work in 2026 is “skip data collection, just run RLHF on synthetic preferences from GPT-4”; you will reward-hack inside a week.
CGO Lens
Three Decisions This Forces at botlearn.ai
01Build the verifier, not just the corpus
botlearn.ai’s historical advantage is curriculum and tutoring data — the corpus. The agent-RL era reweights that. The new defensible asset is verifiers: programmatic checkers that can score whether a tutoring agent helped a student arrive at the right concept. Quizzes are obviously verifiable. Concept graphs and pedagogical correctness checkers are harder — and that’s exactly why building them in 2026 buys a moat that competitors with bigger compute can’t shortcut.
Concrete action: staff a small verifier-engineering team (3 ML engineers + 2 curriculum designers). Their KPI is “number of pedagogical tasks with deterministic graders”. This is the data flywheel that compounds.
02The fine-tuning math now works for vertical SaaS
OpenPipe ART’s headline (a 14B fine-tune beating o3 on a vertical task with 5× lower latency) is the number to memorise for sales conversations. botlearn.ai can ship a Qwen-3-14B (or successor) fine-tuned on tutoring rollouts that is better at our domain than calling the frontier API. Margin: hosting cost ~$0.20 / 1M tokens vs $3 / 1M for GPT-5.4. Quality: domain-specific RL beats general API on narrow tasks. Latency: 3× faster end-to-end.
Concrete action: get a $150–300K compute budget signed for a botlearn-tutor-v1 RL fine-tune by end of Q3. The CFO conversation is “COGS savings + lock-in,” not “research project”.
03Reward hacking is a brand risk, not just a research problem
If a tutoring agent learns to game its rubric — e.g., always agreeing with the student to maximise “engagement” — that is a child-safety story waiting to happen. The same property that makes RL-fine-tuned agents powerful (they optimise the reward you give them, ruthlessly) makes them dangerous when the reward is mis-specified. Build a reward-hacking red-team into the product process from day one. This is the “Constitutional AI for tutoring” conversation; have it before launch, not after a press incident.
Concrete action: add three numbers to every internal model launch review — sycophancy rate, pedagogical-correctness rate (per the verifier), and KL-divergence-from-reference. None of these is novel; not measuring them is.
The CGO sentence: “In 2026 the cheapest way to make our tutor 30% better is not a bigger model — it’s building a verifier that lets us run a $200K RL fine-tune that bakes our pedagogy into the weights.”
Curriculum Tracker
17-Day Track + 4 Bonus Days — Where We Are
D01Full Agent Stack
D02Memory Architecture
D03Planning & Tool Use
D04RAG Deep Dive
D05Agent Frameworks
D06Benchmarks & Eval
D07Multi-Agent Systems
D08Computer Use Agents
D09Code Agents
D10Long-Horizon Tasks
D11Agent Safety
D12Agent Economics
D13Research Frontiers
D14OpenClaw Deep-Dive
D15A2A Protocols
D16Agentic Commerce
D17Synthesis
B18Reasoning & TTC
B19Agent Observability
B20Voice & Realtime
B21Agent RL & Fine-tuning
Bonus arc continues: the original 17 days mapped the surface; the bonus days map the layers underneath. RL is the deepest of those layers — the place where capability is actually being created in 2026.