L4 · 03 · Training & Post-training: SFT, RLHF, DPO

A model that only does next-token prediction is a text completer, not an assistant. Making it helpful, follow instructions and refuse harmful requests is a separate phase: post-training. This page covers the pipeline from raw pre-training to the aligned model behind the API, and what each stage is actually optimising for.

Level 460 minTrack 1Last verified 2026-10

By the end of this page you can

Describe the full pipeline: pre-training → SFT → preference training (RLHF/DPO) → reasoning RL.
Explain what a "base model" is and why chat models differ from it.
Explain RLHF's reward model and why DPO is often simpler.
Explain RL with verifiable rewards (RLVR) — the trick behind reasoning models.
Reason about data: quality, dedup, contamination, and why data > model size past a point.

The pipeline at a glance

Stage 0: pre-training

Train on trillions of tokens of text — the only objective is next-token prediction. This is where almost all the compute goes (weeks, thousands of GPUs) and where the model absorbs knowledge, grammar, code and reasoning patterns. The result is a base model: great at completing text, terrible at answering questions — ask it a question and it might write another question, because that's what text does.

Data is the moat. Quality, dedup and filtering matter more than architecture details. Web text, books, code — cleaned and deduplicated.
Contamination: if benchmark questions leak into training data, eval scores lie. Labs dedup against test sets.
Tokenizers are trained too (BPE): a subword vocabulary that balances compression vs. vocab size.

Stage 1: SFT — learn to be an assistant

Supervised fine-tuning: continue training on (instruction, good response) pairs. This is L3·06's SFT, at frontier scale. The model learns the format of being an assistant: "when addressed, respond helpfully." Even a few thousand to ~100k high-quality examples shift behaviour dramatically — format is easier to learn than knowledge.

Stage 2: preference training — learn to be good

SFT teaches format; preference training teaches taste: which of two responses is better? Humans (or a strong model) rank pairs of answers, and the model is pushed toward the preferred ones.

	RLHF (PPO)	DPO
Idea	Train a reward model on human preferences, then optimise the LLM against it with RL	Skip the reward model; optimise directly on preference pairs
Moving parts	4 models (policy, reward, ref, value) — complex, unstable	2 — simpler, usually the default now
Cost / stability	Expensive, fiddly	Cheap, stable
Ceiling	Higher (reward model generalises)	Very good, occasionally below RLHF's peak

Why preferences, not just more SFT? For "good vs. bad" judgements it's easier to rank than to write the ideal answer. "This response is better than that one" is cheap to label and captures nuance (tone, hedging, safety) that's hard to specify as target text.

Stage 3: RLVR — learn to reason

The breakthrough behind "reasoning" models (o1/R1-style, next page). For tasks with a verifiable answer — maths, code that passes tests — you don't need a reward model: check the answer programmatically and reward the model for getting it right. Trained this way, models learn to produce long chains of thought, self-check, and backtrack — emergent behaviours, not hand-coded ones.

Also in the mix

Constitutional / self-critique: the model evaluates its own outputs against written principles, generating preference data without humans on every item.
Safety training: refusal behaviour is trained in preference data too — which is why "just remove the safety fine-tune" on an open model produces something very different.
Tool-use training: modern models are also trained to emit structured tool calls — that behaviour (L2·04) is learned, not magic.

The mental model to keep

Base model vs. chat model The base model is a text-completion engine that absorbed the internet. Post-training sculpts it into an assistant: SFT gives it the assistant format, preference training gives it taste, RLVR gives it reasoning. When a chat model "refuses", that's learned behaviour from stage 2–3 — and it explains jailbreaks: they're ways of slipping the input back into territory where the base-model behaviour shows through.

Why you should care (even as a user)

Pick base vs. instruct models correctly for local use and fine-tuning (L2·11, L3·06).
Understand refusals and jailbreaks as post-training artefacts — it makes L3·08 security concrete.
Read a model card and know what "SFT + DPO + RLVR" actually means.
Reason about evals: a model that scored 95% might have seen the test data — contamination is real.

Try it (45 min)

Download a base model and its instruct-tuned sibling (Ollama or LM Studio). Ask both "What's the capital of France?" — see the base model ramble vs. the instruct model answer.
Read a model card (any open-weight model) and find: pre-training size, post-training method, evals, licence.
Write 5 preference pairs yourself (two answers, pick the better). Feel how much easier ranking is than writing the ideal answer.

Key takeaways

Pre-training makes a text completer; post-training makes an assistant.
SFT = format, preferences (RLHF/DPO) = taste/safety, RLVR = reasoning via verifiable rewards.
DPO is simpler than RLHF and the usual default; RLVR is what powers reasoning models.
Data quality, dedup and contamination decide more than the training recipe.