L3 · 06 · Fine-tuning, LoRA & Distillation
Fine-tuning changes the model itself by continuing its training on your data. It's powerful and usually the last thing you should reach for — after prompting, structured output and RAG. This page covers when it's justified, the data it needs, parameter-efficient methods (LoRA/QLoRA), distillation, and how to tell if it worked.
By the end of this page you can
- Decide prompting vs. RAG vs. fine-tuning with a clear framework.
- Build a fine-tuning dataset (SFT format) and avoid the classic data mistakes.
- Explain LoRA and QLoRA and pick sensible starting hyperparameters.
- Describe distillation: a big model teaching a small one.
- Evaluate a fine-tune honestly: task eval, regression eval, cost/latency.
The decision order (fine-tune last)
| Your problem | Right fix | NOT fine-tuning |
|---|---|---|
| Model doesn't know your data (policies, SKUs, internal docs) | RAG — inject it at query time | Facts change; a frozen model goes stale |
| Output format is wrong | Structured output / JSON schema | Schema enforcement is cheaper and exact |
| Task is just hard | Better prompt + stronger model | Fine-tuning can't add capability data can't teach |
| Consistent style/format/behaviour across thousands of calls | Fine-tuning — finally justified | — |
| Small model must match big model on a narrow task (cost/latency) | Distillation | — |
| Domain jargon the base model butchers | Fine-tuning (or RAG first) | — |
Rule of thumb
If you can write it as instructions or retrieve it as context, don't fine-tune. Fine-tune for behaviour and form
(style, format, task pattern, domain tone), not for knowledge — knowledge goes stale and belongs in retrieval.
The data is the hard part
Fine-tuning learns whatever you show it — including your mistakes. Supervised fine-tuning (SFT) uses input → desired output pairs:
{"messages": [{"role": "system", "content": "You are a store assistant. Reply with a JSON action."}, {"role": "user", "content": "any earbuds at riverside?"}, {"role": "assistant", "content": "{\"action\": \"check_inventory\", \"sku\": \"EARBUDS-AIR2\", \"store\": \"riverside\"}"}]}
{"messages": [{"role": "system", "content": "You are a store assistant. Reply with a JSON action."}, {"role": "user", "content": "I want to return opened headphones bought 10 days ago"}, {"role": "assistant", "content": "{\"action\": \"check_return_policy\", \"category\": \"opened_electronics\"}"}]}
- Quality over quantity: 500–2000 great examples beat 50k sloppy ones.
- Every example must be perfect — the model copies errors faithfully. Review by hand or with a strong model + human spot-check.
- Cover edge cases: refusals, missing data, "I don't know" — or the model learns to always guess.
- Hold out a test set the trainer never sees, or you can't tell memorising from learning.
- Deduplicate and decontaminate — no near-duplicates, no eval-set leakage.
LoRA and QLoRA: fine-tune without a server farm
Full fine-tuning updates all billions of parameters — expensive and risks "catastrophic forgetting" of general ability. LoRA (Low-Rank Adaptation) freezes the base model and trains only small adapter matrices injected into each layer:
- LoRA: train ~0.1–1% of parameters, keep adapters as small files, swap them per task. Fits on one consumer GPU or Apple silicon (MLX LoRA).
- QLoRA: LoRA on a quantised (4-bit) base — halves the memory again, tiny quality cost. The usual default now.
| Starting hyperparameters | Sensible start |
|---|---|
| LoRA rank (r) | 8–16 (higher = more capacity, more overfitting risk) |
| Learning rate | 1e-4 – 2e-4 for LoRA |
| Epochs | 1–3; more memorises, not learns |
| Batch size / seq len | As large as memory allows / covers your examples |
# On Apple silicon, MLX runs LoRA locally: mlx_lm.lora --model mlx-community/<model>-4bit --train --data ./sft_data # Produces adapter files (not a full model copy) — merge or load alongside the base.
Distillation: big model teaches small model
Use a frontier model to generate high-quality answers, then fine-tune a small model to imitate them. You get near-big-model quality on a narrow task at small-model cost and latency. The catch: check the teacher model's licence — many providers restrict using outputs to train competing models.
Evaluate honestly
- Task eval: your golden set (L3·04) on the held-out test slice — did it improve?
- Regression eval: general capability checks — did it forget things it used to do?
- Compare against the strong baseline: does it beat a good prompt on the big model? If not, why bother?
- Cost/latency: the point of a small model is speed and price — measure both.
- Overfitting signs: training loss drops, eval accuracy flat or falling — stop earlier, fewer epochs, more data.
Try it (90 min)
- Write 50 SFT examples teaching a strict JSON-action format for the Store Assistant.
- Hold out 10 as a test set. Check they cover intents, refusals and missing data.
- Run LoRA on a small local model (MLX on Mac, or a hosted fine-tune service). Evaluate on the test set.
- Compare: fine-tuned small model vs. few-shot prompted base model. Which wins on format compliance? On cost?
Key takeaways
- Fine-tune last: prompt → structure/RAG → agents → then fine-tune, for behaviour not knowledge.
- Data quality decides everything; hold out a test set; include refusals and edge cases.
- LoRA/QLoRA train ~1% of params on one GPU or a Mac; QLoRA is the usual default.
- Distillation shrinks a big model to a cheap small one — mind the teacher's licence.