L5 · 06 · Level 5 Labs

Five labs, one per Level 5 page. These are the least prescriptive labs in the curriculum — at Master level you're doing research and judgement work, not following recipes. Each produces a real artefact you can show. Expect 8–14 hours plus whatever your capstone needs.

Lab 1 · Deep-read and reproduce a paper

1. Pick a paper relevant to your work (RAG, attention, a reasoning method, an eval technique).
2. Pass 1 and 2: write the claim as a falsifiable sentence, note baseline + metric + what they didn't measure.
3. Pass 3: reproduce the mechanism at toy scale (page 01's harness). Run at least one ablation.
4. Write the one-page summary: claim, method, your reproduction, verdict.

Success: a written reproduction with at least one ablation and a "does the claim hold?" verdict.

Lab 2 · Probe a model's internals

1. Pick a local model (Ollama/MLX) or an interpretability-friendly small model.
2. Capture activations for two classes of input (e.g. questions vs. statements). Train or apply a simple probe.
3. Try a logit-lens check or a steering vector if your tooling supports it.
4. Find one sycophancy or reward-hacking instance and describe which post-training choice caused it.

Success: evidence a property is encoded in the activations, plus a written alignment observation.

Lab 3 · On-device vs. cloud bake-off

1. Pick 5 real tasks from your domain (summarise, extract, classify, answer).
2. Run on a small on-device model (Core ML / MLX / Ollama) and a hosted model.
3. Record quality, latency, and which tasks the small model handles vs. needs the cloud.
4. Design the hybrid router for a real feature; write the privacy case for it.

Success: a measured split of what stays on-device vs. what escalates, with a reason per task.

Lab 4 · Multimodal task eval

1. Pick a real multimodal task (screenshot→bug report, document→fields, diagram→description).
2. Build a small eval set (10–20 items) with known-correct answers.
3. Score a VLM; find its specific failure modes (fine print, counting, spatial precision).
4. Decide: is it reliable enough for your use case? With what checks?

Success: an eval score for a multimodal task and a written list of its failure modes.

Lab 5 · An adoption pilot plan

1. Choose a real team and one workflow (code review, debugging, docs, testing).
2. Define the baseline measurement and the 4-week pilot: group, training format, champion.
3. Write the 1-page usage policy for it.
4. Write how you'll measure success honestly and what would make you stop.

Success: a plan you'd actually run: baseline, pilot, policy, honest metrics, kill criteria.

Then the capstone