L3 · 09 · Level 3 Labs

Eight hands-on labs, one per Level 3 page. These are bigger than Level 2 labs — you're now building real systems, not just using tools. Each has a goal, steps and a success criterion. Together they produce the pieces the capstone assembles. Expect 10–16 hours in total.

Setup

• Everything from Level 2 labs: Scala (scala-cli, sbt), Python + uv, a local model runner.
• Python extras: uv add mcp for lab 1; an OpenTelemetry viewer (Langfuse/LangSmith) optional for lab 7.
• A git repo ai-labs-l3/. Commit after every lab — you'll diff experiments.
• A running Store Assistant from Level 2 to extend.

Lab 1 · Build and ship an MCP server

1. Extend store_server.py (page 01) with a third tool, a resource and a prompt.
2. Write unit tests for every tool (call them as functions).
3. Run it under the MCP Inspector; click through each capability.
4. Write a README: what it exposes, required env vars, the MCP config snippet. Give it to a friend/teammate — can they connect without asking you anything?

Success: a teammate connects using only your README, and every tool validates bad input cleanly.

Lab 2 · A durable agent loop

1. Extend agent.py (page 02) to serialise State to JSON after every step.
2. Add resume: kill the process mid-run, restart, continue from the saved state.
3. Add a token budget and a compactor (replace old tool outputs with summaries).
4. Write tests: state round-trips through JSON; limits fire; a looping model gets stopped.

Success: the agent survives a kill-and-resume and stops cleanly on every limit, reporting which one.

Lab 3 · Orchestrator–worker with real merging

1. Take a real research task (e.g. "summarise the return, warranty and pickup policies and find the contradictions").
2. Run it one-agent (log tokens), then as orchestrator–worker: one worker per doc, merge artefacts in code.
3. Add an author–critic loop on the merged summary with a written checklist (max 3 iterations).
4. Break a worker (raise an exception). Make merge handle partial results honestly.

Success: multi-agent beats one-agent on quality for this task, or you've measured that it doesn't and can say why.

Lab 4 · An eval suite that catches regressions

1. Build a 30-case golden set for the Store Assistant: intents, tool calls, refusals, 5 injection attempts.
2. Split it: 20 tune, 10 held-out. Never touch the held-out while developing.
3. Run evals in CI (or a script). Record pass-rate in a file with prompt/model version.
4. Deliberately make a prompt change that regresses. Prove the suite catches it.

Success: the held-out set catches a real regression you introduced on purpose, and results are versioned.

Lab 5 · Hybrid search that beats vector-only

1. Index 20–30 store docs. Build a golden query set: exact codes, semantic, multi-hop.
2. Measure hit@5 for vector-only, BM25-only, and RRF hybrid.
3. Add an LLM re-ranker on the top 20. Re-measure.
4. Write one sentence on where each method wins.

Success: you have a number showing hybrid beats vector-only on exact-code queries, and you know the re-ranker's cost/benefit.

Lab 6 · Fine-tune a strict-format model

1. Write 100 SFT examples teaching the JSON-action format (page 06). Include refusals and missing-data cases.
2. Hold out 15 as a test set. Check coverage: every intent, every edge case.
3. Run LoRA on a small model (MLX on a Mac, or a hosted service). Track training and eval loss.
4. Compare fine-tuned vs. few-shot prompted base model on the held-out set: format compliance, accuracy, cost/call.

Success: you can state, with numbers, whether fine-tuning beat prompting for this narrow task — and whether it was worth it.

Lab 7 · Instrument a live LLM feature

1. Add tracing (page 07's Tracer or a real backend) to the Store Assistant: model, tokens, latency, cost, tool calls, trace ID.
2. Add an exact-response cache and measure hit rate over 50 real-ish queries.
3. Add routing: trivial → small model, hard → frontier. Log which route each request took.
4. Add a cost budget and a kill-switch flag. Test both.

Success: you can answer "what did yesterday's traffic cost and which feature spent it" from your own logs, and the kill switch works.

Lab 8 · Red-team your own assistant

1. Attack the Store Assistant: direct injection in the user message, indirect injection inside a policy doc and a product review.
2. Try to make it: leak its system prompt, call a tool with unauthorised args, exfiltrate data, promise an impossible refund.
3. For every success, add a defense (page 08 layers) and re-test.
4. Write up the top 3 findings and the residual risk honestly.

Success: you have a written attack log, defenses for each success, and you can say clearly what you could not fully prevent.

Then the capstone