L2 · 13 · Level 2 Labs

Twelve hands-on labs, one or more per Level 2 page. Each has a goal, steps and a clear success criterion. Together they build the skills and the pieces you need for the capstone. Expect 10–15 hours in total.

Setup

• Scala via Coursier (cs setup): scala-cli, sbt, JDK 21.
• Python 3.12+ with uv: uv add httpx pydantic pytest hypothesis.
• A local model runner (page 11): Ollama with a small chat model and nomic-embed-text, or LM Studio.
• Optional: an approved hosted model via your company gateway (page 03).
• A git repo ai-labs-l2/. Commit after every lab.

Lab 1 · Ticket triage chain with golden tests

1. Write 30 realistic support tickets (pickup, returns, payment, product, other), with 5 deliberately ambiguous.
2. Build the chain from page 01: classify (label only) → count in code → summarise the top issue.
3. Store the classify prompt in prompts/classify.v1.md. Write golden.jsonl with expected labels.
4. Add self-consistency (5 votes). Route agreement below 0.6 to "needs human".

Success: golden accuracy ≥ 90% on the clear tickets, and the ambiguous ones are flagged for a human.

Lab 2 · Schema-enforced extraction with repair loop

1. Use ReturnExtraction.scala or return_extraction.py (page 02) with a real model and response_format.
2. Write 10 messy customer emails: missing order ID, two items, an unknown product, a non-English sentence.
3. Log how many needed a retry and which validation errors occurred.

Success: no invented order IDs (null when missing), unknown SKUs rejected, and every result either valid or escalated.

Lab 3 · Harden the LLM client

1. Run LlmClient.scala (page 03) against a local model.
2. Stop the server mid-run, then use a wrong model name, then set max_tokens to 5. Observe the retries, the non-retried 404, and the finish_reason=length warning.
3. Add a concurrency limit (at most 4 in-flight calls) and a total-deadline parameter.

Success: each failure type produces the correct behaviour and a clear log line, with no secrets logged.

Lab 4 · Add tools to the Store Assistant

1. Extend StoreAssistant.scala (page 04) with get_product_info(sku) and propose_transfer(sku, from, to, qty). The latter only returns a proposal.
2. Validate arguments in code: known SKUs and stores, quantity 1–10.
3. Run --live with a local model. Ask a multi-step question that needs two tools.

Success: correct tool sequence, invalid arguments return errors the model explains, and no transfer is "performed".

Lab 5 · Measure semantic search

1. Grow the catalogue in ProductSearch.scala to 30 products.
2. Write 20 queries with the expected SKU. Compute hit@1 and hit@3 for the toy embedder, a real embedder, and hybrid (RRF).

Success: a small results table in your notes showing which approach wins and why.

Lab 6 · Policy RAG with abstention

1. Write 3 short policy documents (returns, pickup, price matching) with headings.
2. Use PolicyRag.scala with real embeddings and a real model. Check citations in code.
3. Golden set: 15 answerable and 5 unanswerable questions. Measure hit@3, citation validity and abstention.

Success: all 5 unanswerable questions abstain, and no answer cites a non-existent source.

Lab 7 · Spec → plan → tests → agent

1. Write a spec (page 07 template) for a small feature in a sample Scala service, e.g. "orders held over 7 days are cancelled".
2. Get a plan, edit it, write or approve tests first, and commit them.
3. Let the agent implement with "don't modify tests". Do the break-it-three-ways check.

Success: all tests green, test files unchanged in the implementation commit, and all 3 deliberate bugs caught.

Lab 8 · A skill that passes its trigger set

1. Build scala-munit-suite (page 08) with check_suite.py.
2. Write 12 trigger prompts (8 should trigger, 4 near-misses). Run each in a fresh session.
3. Iterate on the description until at least 11/12 behave correctly.

Lab 9 · Metals MCP + one external server

1. Enable Metals MCP. Ask the agent to inspect a type and compile a module through the tools.
2. Add one external server (e.g. Playwright MCP, pinned version) via .vscode/mcp.json, with any secret supplied through inputs.
3. Run the trust checklist for that server and note the result.

Lab 10 · Kill the mutants

1. Run the ScalaCheck suite from page 10. Change rounding to DOWN. Does a property fail? If not, add one that does.
2. Run Stryker4s (Scala) or mutmut (Python) on a small module, or do a manual mutation test with your agent.
3. Ask AI to write tests that kill the survivors, deriving expectations from the spec.

Success: mutation score improves, and you can explain what each new test protects.

Lab 11 · Local model bake-off

1. Install two local models of different sizes (page 11).
2. Run bench.py, then run labs 1, 2 and 4 against each.
3. Record speed, pass rates and tool-calling reliability.

Lab 12 · Text-to-SQL with guard and evaluation

1. Extend sql_eval.py (page 12) with 10 golden questions on the sample schema.
2. Generate SQL with a model using the strong prompt (schema + semantics + assumptions).
3. Run every candidate through SqlGuard logic (port it to Python or call the Scala version), then evaluate.

Success: execution accuracy reported, and every failure explained (wrong filter, join, definition, or a guard rejection).

When you finish, go to the capstone. Most of its pieces come from these labs.