Level 2 Checkpoint Quiz

20 questions covering all of Level 2. Aim for 16/20 (80%) or more, and finish the capstone, before starting Level 3. Reload the page to retry. Answer order is shuffled each time.

Prompts, structure, APIs (pages 01–03)

1. In a ticket-analysis chain, which step should NOT use an LLM?

Counting how many tickets fall into each category Classifying each ticket's free text Summarising the top issue for managers Suggesting causes for the top issue

Exact work (count, sort, join) belongs in code: cheaper, exact and testable. See page 01.

2. Self-consistency means…

Using the same system prompt everywhere Sampling the same prompt several times and taking the majority answer Setting temperature to 0 Asking the model if it is consistent

Low agreement between samples is also a useful signal to route a case to a human.

3. Schema-enforced structured output guarantees…

That the values are true That no retry will ever be needed That the output matches the schema's shape That business rules are satisfied

Shape, not truth. Domain validation in code is still required. See page 02.

4. A field (e.g. order ID) may be missing from the input. Best schema design?

Make it a required string Remove it from the schema Make it an integer Make it nullable and say "null if not stated, never guess"

Required-but-absent fields push the model to invent values.

5. Which HTTP status should your LLM client NOT retry?

401 Unauthorized 429 Too Many Requests 503 Service Unavailable A connection timeout

401 needs a credential fix. Retry 408/409/429/5xx and network errors with jittered back-off. See page 03.

6. finish_reason == "length" means…

The answer is complete The output was cut off at max tokens The model refused A tool was called

Treat it as an error for structured output. Raise the limit or shorten the task.

7. To benefit from prompt caching, you should put…

The user's question first Random IDs at the start of every prompt Stable content (system prompt, policies, tool definitions) first and variable content last Everything in one message with no system prompt

Caches typically match identical prefixes.

Tools, embeddings, RAG (pages 04–06)

8. In tool calling, the arguments field of a tool call is…

Always valid and safe to execute directly Executed by the model provider Ignored by the host Untrusted input (a JSON string) that your code must validate and authorise

Treat it like HTTP parameters. See page 04.

9. Which tool design is best for an inventory assistant?

check_inventory(sku, storeId) with enums run_sql(query) do_anything(instruction) One tool that both reads stock and reserves it

Specific, constrained tools. Keep reads and writes separate.

10. You switch to a new embedding model. What must you do?

Nothing Re-embed all documents, because vectors from different models aren't comparable Only re-embed new documents Lower the similarity threshold

Store the model version with each vector. See page 05.

11. Users search for exact codes like "CASE-RUGGED" and semantic search misses them. Best fix?

A bigger chat model Higher temperature Hybrid search: keyword + vector, fused with Reciprocal Rank Fusion Remove codes from the catalogue

Keyword search excels at exact codes. RRF combines rankings without comparing raw scores.

12. Where should per-user document permissions be enforced in RAG?

In the prompt ("don't reveal confidential docs") In the model's fine-tuning After the answer is generated At retrieval time, by filtering what can be retrieved

If it's never retrieved, it can't leak. See page 06.

13. A good RAG prompt should include…

Delimited numbered sources, a citation requirement and an explicit "I can't find this" rule As many documents as fit in the context window "Use your general knowledge if unsure" No instructions, just the question

Then verify citations in code.

Agents, skills, MCP (pages 07–09)

14. In a test-first agent loop, what stops the agent "cheating"?

A higher temperature Committing tests first, forbidding test edits (enforced by a hook/CI), and checking test strength Letting it write tests and code together without review Using a bigger model

See page 07.

15. Migrating Scala 2.13 → 3: what should you use before asking an agent to edit by hand?

Nothing, agents are always best A rewrite of the whole codebase in one PR Compiler rewrites (-source:3.0-migration -rewrite) and migration tooling Deleting the tests

Deterministic tools first. Agents handle the judgement calls left over.

16. A skill rarely triggers. What do you change first?

The folder name The scripts The licence field The description: what it does, when to use it, scope and negative scope

Only name + description are visible up front. See page 08.

17. Why give each custom agent (planner, reviewer…) its own limited toolset?

Least privilege and focus: a reviewer that can't edit can't "fix" by deleting tests It makes the model larger Tools are billed per agent It disables MCP

Narrow roles plus artefact hand-offs keep contexts clean.

18. What does Metals MCP give an agent working on Scala code?

A faster LLM Compiler-backed tools: compile, run tests, inspect symbols, docs, find usages Access to production databases Automatic PR merging

Real APIs instead of guesses. See page 09.

Testing, local models, data (pages 10–12)

19. A mutant "survived" in mutation testing. This means…

The code has no bugs The mutation tool crashed Your tests didn't catch a deliberate bug, so there's a gap Coverage is 100%

Feed survivors to AI to write tests from the spec. See page 10.

20. How should you evaluate AI-generated SQL against golden questions?

Compare SQL text exactly Ask the model if it is correct Check that it has no syntax errors Run both queries on a fixed dataset and compare the results

Different SQL can be equally correct. Execution accuracy is what matters. See page 12.

What next?

Level 2 complete? Scored 16+ and finished the capstone? You're ready for Level 3 · Advanced: building MCP servers and agents from scratch, evals, fine-tuning, LLMOps and AI security.

L2 · 15 · Level 2 Checkpoint Quiz

Prompts, structure, APIs (pages 01–03)

Tools, embeddings, RAG (pages 04–06)

Agents, skills, MCP (pages 07–09)

Testing, local models, data (pages 10–12)

What next?