Level 3 Checkpoint Quiz

20 questions covering all of Level 3. Aim for 16/20 (80%) or more, and finish the capstone, before starting Level 4. Reload to retry; answer order is shuffled each time.

MCP servers & agents (pages 01–02)

1. In MCP, the host learns what a server offers by…

Reading the server's source code Guessing from the server name Calling initialize, then tools/list / resources/list / prompts/list The server emails documentation

Capability discovery is part of the protocol. See page 01.

2. With FastMCP, a tool's input schema comes from…

A separate YAML file The function's type hints, with the docstring as description The model invents it at call time There is no schema

Write type hints and docstrings as contracts — they become the schema the model sees.

3. On a stdio MCP server, where should logs go?

stdout Into the JSON-RPC replies Nowhere stderr — stdout is the protocol channel

One stray print on stdout corrupts the JSON-RPC stream.

4. An agent's memory strategy should…

Keep facts verbatim, compact big tool outputs, drop dead ends Keep everything forever Summarise the task itself Store nothing

See page 02.

5. A tool throws an exception. The agent should…

Crash immediately Retry the identical call forever Return the error to the model as data, and cap consecutive failures Delete the tool

Models recover well from errors fed back as results. Three repeats = stuck.

6. Which is NOT a required stopping condition?

Max steps A fixed temperature Max tool calls Token/cost or wall-clock budget

Temperature steers the model; limits guarantee the loop ends.

Multi-agent & evals (pages 03–04)

7. Multi-agent systems are justified mainly when…

Subtasks are independent, need different permissions, or need a separate critic You want to spend more tokens One agent is too simple to impress anyone The task is a single FAQ lookup

See page 03.

8. Agents should communicate via…

Shared full message histories Global variables The model's internal thoughts Small artefacts (JSON, summaries) merged in testable code

Artefacts keep contexts clean and results verifiable.

9. Three pipeline stages at 90% accuracy each give roughly…

90% 73% end-to-end 99% 30%

0.9³ ≈ 0.73. Errors compound multiplicatively — check between stages.

10. The best use of an LLM-as-judge is…

Replacing all code graders Judging with a vague "is this good?" Rubric-scored grading of what code can't check, validated against human labels Grading its own model's output

See page 04.

11. Agent evals should grade…

The trajectory: right tools, right order, within budget, plus the final answer Only the final answer Only token count Nothing, agents can't be tested

A right answer reached via 15 wasted calls or a security violation is still a failure.

12. A golden dataset should NOT…

Be versioned with the prompts Include prior production failures Grow over time Be used to tune prompts AND measure them (same set)

Tuning on the test set = overfitting the test. Keep a held-out slice.

RAG & fine-tuning (pages 05–06)

13. Users search exact codes like "ORD-482913" and vector search misses them. Fix?

A bigger embedding model More chunks Hybrid BM25 + vector, fused with Reciprocal Rank Fusion Raise temperature

See page 05.

14. A re-ranker improves precision by…

Fetching more documents Scoring each retrieved candidate against the query properly, keeping the best few Rewriting the documents Caching answers

Cheap recall (fetch ~50), expensive precision (re-rank to ~5).

15. When does a knowledge graph (GraphRAG) beat chunks?

Multi-hop questions over entity relationships Simple single-hop FAQs When you want a cheaper index Always

Graphs earn their build cost on relational questions, not FAQs.

16. Fine-tune for…

Adding facts that change often Fixing output format when a schema exists Anything, it's always best Consistent behaviour/format, or shrinking a big model to a small one (distillation)

Knowledge goes in RAG; behaviour goes in fine-tuning. See page 06.

17. LoRA works by…

Retraining all parameters faster Freezing the base model and training small adapter matrices Quantising the dataset Using more GPUs

~1% of params trained; QLoRA does it on a quantised base to save memory.

LLMOps & security (pages 07–08)

18. The three "silent killers" of LLM features in production are…

Syntax errors, typos, bad variable names Too many comments Quality regression, cost creep, latency spikes Merge conflicts

See page 07.

19. The most reliable prompt-injection defense is…

Architecture: least-privilege tools, validated args, human approval for consequential actions A longer system prompt saying "don't get injected" Hiding the system prompt better Asking the model nicely

Prompt-level defenses are bypassable; architecture bounds the damage. See page 08.

20. A "rug pull" on an MCP server means…

The server runs out of memory A DNS failure The model refuses to call it A trusted server silently changes its tools or behaviour in an update

Pin versions, review tool schemas on update, keep least privilege.

What next?

Level 3 complete? Scored 16+ and finished the capstone? You're ready for Level 4 · Expert: the maths and transformer internals, training and post-training, inference optimisation, and architecture decisions.

L3 · 11 · Level 3 Checkpoint Quiz

MCP servers & agents (pages 01–02)

Multi-agent & evals (pages 03–04)

RAG & fine-tuning (pages 05–06)

LLMOps & security (pages 07–08)

What next?