L3 · 10 · Capstone: A Production-Grade Agent

Everything in Level 3 converges here: an agent (page 02) whose tools come from your own MCP server (page 01), orchestrated where needed (page 03), measured by an eval suite (page 04), observed in production (page 07) and reviewed for security (page 08). This page gives the architecture, the build order and the acceptance criteria — the labs built most of the pieces already.

Architecture

Build order

1. MCP server first (lab 1): inventory, order status, policy lookup as tools + a policy resource. Unit-tested, README'd.
2. Agent loop (lab 2): state, limits, memory, error recovery — talking to the MCP server, not to functions.
3. Orchestration only where it pays (lab 3): maybe a researcher–critic pair for policy answers; skip it if one agent suffices.
4. Eval suite (lab 4): golden set, trajectory grading, safety cases in CI.
5. Observability (lab 7): tracing, cost, kill switch.
6. Security review (lab 8): red-team, fix, write it up.

Reference skeleton (Scala)

The agent talks to the MCP server over stdio — the same loop shape as page 02, but tool calls go through the MCP client. Here's the core, compilable shape (a full implementation is the project):

import scala.collection.mutable

// Simulated MCP client: in production this speaks JSON-RPC to a real server subprocess.
// The interface is what matters — the agent never touches store data directly.
trait McpClient:
  def callTool(name: String, args: Map[String, String]): String
  def listTools(): List[String]

class FakeMcp extends McpClient:
  private val inv = Map("EARBUDS-AIR2" -> 14)
  def listTools() = List("check_inventory", "order_status")
  def callTool(name: String, args: Map[String, String]) = name match
    case "check_inventory" =>
      args.get("sku").map(s => s"""{"sku":"$s","qty":${inv.getOrElse(s.toUpperCase, 0)}}""")
        .getOrElse("""{"error":"missing sku"}""")
    case "order_status" =>
      s"""{"order":"${args.getOrElse("order_id", "?")}","status":"READY_FOR_PICKUP"}"""
    case _ => """{"error":"unknown tool"}"""

case class Limits(maxSteps: Int = 10, maxToolCalls: Int = 6)

class Agent(mcp: McpClient, limits: Limits):
  private val trace = mutable.Buffer[String]()

  def run(task: String): String =
    var steps, toolCalls = 0
    var done = false
    val answer = new StringBuilder
    while !done && steps < limits.maxSteps do
      steps += 1
      // Placeholder model: in production this is the LLM deciding tool calls.
      // Here we drive the loop shape with a scripted "model".
      val decision = if steps == 1 then """TOOL:check_inventory {"sku":"EARBUDS-AIR2"}""" else "ANSWER:done"
      if decision.startsWith("TOOL:") then
        if toolCalls >= limits.maxToolCalls then
          answer.append("stopped: tool budget"); done = true
        else
          toolCalls += 1
          val Array(name, rest) = decision.stripPrefix("TOOL:").split(" ", 2)
          val result = mcp.callTool(name, parseArgs(rest))
          trace += s"step $steps: $name -> $result"
          if result.contains("error") then { answer.append(s"error: $result"); done = true }
      else
        answer.append("completed"); done = true
    if !done then answer.append("stopped: step limit")
    answer.toString

  private def parseArgs(s: String): Map[String, String] =
    """"(\w+)":"([^"]*)"""".r.findAllMatchIn(s).map(m => m.group(1) -> m.group(2)).toMap

  def getTrace: List[String] = trace.toList

object StoreOpsAgent:
  def main(args: Array[String]): Unit =
    val agent = Agent(FakeMcp(), Limits())
    val result = agent.run("how many earbuds?")
    println(s"answer: $result")
    agent.getTrace.foreach(println)
    assert(agent.getTrace.head.contains("EARBUDS-AIR2"))

The point isn't the fake model — it's that the agent never touches inventory data except through the MCP client, every tool call is traced and counted, and limits guarantee termination. Swap FakeMcp for a real stdio client and the structure is unchanged.

Acceptance criteria

• MCP server: 3+ tools, 1+ resource, unit tests, README with config snippet; a teammate can connect unaided.
• Agent: serialisable state, all four limits enforced and reported, tool errors fed back not crashed.
• Evals: 30+ golden cases incl. held-out slice; trajectory checks; safety cases always run; results versioned; a deliberate regression is caught.
• Observability: every call traced with tokens/latency/cost; you can report yesterday's spend per feature; kill switch verified.
• Security: red-team log of 10+ attempts; each success has a fix or a documented accepted risk; OWASP checklist completed.
• Write-up: one page on what worked, what didn't, and what you'd do differently.