L2 · 14 · Capstone: the Store Assistant

Put Level 2 together into one working application: an assistant for retail store staff that answers order and stock questions with tools, policy questions with RAG, and "what should I recommend?" questions with semantic search, with grounding, guardrails, tests and an evaluation set. A complete runnable Scala 3 starter is included.

Architecture

Milestones

1. Run the starter below offline. Read every part and map it to Level 2 pages.
2. Tests: MUnit tests for the router, each fact gatherer, and the guard that rejects unknown order IDs.
3. Golden set: 25 questions (orders, stock, policy, recommendations, unanswerable) with expected intent and key facts. Write a runner that prints accuracy.
4. Live mode: run with --live against a local model (page 11) or an approved hosted model (page 03). Compare the answers with the template answers.
5. Upgrade retrieval: replace the toy embedder with real embeddings (page 05). Re-run the golden set.
6. LLM router (stretch): replace the keyword router with structured-output classification (page 02), keeping the keyword router as a fallback. Compare accuracy on the golden set.
7. Tool loop (stretch): for multi-step questions ("is it in stock anywhere near Central?"), use the agent loop from page 04.
8. Review: run a reviewer agent and a security pass (prompt injection via policy docs, data exposure). Fix the findings.
9. Write it up: README with architecture, how to run, golden-set results, and known limitations.

The starter (Scala 3, runs offline)

//> using scala 3
//> using dep io.circe::circe-core:0.14.15
//> using dep io.circe::circe-parser:0.14.15
import io.circe.Json
import io.circe.parser.parse
import io.circe.syntax.*
import java.net.URI
import java.net.http.{HttpClient, HttpRequest, HttpResponse}

// ===================== Data (replace with real services) =====================
final case class Product(sku: String, name: String, description: String)
final case class Order(id: String, status: String, storeId: String, detail: String)
final case class PolicyChunk(section: String, text: String)

val Stores = List("RIVERSIDE", "CENTRAL", "AIRPORT")
val Products = Vector(
  Product("HEADPHONES", "Over-ear headphones", "noise cancelling headphones for flights and long travel"),
  Product("EARBUDS",    "Sport earbuds",       "wireless earbuds for running jogging and gym, sweat resistant"),
  Product("CHARGER",    "Car charger",         "usb-c car adapter for charging phones while driving in the car"),
  Product("CASE",       "Rugged case",         "phone case with drop protection for when you drop your phone")
)
val Stock: Map[(String, String), Int] =
  Map(("HEADPHONES", "RIVERSIDE") -> 3, ("EARBUDS", "CENTRAL") -> 6, ("CHARGER", "CENTRAL") -> 12).withDefaultValue(0)
val Orders = Map(
  "ORD-482913" -> Order("ORD-482913", "READY_FOR_PICKUP", "RIVERSIDE", "shelf B4"),
  "ORD-500122" -> Order("ORD-500122", "IN_TRANSIT", "CENTRAL", "estimated ready 2026-10-10")
)
val Policies = Vector(
  PolicyChunk("Returns > Standard",      "Most items can be returned within 14 days with a receipt or order ID."),
  PolicyChunk("Returns > Opened audio",  "Opened headphones and earbuds can be returned within 14 days only if defective."),
  PolicyChunk("Pickup > Holding period", "Pickup orders are held for 7 days, then cancelled and refunded."),
  PolicyChunk("Pickup > Identification", "Customers must show the order confirmation and a photo ID matching the order name.")
)

// ===================== Retrieval (toy embedder; use real embeddings from page 05) =====================
type Vec = Array[Double]
def embed(text: String): Vec =
  val v = new Array[Double](512)
  text.toLowerCase.split("\\W+").filter(_.length > 2).foreach(w => v(math.floorMod(w.hashCode, 512)) += 1)
  val n = math.sqrt(v.map(x => x * x).sum)
  if n == 0 then v else v.map(_ / n)

def topK[A](query: String, items: Vector[A], text: A => String, k: Int): Vector[(A, Double)] =
  val q = embed(query)
  items.map(a => a -> embed(text(a)).zip(q).map(_ * _).sum).filter(_._2 > 0).sortBy(-_._2).take(k)

// ===================== Router =====================
enum Intent:
  case OrderStatus(orderId: String)
  case StockCheck(sku: Option[String], store: Option[String])
  case PolicyQuestion
  case Recommend
  case Unknown

private val OrderIdRe = """ORD-\d{6}""".r
private def has(q: String, words: String*): Boolean = words.exists(w => s" $q ".contains(s" $w "))

def route(question: String): Intent =
  val q = question.toLowerCase.replaceAll("[^a-z0-9 -]", " ")
  val orderId = OrderIdRe.findFirstIn(question.toUpperCase)
  if orderId.isDefined && !has(q, "return", "refund") then Intent.OrderStatus(orderId.get)
  else if has(q, "stock", "available", "have") then
    Intent.StockCheck(Products.map(_.sku).find(s => q.contains(s.toLowerCase)), Stores.find(s => q.contains(s.toLowerCase)))
  else if has(q, "return", "refund", "policy", "hold", "held", "id", "identification") then Intent.PolicyQuestion
  else if has(q, "recommend", "suggest", "need", "looking", "want") then Intent.Recommend
  else Intent.Unknown

// ===================== Grounding =====================
final case class Grounded(intent: Intent, facts: List[String], sources: List[String])

def gather(question: String): Grounded = route(question) match
  case i @ Intent.OrderStatus(id) => Orders.get(id) match
    case Some(o) => Grounded(i, List(s"Order ${o.id} is ${o.status} at ${o.storeId} (${o.detail})."), List("order-system"))
    case None    => Grounded(i, List(s"No order $id was found. Check the ID with the customer."), List("order-system"))
  case i @ Intent.StockCheck(Some(sku), store) =>
    val wanted = store.toList.map(s => s"$sku at $s: ${Stock((sku, s))} in stock.")
    val others = Stores.filter(s => !store.contains(s) && Stock((sku, s)) > 0).map(s => s"$s has ${Stock((sku, s))}.")
    val facts  = wanted ++ (if others.isEmpty then List(s"No other store has $sku.") else List(s"Other stores: ${others.mkString(" ")}"))
    Grounded(i, facts, List("inventory"))
  case i @ Intent.StockCheck(None, _) =>
    Grounded(i, List(s"Which product? Known SKUs: ${Products.map(_.sku).mkString(", ")}."), Nil)
  case i @ Intent.PolicyQuestion =>
    val hits = topK(question, Policies, c => s"${c.section} ${c.text}", 2)
    Grounded(i, hits.toList.zipWithIndex.map { case ((c, _), n) => s"[${n + 1}] ${c.text}" }, hits.toList.map(_._1.section))
  case i @ Intent.Recommend =>
    val hits = topK(question, Products, p => s"${p.name} ${p.description}", 2)
    Grounded(i, hits.toList.map((p, _) => s"${p.name} (${p.sku}): ${p.description}."), hits.toList.map(_._1.sku))
  case Intent.Unknown => Grounded(Intent.Unknown, Nil, Nil)

// ===================== Answer generation =====================
type Generator = (String, Grounded) => String

val templateGenerator: Generator = (_, g) =>
  if g.facts.isEmpty then "I can help with order status, stock, store policies and product suggestions."
  else g.facts.mkString(" ")

def llmGenerator(base: String, model: String, key: Option[String]): Generator = (question, g) =>
  if g.facts.isEmpty then templateGenerator(question, g)
  else
    val system = "You help retail store staff. Answer ONLY from the facts given. Keep [n] citations. " +
      "Never invent numbers, statuses or policies. Max 3 sentences."
    val user = s"Facts:\n${g.facts.mkString("\n")}\n\nQuestion: $question"
    val body = Json.obj("model" -> model.asJson, "temperature" -> 0.asJson,
      "messages" -> Json.arr(Json.obj("role" -> "system".asJson, "content" -> system.asJson),
                             Json.obj("role" -> "user".asJson, "content" -> user.asJson))).noSpaces
    val b = HttpRequest.newBuilder(URI.create(s"$base/chat/completions"))
      .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString(body))
    key.foreach(k => b.header("Authorization", s"Bearer $k"))
    scala.util.Try(HttpClient.newHttpClient().send(b.build(), HttpResponse.BodyHandlers.ofString())).toOption
      .filter(_.statusCode() == 200)
      .flatMap(r => parse(r.body()).toOption)
      .flatMap(_.hcursor.downField("choices").downN(0).downField("message").get[String]("content").toOption)
      .getOrElse(templateGenerator(question, g) + "  (AI unavailable: showing raw facts)")

// ===================== App =====================
@main def storeAssistantApp(args: String*): Unit =
  val generate =
    if args.contains("--live") then llmGenerator(sys.env.getOrElse("LLM_BASE_URL", "http://localhost:11434/v1"),
      sys.env.getOrElse("LLM_MODEL", "llama3.2"), sys.env.get("LLM_API_KEY"))
    else templateGenerator
  val questions = args.filterNot(_ == "--live").toList match
    case Nil => List(
      "Is order ORD-482913 ready?",
      "Do we have earbuds in stock at Riverside?",
      "Can a customer return opened earbuds they just don't like?",
      "Customer is looking for something to charge the phone while driving",
      "How long do we hold pickup orders?",
      "Where is order ORD-999999?",
      "What's the weather tomorrow?"
    )
    case qs => qs
  questions.foreach { q =>
    val g = gather(q)
    println(s"\nQ: $q\n   intent : ${g.intent}\n   sources: ${if g.sources.isEmpty then "-" else g.sources.mkString(", ")}\n   A: ${generate(q, g)}")
  }

Expected offline behaviour: order status from the order system, stock with alternatives (Riverside has no earbuds, but Central has 6), the "opened audio" policy for the returns question, the car charger as the top recommendation, a clear "not found" for an unknown order, and a polite scope message for the weather.

Golden evaluation set

Is order ORD-482913 ready?                         OrderStatus     READY_FOR_PICKUP
Do we have earbuds in stock at Riverside?         StockCheck      CENTRAL has 6
Can they return opened earbuds they don't like?   PolicyQuestion  only if defective
Something to charge the phone in the car          Recommend       CHARGER
Where is order ORD-999999?                        OrderStatus     No order
What's the weather tomorrow?                      Unknown         order status, stock

Write a small runner (Scala or Python) that loads the file, calls gather + the generator, and reports intent accuracy and must-contain pass rate. Keep it in CI so every change to prompts, models or retrieval is measured.

Quality and safety checklist

• Numbers, statuses and policy text come only from tools and documents. The model only phrases them.
• Unknown IDs and products produce clear "not found" answers, never guesses.
• Live mode falls back to raw facts if the model is unavailable.
• Policy documents are treated as data (delimited), and write actions don't exist yet. If you add them, use confirm-first.
• No customer personal data in logs or prompts beyond what the answer needs.
• Golden set results recorded in the README with model and prompt versions.