L3 · 07 · LLMOps: Deploy, Observe, Optimise

Shipping an LLM feature is the start, not the end. Unlike normal code, it can regress without a deploy (model updates), cost can balloon quietly (token sprawl), and quality problems are invisible unless you measure them. LLMOps is the operational discipline: tracing, evals in production, caching, routing, cost control and guardrails.

What breaks in production

Trace every call

You can't fix what you can't see. Log structured records per LLM call — and per tool call inside an agent run:

import json
import time
from contextlib import contextmanager


class Tracer:
    """Minimal LLM call tracer. In production, send spans to an observability backend
    (e.g. an OpenTelemetry collector, LangSmith, Langfuse, Braintrust)."""

    def __init__(self):
        self.records = []

    @contextmanager
    def span(self, name: str, **attrs):
        start = time.perf_counter()
        rec = {"span": name, **attrs}
        try:
            yield rec
        finally:
            rec["latency_ms"] = round((time.perf_counter() - start) * 1000, 1)
            self.records.append(rec)

    def report(self):
        for r in self.records:
            print(json.dumps(r))


if __name__ == "__main__":
    tr = Tracer()
    with tr.span("chat.completions", model="demo", feature="store-assistant") as span:
        time.sleep(0.01)                      # pretend to call the API
        span["prompt_tokens"] = 420
        span["completion_tokens"] = 95
        span["cost_usd"] = 0.0008
        span["finish_reason"] = "stop"
        span["status"] = "ok"
    tr.report()

Log the what (feature, model, token counts, latency, cost, finish_reason, status) and a trace ID that links the LLM call to the tool calls and the user request. Be careful with prompts/responses: they may contain PII — redact or sample, don't log raw user data by default.

Caching: three kinds

For a store assistant, "what's your return policy" asked 500 times a day is one LLM call, not 500. But cache carefully: answers with personal or time-sensitive data shouldn't be served from a cache.

Routing: right model, right price

Not every call needs the frontier model. Route by task:

intent classification, extraction, short answers  -> small/fast model
complex reasoning, long documents, agent planning -> frontier model
refusals / safety checks                          -> fast, cheap model or code
embeddings                                        -> dedicated embedding model

Implement it as a thin layer in your client (L2·03's router pattern grows up): choose the model per request, log which was picked, and add a fallback — if the primary model errors or times out, retry a backup. Track quality per route so "we saved money by routing down" doesn't quietly become "we tanked accuracy".

Cost control

• Cost per request and per feature on a dashboard — tokens × price, summed by feature tag.
• Budgets that alert: daily spend ceiling; a runaway agent loop shouldn't drain the account overnight.
• Token budgets per call (max_tokens) and per agent run (L3·02 limits).
• Trim context: compaction, smaller retrieved sets, drop history you don't need.
• Prompt caching and smaller models for the bulk traffic; frontier for the hard slice.
• Watch the tail: a few users running giant contexts can dominate spend.

Guardrails in production

The deploy-and-monitor loop

• Canary new prompts/models to a % of traffic; compare eval scores before full rollout.
• Sample live traffic into the golden set monthly (privacy-aware) — real inputs keep evals honest.
• Collect lightweight feedback (👍/👎, "was this helpful?") as a weak quality signal.
• Alert on eval-score drop, cost spike, latency regression — the three silent killers.