L4 · 06 · AI-Native Architecture & Build vs Buy
Adding AI to a product isn't like adding a normal feature: outputs are probabilistic, costs scale per call, the ground shifts under you monthly, and "it worked in the demo" is not an SLA. This page is the architecture-level view: designing AI-native systems, deciding what to build vs. buy, structuring a platform team, and reasoning about reliability and cost at the system level.
By the end of this page you can
- Explain what makes AI-native systems different from classic software.
- Apply a build-vs-buy framework at the model, framework and feature level.
- Design for probabilistic output: evals, fallbacks, human-in-the-loop.
- Structure a platform/gateway layer that lets teams adopt AI safely.
- Model cost, reliability and vendor risk at the architecture level.
Why AI-native is different
| Classic software | AI-native software |
|---|---|
| Deterministic: same input, same output | Probabilistic: plausible-but-wrong is a normal outcome |
| Bugs are code errors | Failures are silent: a bad answer, not an exception |
| Unit tests prove correctness | Evals measure quality on a distribution |
| Cost scales with users (roughly flat/request) | Cost scales per call with tokens — context length matters |
| Dependencies are stable APIs | The model under you updates silently |
The reference architecture
The gateway is the key idea: one place where every LLM call passes through. Routing, budgets, guardrails, caching, tracing and fallback live there — not scattered across features. It gives you a single point to change models, enforce policy and observe cost, and it lets product teams use AI without each one reinventing the plumbing.
Build vs buy, at three levels
| Level | Buy (API / platform) | Build (self-host / own) |
|---|---|---|
| Model | Default for most teams: frontier quality, zero ops | When: data residency, volume cost, offline, custom fine-tune at scale |
| Serving/framework | Managed platforms get you running fast | When you outgrow their limits or need control (your own gateway, vLLM) |
| Feature (e.g. support bot) | SaaS copilots for generic needs | When it's your differentiator or needs your data/tools |
Designing for probabilistic output
- Evals before launch (L3·04) and in CI — the acceptance test is a score, not "it ran".
- Graceful degradation: when confidence is low, abstain, fall back to a smaller model, a rule, or a human.
- Human-in-the-loop for consequential actions: design the approval step, not just the prompt.
- Guardrails as a layer, not scattered prompt pleading (L3·08).
- Idempotent tools so a retried tool call doesn't double-charge or double-book.
- Version everything — prompts, models, eval sets — so "it got worse" is measurable and revertable.
The platform team pattern
As AI adoption grows inside a company, the winning structure is a small platform team that owns the gateway, eval infrastructure, observability and approved model list — while product teams build features on top. This is the same shape as internal developer platforms, with the same benefits (consistency, safety, speed) and the same risk (the platform becomes a bottleneck if it's mandatory and slow).
Cost and reliability at the system level
- Cost model: cost per task = tokens × price × calls/task. Reasoning models and long contexts change it by 10–100×.
- Vendor risk: a single provider is a single point of failure and a pricing hostage. Design the gateway so a second provider is a config change.
- Reliability: LLM features inherit provider uptime + your model's error rate. SLA thinking applies: what does "available" mean when the answer can be wrong?
- Data boundary: where does data go? Which providers retain it? For many companies this decides build-vs-buy more than price.
- Sketch your own system's AI gateway: routing, budgets, guardrails, tracing, fallback. What would each feature need?
- Do the build-vs-buy table for one real feature you know (support bot, code review, search). Which cells change in a year?
- Write a cost model for a feature: tokens/call × calls/day × price. Where does it break at 10× usage?
Key takeaways
- AI-native = architecture for uncertainty: evals, guardrails, fallbacks, human approval.
- Put a gateway between products and models: one place for routing, cost, safety, observability.
- Build vs buy is per-level (model/serving/feature) and drifts — keep models swappable.
- A small platform team enables product teams; don't let it become the bottleneck.