L4 · 04 · Inference Optimisation & Serving

Why is generation slow, why does memory blow up on long prompts, and how do servers answer 100 users at once? Inference is its own engineering field. This page covers the KV cache, batching, paged attention, speculative decoding, quantisation for serving, and how to read a latency benchmark.

Level 460 minTracks 1, 18Last verified 2026-10

By the end of this page you can

Explain the two phases (prefill vs. decode) and why they bottleneck differently.
Explain the KV cache and why long prompts and long generations cost memory.
Explain batching, continuous batching and paged attention.
Explain speculative decoding and quantisation for serving.
Choose a serving engine and reason about latency vs. throughput.

Two phases, two bottlenecks

	Prefill	Decode
What	Process the whole prompt at once (parallel over tokens)	Generate tokens one at a time (sequential)
Bottleneck	Compute — big matrix multiplies	Memory bandwidth — reading all weights per token
You feel it as	Time to first token (TTFT)	Tokens/second
Worst when	Huge prompts (RAG, long docs)	Long generations, many concurrent users

Why Macs and GPUs differ Generation speed is dominated by how fast you can stream the weights from memory — memory bandwidth, not FLOPS. That's why a high-bandwidth Mac runs a 4-bit model surprisingly well, and why datacenter GPUs (huge bandwidth) serve many users at once. Same reason smaller/quantised models feel faster: less data to read per token.

The KV cache

Without caching, generating token 1000 would recompute attention over all 1000 previous tokens from scratch. The KV cache stores every past token's keys and values, so each new token only computes its own Q/K/V and attends to the cache. This is why generation is feasible — and why memory grows:

KV bytes ≈ 2 (K and V) × n_layers × n_kv_heads × head_dim × seq_len × bytes_per_value

For a 70B model at 100k context, that's tens of GB — per request. Multiply by concurrent users and the cache, not the weights, becomes the memory problem. This is why long context is expensive even on a hosted API.

Serving many users: batching and paging

Static batching: wait for N requests, run them together. Simple but wastes GPU when they finish at different times.
Continuous batching: as one request finishes a step, slot another in immediately. The default in modern servers; huge throughput win.
Paged attention (vLLM): manage the KV cache like OS virtual memory — allocate in blocks, share across requests, no giant contiguous buffers. Far less wasted memory → more concurrent users.

Going faster

Technique	What it is	Trade-off
Speculative decoding	A small "draft" model guesses several tokens; the big model verifies them in one pass. Accept what's right.	~2–3× speedup, same output; extra memory for draft model
Quantisation (serving)	4/8-bit weights (and sometimes KV) via AWQ/GPTQ/FP8.	Less memory, faster bandwidth-bound decode; small quality cost
Flash/kernels	Fused attention kernels (FlashAttention) compute attention without materialising the n×n matrix.	Faster + less memory; requires supported hardware
Prefix/prompt caching	Re-use the KV cache for a shared prompt prefix across requests.	Huge win for agents with the same long system prompt

def speculative_step(draft, big, prompt_tokens, k=3):
    """Draft proposes k tokens; big model verifies them in ONE forward pass."""
    proposed = draft.generate(prompt_tokens, k)          # cheap, k tokens
    verified = []
    for i, tok in enumerate(proposed):
        # big model scores all k positions in parallel (not k separate calls)
        accepted = big.agrees(prompt_tokens + verified, tok)
        if not accepted:
            verified.append(big.sample(prompt_tokens + verified))  # big model fixes it
            break
        verified.append(tok)
    return verified


class Toy:
    def __init__(self, script): self.script = script
    def generate(self, ctx, k): return self.script[:k]
    def agrees(self, ctx, tok): return tok in self.script
    def sample(self, ctx): return self.script[len(ctx)]   # next correct token


if __name__ == "__main__":
    big = Toy(["the", "cat", "sat"])
    draft = Toy(["the", "dog", "sat"])
    print(speculative_step(draft, big, []))
    # draft got "the" right, "dog" wrong -> big model emits "cat", stops
    assert speculative_step(draft, big, []) == ["the", "cat"]

The point: the expensive model does one parallel scoring pass instead of k sequential generations.

Choosing a serving engine

Engine	Best for
vLLM	High-throughput GPU serving; paged attention, continuous batching. The common default.
llama.cpp	CPU / edge / single user; GGUF models (L2·11).
MLX	Apple silicon, local and near-production.
TensorRT-LLM / TGI	NVIDIA-optimised / general hosted serving.
Managed APIs	No ops at all — you pay per token instead of per GPU.

Reading a benchmark

TTFT (time to first token) — prefill speed; what interactive users feel.
Tokens/sec per request — decode speed.
Total throughput (tokens/sec across all users) — what you pay for on a server.
p50/p95/p99, not just averages — the tail is what SLOs are made of.
Latency and throughput trade off: bigger batches = more throughput, worse per-user latency.

Try it (60 min)

Serve a model with vLLM or llama.cpp server. Hit it with 1 request, then 10 concurrent. How do TTFT and tokens/s change?
Send a 4k-token prompt vs a 20-token prompt. Compare TTFT — feel the prefill cost.
Estimate KV-cache memory for your model at 8k and 128k context. When does it exceed the weights?
Compare a 4-bit and 8-bit quantisation on the same hardware: speed vs. quality on your task.

Key takeaways

Prefill is compute-bound (TTFT); decode is memory-bandwidth-bound (tokens/sec).
The KV cache makes generation possible but grows with context × users.
Continuous batching + paged attention are why servers handle many users.
Speculative decoding, quantisation, fused kernels and prefix caching each buy speed — for a price.