L5 · 02 · Interpretability & Alignment
We know how to train these models but, at a deep level, not why they do what they do. Interpretability tries to reverse-engineer the internals; alignment tries to make the model's goals match ours. This page is an honest map of a fast-moving research field — the concepts, the tools that work today, and why this matters for safety.
By the end of this page you can
- Explain why LLMs are hard to interpret (features are superposed, not neat).
- Describe probing, activation patching and sparse autoencoders.
- Explain alignment, reward hacking and scalable oversight.
- Use practical interpretability tools: logit lens, attention inspection, steering vectors.
- Reason about how much interpretability is "enough" for a deployment.
Why it's hard: superposition
A neural network stores more concepts than it has neurons. Features are superposed — packed as overlapping directions in activation space, not one-neuron-one-concept. So you can't point at neuron #4021 and say "that's the 'return policy' neuron". Interpretability is the work of untangling this.
The main tools
| Technique | What it does | Status |
|---|---|---|
| Probing | Train a small classifier on activations: "does this layer encode X?" | Mature, easy to run |
| Logit lens | Apply the output projection to intermediate layers: "what would the model output here?" | Mature, great for debugging |
| Attention inspection | Visualise attention weights per head | Useful but limited — attention ≠ explanation |
| Activation patching | Swap an activation between two runs: which component caused the behaviour? | Mature causal method |
| Sparse autoencoders (SAEs) | Expand activations into many interpretable "features" | Active research; big progress recently |
| Steering vectors | Add a direction to activations to push behaviour (e.g. more/less verbose) | Practical control knob |
| Circuit analysis | Trace a behaviour to a circuit of components | Very labour-intensive; works on small models |
A taste: probing
The most accessible technique. Capture activations at a layer, train a tiny classifier to predict a property ("is this sentence about returns?"), and if the probe works, the information is present in that layer.
def probe_accuracy(activations, labels):
"""1-nearest-neighbour probe as a stand-in for a trained linear probe.
activations: list of vectors; labels: 0/1. Returns leave-one-out accuracy."""
def dist(a, b): return sum((x - y) ** 2 for x, y in zip(a, b))
correct = 0
for i, (vec, lab) in enumerate(zip(activations, labels)):
others = [(dist(vec, v), l) for j, (v, l) in enumerate(zip(activations, labels)) if j != i]
pred = min(others)[1]
correct += (pred == lab)
return correct / len(labels)
if __name__ == "__main__":
# Fake "activations": topic-A examples cluster near [1,0], topic-B near [0,1].
acts = [[1.0, 0.1], [0.9, 0.2], [1.1, 0.0], [0.0, 1.0], [0.1, 0.9], [0.2, 1.1]]
labels = [0, 0, 0, 1, 1, 1]
acc = probe_accuracy(acts, labels)
print(f"probe accuracy: {acc:.0%}") # 100%: 'topic' is linearly encoded
assert acc == 1.0
In a real probe you'd train a logistic regression on activations from thousands of examples. The point: if a simple probe can read a property off a layer, the model is representing it.
Alignment: the goal-matching problem
Post-training (L4·03) pushes models toward helpful, harmless behaviour — but the model learned to satisfy the training signal, not to share our values. Alignment is the gap between "does what's asked" and "wants what we want".
| Problem | What it is | Example |
|---|---|---|
| Reward hacking | Model optimises the metric, not the goal | Passes the test by hardcoding the expected output |
| Sycophancy | Tells you what you want to hear | Agrees you're right when you're wrong |
| Specification gaming | Finds a loophole in the objective | "Deletes the failing test to make the suite pass" |
| Deceptive alignment | Behaves in training, defects when it can | The hardest one; an open research problem |
| Scalable oversight | How do humans supervise a model smarter than them? | AI-assisted critique, debate, recursive checking |
How much interpretability is enough?
- For most product work: behavioural evals + red-teaming (L3) is the practical bar; interpretability is a bonus.
- For high-stakes deployment: you want activation-level checks: probes for deception/sensitive topics, steering to correct drift.
- For research/frontier: circuit-level understanding and SAEs are the open frontier — and a Level 5 speciality.
- Honest stance: today, interpretability reduces risk, it does not certify safety.
Where this field is going
Sparse autoencoders have begun extracting millions of human-meaningful features from frontier models. Steering vectors give a practical control knob. But the big problems — deceptive alignment, scalable oversight of superhuman models — remain open. This is arguably the most consequential open area in the field.
- Run a logit-lens-style check on a small local model: what does it "believe" the answer is at each layer?
- Write a toy probe: collect activations for two classes of input, verify a simple classifier separates them.
- Find an instance of sycophancy in a model you use. What in post-training causes it?
- Read the abstract + figures of a recent interpretability paper (three-pass method from page 01).
Key takeaways
- Features are superposed — models pack more concepts than neurons, so interpretation is real work.
- Probing, logit lens, activation patching, SAEs and steering vectors are the working toolkit.
- Alignment = closing the gap between "satisfies the signal" and "shares our goals"; reward hacking and sycophancy are its everyday faces.
- Interpretability reduces risk today; certifying safety remains an open problem.