L4 · 01 · The Maths You Actually Need

Levels 1–3 treated the model as an API. Level 4 opens it. You don't need a maths degree — you need six ideas, each of which fits in a few lines of code. This page implements all of them in plain Python: vectors, dot products, matrices, softmax, cross-entropy and gradient descent. These exact pieces reappear in the next page inside a transformer.

Vectors and the dot product

A vector is a list of numbers. In ML it usually represents "a point in meaning-space" (embeddings, L2·05) or "a row of activations" inside the model. The dot product of two vectors multiplies them element-wise and sums — it measures how much they point the same way:

import math


def dot(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def matmul(A: list[list[float]], B: list[list[float]]) -> list[list[float]]:
    """(m x n) @ (n x p) -> (m x p). Row i of A dotted with column j of B."""
    B_cols = list(zip(*B))                      # transpose: columns of B
    return [[dot(row, col) for col in B_cols] for row in A]


def cosine(a: list[float], b: list[float]) -> float:
    return dot(a, b) / (math.sqrt(dot(a, a)) * math.sqrt(dot(b, b)))


if __name__ == "__main__":
    print(dot([1, 2, 3], [4, 5, 6]))                    # 32.0
    A = [[1, 2], [3, 4]]                                # 2x2
    B = [[5, 6], [7, 8]]                                # 2x2
    print(matmul(A, B))                                 # [[19, 22], [43, 50]]
    print(round(cosine([1, 0], [1, 1]), 3))             # 0.707
    assert matmul(A, B) == [[19.0, 22.0], [43.0, 50.0]]

Softmax: scores into probabilities

A model outputs raw scores (logits) — arbitrary numbers like [2.1, -0.5, 0.9]. Softmax squashes them into a probability distribution: everything positive, summing to 1:

import math


def softmax(logits: list[float]) -> list[float]:
    m = max(logits)                    # subtract max for numerical stability
    exps = [math.exp(x - m) for x in logits]
    total = sum(exps)
    return [e / total for e in exps]


if __name__ == "__main__":
    p = softmax([2.1, -0.5, 0.9])
    print([round(x, 3) for x in p], "sum =", round(sum(p), 6))
    # -> [0.645, 0.048, 0.307]  sum = 1.0
    assert abs(sum(p) - 1.0) < 1e-9

Why exponentials? They make scores positive and amplify differences — the highest logit dominates. Next page: this exact function is what picks the next token, and what attention uses to weight values.

Cross-entropy: how surprised was the model?

Training needs a number for "how wrong was that prediction". If the model put probability p on the correct token, the cross-entropy loss is -log(p):

• p = 1.0 (certain and right) → loss = 0
• p = 0.5 → loss ≈ 0.69
• p = 0.01 (confident and wrong) → loss = 4.6 — heavily punished

import math


def cross_entropy(probs: list[float], correct: int) -> float:
    return -math.log(max(probs[correct], 1e-12))


if __name__ == "__main__":
    for p in [1.0, 0.9, 0.5, 0.1, 0.01]:
        print(f"p={p:<5} loss={round(-math.log(p), 3)}")
    # 0.0, 0.105, 0.693, 2.303, 4.605

Training sums this over billions of tokens. "Perplexity" you'll see in papers is just e^(average loss) — "on average, the model was choosing between this many equally likely options".

Gradient descent: how learning actually happens

A model is a giant function f(weights, input) → logits. Training: compute the loss, find each weight's gradient (how much it contributed to the loss), and nudge every weight a tiny step downhill. Repeat billions of times.

def loss_and_grad(w: float, data: list[tuple[float, float]]) -> tuple[float, float]:
    """Mean squared error for y = w*x, plus d(loss)/dw."""
    n = len(data)
    loss = sum((w * x - y) ** 2 for x, y in data) / n
    grad = sum(2 * (w * x - y) * x for x, y in data) / n
    return loss, grad


if __name__ == "__main__":
    data = [(1, 2), (2, 4), (3, 6), (4, 8)]   # true w = 2
    w, lr = 0.0, 0.05
    for step in range(50):
        loss, g = loss_and_grad(w, data)
        w -= lr * g                          # step downhill
    print(f"learned w = {w:.3f}  loss = {loss:.6f}")   # w -> 2.0
    assert abs(w - 2.0) < 0.01

That's the whole algorithm. A real model has billions of ws and computes all gradients at once with backpropagation — the chain rule applied layer by layer — but the idea is identical: measure the error, nudge every knob a little.

Putting it together: a 20-line language model

import math
import random


def softmax(logits):
    m = max(logits)
    e = [math.exp(x - m) for x in logits]
    s = sum(e)
    return [v / s for v in e]


# A tiny bigram "language model": P(next | current) is one lookup + softmax.
# Real LLMs compute these logits with a transformer instead of a table.
VOCAB = ["cat", "dog", "sat", "ran"]
LOGITS = {                       # learned "weights" for next-token prediction
    "cat": [0.1, 0.1, 2.0, 0.3],
    "dog": [0.1, 0.1, 0.4, 2.0],
    "sat": [2.0, 1.8, 0.0, 0.0],
    "ran": [1.9, 2.0, 0.0, 0.0],
}


def sample_next(word: str, rng) -> str:
    probs = softmax(LOGITS[word])
    return rng.choices(VOCAB, weights=probs)[0]


if __name__ == "__main__":
    rng = random.Random(42)
    word = "cat"
    out = [word]
    for _ in range(4):
        word = sample_next(word, rng)
        out.append(word)
    print(" ".join(out))         # e.g. "cat sat dog ran dog"
    assert len(out) == 5

Every concept on this page appears in it: a weight table (matrices), softmax, sampling, cross-entropy lurking as "how good are these weights". The transformer just replaces the lookup table with a very good way to compute the logits.