02 · How LLMs Actually Work

You don't need the maths to understand how an LLM works. You do need the right mental model. With it you can predict when the model will shine, when it will fail, and why prompting and context matter so much.

The core idea: predict the next token

An LLM does one thing over and over. Given some text, it predicts what piece of text comes next, adds that piece, and repeats. Everything else (answering questions, writing code, translating, reasoning) emerges from doing this extremely well after training on a vast amount of text.

Tokens: how models see text

Models don't read characters or words. They read tokens: chunks of text from a fixed vocabulary, typically 30,000 to 200,000+ entries. A tokenizer splits text into tokens and maps each one to an integer ID.

"Scala is functional"       -> ["Sc", "ala", " is", " functional"]
"val total = xs.sum"        -> ["val", " total", " =", " xs", ".", "sum"]
"unbelievably"              -> ["un", "belie", "vably"]
"    def f():" (Python)     -> ["    ", "def", " f", "():"]

• Rule of thumb: in English, 1 token ≈ 4 characters ≈ ¾ of a word. 1,000 tokens ≈ 750 words. Code, non-English text and unusual identifiers often use more tokens.
• Why it matters: pricing, speed and the context window are all measured in tokens.
• Quirks it explains: models sometimes struggle to count letters in a word or reverse strings, because they see tokens, not individual letters.

Embeddings: meaning as numbers

Each token ID is turned into an embedding: a list of hundreds or thousands of numbers (a vector). Training arranges these vectors so that similar meanings end up close together.

You can do "maths with meaning". The classic example is that king − man + woman lands near queen. Embeddings are also used outside the LLM to search documents by meaning. That is the foundation of RAG.

The Transformer and attention

The Transformer is the neural network architecture behind modern LLMs. Its key trick is attention: for every token, the model decides which other tokens in the context are relevant and blends information from them.

A Transformer stacks dozens of layers of attention plus small feed-forward networks. Early layers pick up syntax and local patterns. Deeper layers represent more abstract things: types, intent, style, logic. Attention works in parallel across all tokens, so Transformers train efficiently on GPUs, and that is why they scaled.

How a model is trained

• Parameters / weights: the numbers the model learns. Model sizes range from about 1 billion (runs on a laptop) to hundreds of billions or more.
• Knowledge cutoff: the model only "knows" data up to when training data was collected. Anything newer must be given in the prompt or through tools such as search.
• RLHF (Reinforcement Learning from Human Feedback) and related methods teach style, helpfulness, honesty and refusal behaviour.
• RL with verifiable rewards: models practise on problems where answers can be checked automatically (tests pass, maths is correct). This is a big reason coding ability improved so quickly.

Inference: generating an answer

When you send a prompt, the model runs inference. It computes a probability for every possible next token, picks one according to the sampling settings, and repeats until it emits a stop token or hits the limit.

The context window

The context window is the maximum number of tokens the model can consider at once: the system prompt, conversation history, files, tool results and its own answer. Modern models range from tens of thousands to a million or more tokens.

• The model has no memory between calls. Each request sends the whole conversation again. "Memory" features are the product re-inserting saved notes.
• More context is not always better. Models can lose track of details buried in the middle of very long contexts ("lost in the middle"), and long contexts cost more and run slower. See Context Engineering.

Messages and roles

Chat models receive a list of messages, each with a role:

{
  "messages": [
    { "role": "system",    "content": "You are a senior Scala 3 reviewer. Be concise." },
    { "role": "user",      "content": "Review this function: def avg(xs: List[Int]) = xs.sum / xs.size" },
    { "role": "assistant", "content": "It throws ArithmeticException on an empty list..." },
    { "role": "user",      "content": "Fix it using Option." }
  ],
  "temperature": 0.2,
  "max_tokens": 800
}

• system (sometimes called developer): sets the persona, rules and constraints. It carries higher priority.
• user: your requests.
• assistant: the model's previous replies.
• tool: results returned from tools the model called (agents, MCP; see pages 08–09).

Why hallucinations happen

A hallucination is fluent, confident output that is false: a made-up API, a non-existent library version, a fake citation. It happens because the model is trained to produce plausible continuations, not to check facts. When the true answer was rare or absent in its training data, a plausible-looking guess often wins.

• Ground the model: paste the real docs, error output or source code into context.
• Let it use tools: compile, run tests, search docs (agents do this).
• Ask for sources or "say if you are unsure", and treat that as a hint only.
• Verify anything critical: versions, APIs, security, money, legal, medical.

Reasoning ("thinking") models

Many current models can spend extra tokens thinking before answering: they work through a problem step by step internally, check intermediate results, and then give a final answer. This uses more test-time compute. It helps with multi-step logic, debugging, maths and planning. It adds latency and cost and is unnecessary for simple tasks.

Open-weight vs closed models

Build intuition: a toy next-word model

Real LLMs use neural networks. A tiny bigram model (next word based only on the previous word) still shows the core loop: learn probabilities from data, then sample.

import random
from collections import defaultdict, Counter

corpus = "the cat sat on the mat . the dog sat on the log . the cat saw the dog ."
words = corpus.split()

# 1. "Training": count which word follows which
follows: dict[str, Counter] = defaultdict(Counter)
for prev, nxt in zip(words, words[1:]):
    follows[prev][nxt] += 1

# 2. "Inference": sample the next word, with temperature
def next_word(prev: str, temperature: float = 1.0) -> str:
    options = follows[prev]
    weights = [count ** (1 / temperature) for count in options.values()]
    return random.choices(list(options), weights=weights)[0]

def generate(start: str, n: int = 8, temperature: float = 1.0) -> str:
    out = [start]
    for _ in range(n):
        out.append(next_word(out[-1], temperature))
    return " ".join(out)

print(generate("the", temperature=0.3))  # focused
print(generate("the", temperature=2.0))  # more random

//> using scala 3
import scala.util.Random

@main def toyBigram(): Unit =
  val words = "the cat sat on the mat . the dog sat on the log . the cat saw the dog .".split(" ").toVector

  // 1. "Training": count transitions
  val follows: Map[String, Map[String, Int]] =
    words.zip(words.tail)
      .groupMap(_._1)(_._2)
      .view.mapValues(_.groupMapReduce(identity)(_ => 1)(_ + _)).toMap

  // 2. "Inference": sample with temperature
  def nextWord(prev: String, temperature: Double): String =
    val opts    = follows(prev).toVector
    val weights = opts.map((_, c) => math.pow(c, 1 / temperature))
    var r       = Random.nextDouble() * weights.sum
    opts.zip(weights).find { case (_, w) => r -= w; r <= 0 }.map(_._1._1).getOrElse(opts.last._1)

  def generate(start: String, n: Int, t: Double): String =
    Iterator.iterate(start)(w => nextWord(w, t)).take(n + 1).mkString(" ")

  println(generate("the", 8, 0.3))
  println(generate("the", 8, 2.0))

An LLM replaces "count the previous word" with "a Transformer with billions of parameters that looks at thousands of previous tokens". The loop is the same.