L2 · 11 · Local Models: Ollama, LM Studio, MLX

Running models on your own machine gives you privacy, offline use, zero per-token cost and a much better feel for how models behave. This page covers choosing a model that fits your hardware, quantisation, the main local runners (including MLX on Apple silicon), and when local is the right call.

When local makes sense

Will it fit? A memory rule of thumb

memory needed ≈ parameters × (bits per weight / 8)  +  20–30% overhead (context / KV cache, runtime)

Quantisation

Quantisation stores weights with fewer bits (for example 4 instead of 16). Models become 3–4× smaller and faster, with a modest quality loss that is usually acceptable at 4–8 bits.

Rule: prefer a bigger model at 4-bit over a smaller model at 16-bit for the same memory, then test on your own tasks.

The local runners

Ollama: simplest CLI + server

ollama pull llama3.2              # small, fast general model
ollama run llama3.2 "Summarise our 14-day return policy in one line: ..."
ollama pull nomic-embed-text      # embeddings for page 05
ollama list                       # installed models
# OpenAI-compatible API at http://localhost:11434/v1

LM Studio: desktop app

Browse, download and chat with models in a GUI. It shows whether a model fits your memory. Start its local server to get an OpenAI-compatible endpoint (by default on http://localhost:1234/v1). Good for comparing models side by side.

llama.cpp: the engine

llama-server -m ./models/some-model-Q4_K_M.gguf --port 8080
# OpenAI-compatible API at http://localhost:8080/v1

MLX on Apple silicon

uv tool install mlx-lm            # or: pip install mlx-lm
mlx_lm.generate --model mlx-community/<model>-4bit --prompt "Explain opaque types in Scala 3"
mlx_lm.server   --model mlx-community/<model>-4bit --port 8081   # OpenAI-compatible server

MLX also supports fine-tuning (LoRA) on a Mac, which is covered in Level 3.

One client, any runner

Because all of these speak the OpenAI-compatible API, the clients from pages 03–06 work unchanged. Just set the base URL:

Benchmark on your own tasks

import json
import os
import time

import httpx

BASE = os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1")
MODEL = os.environ.get("LLM_MODEL", "llama3.2")

TASKS = [  # (prompt, check) - checks are deliberately simple; use golden sets for real evals
    ("Classify as PICKUP, RETURN, PAYMENT or OTHER. Label only: 'My card was charged twice'", lambda a: "PAYMENT" in a.upper()),
    ("What is 17% of 2,400? Number only.", lambda a: "408" in a),
    ("Return only JSON {\"sku\":string,\"qty\":int} for: two chargers, sku CHARGER", lambda a: '"qty": 2' in a or '"qty":2' in a),
]


def run(prompt: str) -> tuple[str, float, float, int]:
    start, first, chunks, text = time.perf_counter(), None, 0, []
    with httpx.stream("POST", f"{BASE}/chat/completions", timeout=120,
                      json={"model": MODEL, "stream": True, "temperature": 0,
                            "messages": [{"role": "user", "content": prompt}]}) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line.startswith("data: ") or line.endswith("[DONE]"):
                continue
            delta = json.loads(line[6:])["choices"][0].get("delta", {}).get("content")
            if delta:
                first = first or time.perf_counter()
                chunks += 1
                text.append(delta)
    end = time.perf_counter()
    return "".join(text), (first or end) - start, end - start, chunks


if __name__ == "__main__":
    passed = 0
    for prompt, check in TASKS:
        answer, ttft, total, chunks = run(prompt)
        ok = check(answer)
        passed += ok
        rate = chunks / max(total - ttft, 1e-6)
        print(f"{'PASS' if ok else 'FAIL'}  first-token {ttft:.2f}s  ~{rate:.0f} chunks/s  {answer.strip()[:60]!r}")
    print(f"\n{MODEL}: {passed}/{len(TASKS)} passed")

Run it for 2–3 candidate models and keep the results in your learning log. You will quickly build intuition for which sizes handle which tasks.

Practical tips

• Use instruction-tuned models ("instruct" or "chat" variants) for chat and tools, not base models.
• Context length costs memory. Increase it only when needed (most runners have a setting).
• Tool calling and JSON quality vary a lot between local models. Test with your real tools and schemas (pages 02 and 04).
• Check licences of open-weight models before using them for work. Some restrict commercial use.
• Local still needs policy approval. Data never leaves the machine, but model provenance and licences still matter to employers.
• Keep models tidy: they are several GB each (ollama rm, LM Studio's model manager).