L5 · 03 · On-device AI: Apple Silicon, Core ML & MLX

Running models on the device — not a server — is a different discipline: hard memory limits, battery budgets, the Neural Engine alongside CPU and GPU, and privacy as a feature. This page is a deep dive into on-device inference, centred on Apple silicon (unified memory, Core ML, MLX), with the general principles that apply to any edge device.

Why on-device is different

Apple silicon's three engines

• Unified memory means no PCIe copy between CPU/GPU/ANE — a big advantage over PC architectures, and why Macs hold large models (L2·11).
• Memory bandwidth is the decode bottleneck (L4·04). Pro vs. Max vs. Ultra chips differ mostly in bandwidth, which is why a Max generates tokens faster.
• ANE is the most power-efficient for sustained ML but is less flexible than the GPU — Core ML decides where ops run.

The three ways to run a model on a Mac

The on-device constraint checklist

• Size: a phone/tablet budget is ~1–4 GB for the model. Quantise hard (4-bit, sometimes 3–2-bit) and consider distillation (L3·06).
• Power: sustained inference drains battery and heats the device → thermal throttling. Prefer ANE for sustained work.
• First-token latency: users feel it instantly. Small models, prefix caching, streaming output all help.
• Model format: Core ML wants .mlpackage; MLX wants its own; llama.cpp wants GGUF. Conversion is a step in the pipeline.
• Quality at small size: a 3B on-device model won't match a frontier model — scope the task to what small models do well.

Framework Models API (Apple's on-device LLM)

On Apple platforms there's a system-provided foundation model exposed to apps (Apple Intelligence's on-device model). Characteristics that matter for design:

• It runs on-device, is free to the app, and preserves privacy by design.
• It's a small model: good for summarising, extraction, structured output and short text — not a frontier reasoner.
• Guided generation gives structured output directly; tool calling lets it invoke app functions.
• When it's not enough, the design is hybrid: on-device for most, escalate to a server model for hard cases (with user consent).

Design pattern: the hybrid on-device app

CONFIDENCE_FLOOR = 0.75


def on_device(query: str) -> dict:
    """Small local model: cheap, private, offline. Returns a confidence."""
    # stand-in for Core ML / MLX model output
    simple = len(query.split()) <= 8
    return {"answer": f"local answer for {query[:24]!r}",
            "confidence": 0.9 if simple else 0.5}


def cloud(query: str) -> dict:
    return {"answer": f"cloud answer for {query[:24]!r}", "confidence": 0.98}


def answer(query: str) -> dict:
    r = on_device(query)
    if r["confidence"] >= CONFIDENCE_FLOOR:
        r["via"] = "on-device"
        return r
    r = cloud(query)            # escalate only the hard cases
    r["via"] = "cloud"
    return r


if __name__ == "__main__":
    for q in ["return policy?", "write a detailed comparison of our five return edge cases"]:
        r = answer(q)
        print(f"{r['via']:>9} ({r['confidence']})  {q[:40]}")
    assert answer("hi")["via"] == "on-device"
    assert answer("a very long complex question with so many words indeed")["via"] == "cloud"

Most queries stay private and instant on-device; only hard ones hit the network. This is the architecture pattern behind privacy-first AI features.

Privacy as a feature

On-device inference isn't just a constraint — it's a product capability: "your data never leaves the device." Health, personal documents, offline fieldwork and regulated industries all pay a premium for it. When you can answer "where does the data go?" with "nowhere", a whole class of L4·07 concerns disappears.