L5 · 01 · Reading & Reproducing Research Papers

Level 5 is where you stop learning only from tutorials and start learning from the frontier itself. Papers are where new capabilities first appear — but they're dense, assume context, and sometimes oversell. This page gives you a method: read in three passes, extract the claim, reproduce it small, and write up what you found.

The three-pass read

Don't read a paper like a book. Keshav's "How to Read a Paper" gives the standard method:

Most papers deserve only pass 1. Pass 2 for papers you'll cite or build on. Pass 3 for the few that change how you work.

Extract the claim

Every paper makes a claim. Write it as a falsifiable sentence: "Method X improves metric Y on benchmark Z by N%, compared to baseline B." Then check:

• What's the baseline? Beating a weak baseline proves little. Is it the prior state of the art, or a strawman?
• Is the benchmark leaked? Test data in the training set inflates scores (contamination, L4·03).
• Do the error bars overlap? A 2% gain within noise is not a gain.
• Does the claim match the abstract? Abstracts sometimes claim more than the results section shows.
• What did they not measure? Cost, latency, failure modes are often absent.

Reproduce small, then ablate

You rarely need the full result. Reproduce the mechanism at toy scale — then ablate: remove one piece at a time to see which parts actually cause the effect. A result that only works with all 6 tricks is fragile; one that survives ablations is robust.

def experiment(use_technique: bool, seed: int = 0) -> float:
    """A reproduction harness: same task, technique on/off, fixed seeds.
    Returns a score; compare across seeds, not single runs."""
    import random
    rng = random.Random(seed)
    # Placeholder for the real task. Stand-in: technique adds a small boost.
    base = 0.70 + rng.uniform(-0.02, 0.02)
    return base + (0.05 if use_technique else 0.0)


def run(n_seeds: int = 5) -> dict:
    on = [experiment(True, s) for s in range(n_seeds)]
    off = [experiment(False, s) for s in range(n_seeds)]
    return {"with": sum(on) / n_seeds, "without": sum(off) / n_seeds}


if __name__ == "__main__":
    r = run()
    print(f"with technique:    {r['with']:.3f}")
    print(f"without technique: {r['without']:.3f}")
    print(f"delta:             {r['with'] - r['without']:+.3f}")
    assert r["with"] > r["without"]

The discipline: same data, same seeds, only the technique differs. Multiple seeds, because one lucky run proves nothing — the same rule as your eval suites (L3·04).

Using AI to read papers (carefully)

Rule: use AI to lower the entry cost, not to replace the read. You'll internalise far less if it pre-digests everything.

Staying current without drowning

• Curated feeds over firehose: a few newsletters/aggregators, not every arXiv ping.
• Pass 1 widely, pass 3 rarely. Skim a dozen papers a month; deep-read one.
• Keep a reading log: claim, method, verdict, one-line takeaway per paper.
• Prefer papers with code — reproducible claims are worth more.
• Talk it out: a reading group or one colleague you explain papers to forces the third pass.

Write it up

After a pass-3 read or reproduction, write one page: claim, method, your reproduction, what held up, what didn't, and "should we care?" This is what turns reading into knowledge you can share — and it's the capstone shape for Level 5.