L3 · 05 · Advanced RAG & Knowledge Graphs

Basic RAG (embed, retrieve top-k, prompt) works for demos. Production retrieval needs hybrid search, re-ranking, query rewriting and honest evaluation. This page adds each layer, explains when knowledge graphs help, and finishes with agentic retrieval — where the model decides what to look up.

The full retrieval pipeline

Each stage fixes a failure of the previous one. Add them only when you can show the failure exists (L3·04 evals).

Stage 1: fix the query first

Users type "the thing for earbuds" when they mean "EARBUDS-AIR2 silicone tips". Before retrieving:

Stage 2: hybrid retrieval

Vector search understands meaning but misses exact codes ("ORD-482913"). Keyword search (BM25) nails codes but misses meaning. Use both, fuse the rankings with Reciprocal Rank Fusion — no score calibration needed, just rank positions:

import math
import re
from collections import Counter


def tokenize(text: str) -> list[str]:
    return re.findall(r"[a-z0-9\-]+", text.lower())


def bm25_scores(query: str, docs: list[str], k1: float = 1.5, b: float = 0.75) -> list[float]:
    """Classic BM25 over the corpus."""
    q_terms, doc_terms = tokenize(query), [tokenize(d) for d in docs]
    avg_len = sum(len(t) for t in doc_terms) / len(doc_terms)
    df = Counter(t for terms in doc_terms for t in set(terms))
    n = len(docs)
    scores = []
    for terms in doc_terms:
        tf = Counter(terms)
        score = sum(
            math.log((n - df[t] + 0.5) / (df[t] + 0.5) + 1)
            * (tf[t] * (k1 + 1)) / (tf[t] + k1 * (1 - b + b * len(terms) / avg_len))
            for t in q_terms if t in tf)
        scores.append(score)
    return scores


def ranks(scores: list[float]) -> list[int]:
    """Index order sorted by score descending."""
    return sorted(range(len(scores)), key=lambda i: -scores[i])


def rrf(bm25_rank: list[int], vec_rank: list[int], k: int = 60) -> list[int]:
    """Fuse two rankings by rank position, not raw score."""
    fused = Counter()
    for pos, idx in enumerate(bm25_rank):
        fused[idx] += 1 / (k + pos + 1)
    for pos, idx in enumerate(vec_rank):
        fused[idx] += 1 / (k + pos + 1)
    return [i for i, _ in fused.most_common()]


if __name__ == "__main__":
    docs = [
        "Return policy: unopened items within 30 days, receipt required.",
        "Order ORD-482913 is ready for pickup at the Riverside store.",
        "Wireless earbuds with noise cancellation and 24h battery.",
        "Opened electronics may be returned within 14 days with receipt.",
    ]
    # Query with an exact code -> BM25 wins. Semantic query -> vector wins.
    # Here we fake vector ranking to show fusion; real code embeds query+docs.
    for query, fake_vec_rank in [
        ("ORD-482913 status", [1, 3, 0, 2]),
        ("can I bring back opened earphones", [3, 2, 0, 1]),
    ]:
        fused = rrf(ranks(bm25_scores(query, docs)), fake_vec_rank)
        top = docs[fused[0]]
        print(f"{query!r} -> doc {fused[0]}: {top[:55]}...")
    assert True

Notice the fusion: BM25 ranked the exact-code doc first, the fake vector ranking agreed, and the merged list surfaces the right doc for both query types.

Stage 3: re-rank for precision

Retrieval is recall-oriented: fetch 30–50 candidates cheaply. A re-ranker (a cross-encoder or an LLM) then scores each candidate against the query properly and keeps the top 5. It's slower per item but applied to few items — the classic "cheap recall, expensive precision" split:

• Cross-encoder re-ranker (a small model that reads query+doc together): fast, consistent, run locally.
• LLM re-ranker: prompt "rate relevance 0–10" per doc, or listwise ranking. More expensive, more flexible.

Stage 4: when chunks aren't enough — knowledge graphs

Plain chunk retrieval fails on multi-hop questions: "Which stores selling the earbuds also had a June staffing gap?" needs store → product → staffing relationships, not text similarity.

GraphRAG extracts entities and relationships into a graph at index time (LLM-assisted), then retrieves subgraphs and community summaries, not just chunks. More work to build; only worth it for genuinely relational questions.

Stage 5: agentic retrieval

The endpoint of RAG sophistication is giving the model retrieval as a tool: it decides whether to search, what to search for, reads the result, and searches again if needed (a small agent loop, L3·02). Use it when queries are unpredictable or multi-step. The cost is latency and non-determinism — for fixed FAQ-style traffic, a static pipeline is cheaper and more predictable.

Measure retrieval separately from answering

If recall@5 is 60%, no prompt will save you — fix retrieval. If recall is 95% but answers are wrong, fix generation. Always know which side is failing before you tune.