Skip to content
Principal AI Engineer Handbook

Hybrid Retrieval and Evaluation

Listen to this page5:19
Read the transcript

1. A Retrieval Pipeline Built to Be Measured

Host: Today we’re digging into a retrieval lab that its own authors describe as being built to be measured, not just built to work. The premise is that a pipeline should tell you, in numbers, whether a change to chunking or fusion actually helped, instead of leaving you with a vague sense that the answers seem better. So before we get into the fusion math and the reranking, I want to start with the foundation: what makes this a real engineering artifact instead of a demo you’d throw together in an afternoon?

Guest: The tell is in the boring details nobody bothers with in a toy. Chunking here respects section headings and only splits a paragraph if that paragraph alone blows the size budget, with overlap carried across boundaries inside a section, because chunking on a fixed character count is the single most common way to quietly wreck retrieval. Then the lexical side is real Okapi BM25 with IDF computed over the actual corpus, not a keyword filter pretending to be search, and reranking only ever touches the fused candidate set so the expensive scoring function stays affordable. On top of that, groundedness is checked as a retrieval-boundary property, catching citations that point at chunks the retriever never returned, and everything gets scored with precision at k, recall at k, and MRR against a labeled set — so a change produces a number that moved, not just a feeling that it did.

2. Why You Can’t Average BM25 and Cosine Similarity

Host: So once you’ve got BM25 scores and cosine similarities sitting side by side, the obvious move is to normalize both to zero-one and average them. Why doesn’t that hold up?

Guest: Because they’re not just on different scales, they’re different kinds of quantity. BM25 is an unbounded sum of IDF-weighted term contributions, so its range shifts with query length and corpus statistics, while cosine is always bounded in negative one to one regardless of query. Whatever normalization you pick only reflects the distribution of the query you tuned it on, so the next query silently breaks it.

Host: So Reciprocal Rank Fusion is the answer specifically because it refuses to touch magnitude at all.

Guest: Right, it only looks at position — a document ranked second by both retrievers scores the same whether its BM25 score was 4 or 400. That’s a real cost, you’re throwing away information about how confident each retriever was. But what you buy is a fusion method that behaves the same way query after query, which is worth more than the precision you lose, since the averaging alternative was never consistent to begin with.

3. The Honesty of a Fake Embedding

Host: So let’s talk about the embedding side of this vector retriever, because I noticed you’re not calling out to an actual model — you’re using something called deterministic feature hashing. What is that actually doing, and what does it get you?

Guest: It exercises the vector-retrieval code path without needing API keys or ML dependencies, and it keeps fusion honest because BM25 and this hashed vector are genuinely two different ranking signals, not the same thing computed twice. But it’s not semantic similarity — it matches on shared tokens, so a query and a chunk saying the same thing in different words will score near zero, which means nothing in this lab supports a claim like ‘vector search found the paraphrase.’ That’s actually the point I want people to take away: the harness itself is the transferable artifact, so swap in a real embedding model and you should watch precision and recall visibly move, and if they don’t, you’ve just found out the problem lives somewhere else in the pipeline.

4. From Lab Loop to Production Checklist

Host: So let’s actually close the loop for people who want to try this. You spin up the app, hit slash v1 evaluate to get your baseline precision, recall, and MRR, then go change one thing — shrink the chunk size, or just disable reranking — and run it again. What should they expect to see, and why does that small loop matter more than the numbers themselves?

Guest: You’ll see the metrics move either way, but for different reasons: chunking is a retrieval decision, not a preprocessing detail, so getting it wrong degrades every downstream stage, often invisibly, while reranking works on the small fused candidate set and is supposed to clean up the top of the list. The point isn’t the specific numbers on a four-document toy corpus, it’s that you now have a mechanical way to ask ‘did that change help’ instead of eyeballing a few search results and declaring victory. And that’s exactly the habit the enterprise RAG architecture insists on at production scale too — it lists continuous evaluation on a labeled set and groundedness rate as the only real detectors for retrieval failures that are otherwise silent, because a system can serve confident, well-formatted, completely wrong answers all day with no error in your logs. This lab is the small, honest version of that discipline: the same loop, the same instinct to measure rather than believe, just without the traffic and the stakes yet.

Not covered

The planner wanted these and found nothing in the source to support them:

  • A live demonstration of swapping in a real embedding model and observing the precision/recall delta
  • A detailed walkthrough of the async-ai-gateway’s Redis atomicity fix as a parallel case study
  • Specific numeric results from running the eight-case labeled evaluation set

Generated from this page by Claude Sonnet 5 on , spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

Most RAG systems fail quietly: retrieval gets slightly worse, answers get slightly less grounded, and nothing errors. This lab builds the pipeline and the instrument that would catch that — structure-aware chunking, lexical and vector retrieval fused by rank, a reranking stage, a groundedness check, and an evaluation harness reporting precision@k, recall@k, and MRR over a labeled query set.

Source: labs/hybrid-retrieval

Retrieval pipeline

Section titled “Retrieval pipeline”

What it demonstrates

Section titled “What it demonstrates”
  • Structure-aware chunking. Chunks respect section headings and never split a paragraph unless that paragraph alone exceeds the size budget, with overlap carried across boundaries within a section. Chunking on a fixed character count is the most common way to make retrieval quietly bad.
  • Real BM25, not a keyword filter. Okapi BM25 with IDF computed over the actual corpus, so lexical ranking reflects term rarity rather than raw frequency.
  • Rank fusion, not score averaging. Reciprocal Rank Fusion combines the two rankings by position. BM25 scores and cosine similarities live on incomparable scales; averaging them directly produces a number with no meaning, dominated by whichever scale happens to be larger.
  • Reranking scoped to the fused candidate set. Only the fused candidates are rescored, which is what makes a more expensive scoring function affordable.
  • Groundedness as a retrieval property. The checker flags citations pointing at chunks the retriever never returned — catching the failure where an answer cites something plausible that was never actually retrieved.
  • Measurement, not vibes. precision@k, recall@k, and MRR over a labeled set, so a change to chunking or fusion produces a number that moved rather than an impression.

Why fuse by rank instead of averaging scores

Section titled “Why fuse by rank instead of averaging scores”

This is the design decision most worth internalizing. A BM25 score is an unbounded sum of IDF-weighted term contributions; a cosine similarity is bounded in [-1, 1]. They are not merely on different scales — they are different kinds of quantity, and no fixed normalization makes them commensurable across queries, because BM25’s range shifts with query length and corpus statistics while cosine’s does not.

Reciprocal Rank Fusion sidesteps the problem entirely by discarding magnitudes and using only position: a document ranked second by both retrievers scores the same regardless of whether its BM25 score was 4 or 400. That throws away real information, which is a genuine cost — but it buys a combination that behaves consistently across queries, which score averaging does not.

What the hashing trick does not demonstrate

Section titled “What the hashing trick does not demonstrate”

The embedding function is deterministic feature hashing. It exercises the vector-retrieval path, makes every test reproducible without API keys or ML dependencies, and keeps fusion honest because its two inputs really are different ranking signals.

It does not provide semantic similarity. Hashed features match on shared tokens, so a query and a chunk expressing the same idea in different words score near zero. No result from this lab supports a claim like “vector search retrieved the paraphrase” — it cannot.

That limitation is why the evaluation harness is the most transferable part of the lab. Swapping in a real embedding model should visibly move precision and recall on the labeled set. If it does not, the problem is somewhere else in the pipeline, and you now have the instrument to find out where.

Run it

Section titled “Run it”
Terminal window
cd labs/hybrid-retrieval
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
uvicorn retrieval.app:app --reload

The app preloads a small demo corpus (four documents, two headed sections each) and a matching eight-case labeled evaluation set.

Terminal window
# Hybrid search
curl -s -X POST localhost:8000/v1/search \
  -H 'content-type: application/json' \
  -d '{"query": "how long are logs retained", "k": 3}'


# Run the labeled evaluation set: precision@k, recall@k, MRR
curl -s -X POST localhost:8000/v1/evaluate

Change the chunk size or disable reranking, re-run /v1/evaluate, and watch the metrics move — that loop is the point of the lab.

Verify it

Section titled “Verify it”
Terminal window
pytest      # 43 tests
ruff check .
mypy src

Principal-level discussion points

Section titled “Principal-level discussion points”
  1. Chunking is a retrieval decision, not a preprocessing detail. Splitting mid-paragraph or ignoring section structure degrades every downstream stage, and does so invisibly.
  2. Rank fusion beats score fusion when the underlying scores are incommensurable — which, for lexical and vector retrieval, they always are.
  3. Reranking is affordable precisely because it runs on a small fused candidate set. A reranker applied to the whole corpus is just an expensive retriever.
  4. Groundedness checking belongs at the retrieval boundary: an answer citing a chunk that was never retrieved is a detectable failure, and detecting it does not require a model.
  5. Without a labeled set and metrics, “we improved retrieval” is unfalsifiable. The evaluation harness is what converts RAG work from tuning-by-impression into engineering.
Section titled “Related”