Architecture: Enterprise RAG Platform

🎧Listen to this page16:40

Read the transcript

1. The quiet failure problem

Host: So we’re building an enterprise RAG platform, and I want to start with the thing that makes this genuinely scary rather than just technically interesting. These systems don’t fail with a stack trace. You can make a chunking change, retrieval gets a little worse, answers get a little less grounded, and nothing errors — no exception, no failed health check, no alert. The demo that worked last month and the system quietly returning wrong answers today look identical from the outside.

Guest: Right, and at organizational scale that quiet failure mode has two very specific structural causes, not just general fuzziness. First, the corpus is permissioned — if your retrieval layer doesn’t respect document ACLs, it will happily pull a chunk from something the asker isn’t allowed to see, and the model will paraphrase that content into a confident, fluent answer. Second, ingestion and query are actually two separate pipelines that have to agree — on tokenization, on the embedding model, on chunk boundaries — and there’s nothing stopping them from silently drifting apart over time. So the real design problem isn’t ‘build a RAG system,’ it’s ‘make retrieval quality something you measure and make access control something the retrieval layer enforces, instead of just hoping the application layer catches it.’

2. What the system must guarantee

Host: Okay, so if that’s the failure mode, what does the system actually have to guarantee to avoid it? Walk me through the checklist you’d hold an enterprise RAG platform to.

Guest: There are really seven things. Permission-aware retrieval, so a document the asker can’t read is never even a candidate. Pipeline symmetry, so query-time processing matches index-time processing and drift gets detected instead of discovered in production. Hybrid retrieval combining lexical and semantic signals, because each covers the other’s blind spots. Reranking over a bounded candidate set so you can afford a more expensive, more accurate scorer. Groundedness checking so an answer citing something never actually retrieved is catchable. Continuous evaluation with a labelled set and real metrics, so a change produces a number that moved. And freshness — index staleness has to be bounded and observable, not just assumed.

Host: That’s a long list, and I’d guess each one has a catch. What actually makes these hard to build rather than just check off?

Guest: Each requirement has a matching trap. Access control can’t be a post-filter — retrieving documents and then discarding the unauthorized ones leaks through result counts, latency, and any relevance score computed over the full corpus, and it wastes your retrieval budget on stuff you’re about to throw away. Hybrid retrieval sounds simple until you realize BM25 and cosine similarity produce scores that are fundamentally incommensurable, one’s an unbounded weighted sum, the other’s bounded, and their distributions shift independently per query, so there’s no fixed normalization that makes them comparable. And two of these decisions are brutally sticky: the embedding model, because re-embedding the whole corpus is a migration not a config change, and chunk boundaries, because they’re fixed at ingestion and determine forever what can ever be retrieved as a single unit.

3. Walking the request path

Host: So let’s actually walk a document and a query through this thing, start to finish. Where does a document’s life begin?

Guest: At ingestion, it gets chunked with structure awareness — respecting headings, tables, list boundaries, not just cutting every five hundred tokens — and then it’s written into two separate indexes, one lexical, one vector. When a query comes in later, it goes through that same processing, and here’s the key move: both indexes are searched within the asker’s permission scope, not after. The scoping happens at the search step itself, not as cleanup afterward.

Host: And that solves the incommensurable-scores problem you mentioned — how do the two rankings actually get combined if you can’t compare BM25 and cosine directly?

Guest: You don’t compare the scores at all — you fuse by position. Each index hands back a ranked list, and you merge those rankings by looking at where a document sits in each list, not its raw score. That fused candidate set is the only thing that goes into the reranker, which then feeds both the actual answer path and, separately, the evaluation harness watching the same results.

4. Six ways this breaks silently

Host: So let’s go through the ways this actually breaks in production, because I think people hear ‘silent failure’ as an abstraction until they see the list. Start with the first one — filtering after retrieval instead of before.

Guest: Post-filter leakage is the sneaky one because it feels safe — you retrieve from the whole corpus, then strip out anything the user isn’t authorized to see, so no restricted text ever reaches them. But the result count still reveals how many restricted documents matched, and latency correlates with how much corpus-wide work happened, so the side channel leaks even when the text doesn’t. And separately, you’ve spent your top-k budget pulling in documents that just get thrown away, so the authorized user gets quietly worse answers as more restricted content piles up. Second one is embedding-version mismatch — you swap in a new query-side embedding model without re-indexing the corpus, and now you’re comparing vectors from two different spaces. Nothing errors, relevance just collapses toward random, and it reads as ‘the model got worse’ instead of what it actually is, which is an incompatible index. Third is chunking — fixed-size chunks cut mid-sentence, mid-table, mid-list, so the chunk with the question’s keywords and the chunk with the actual answer end up split apart, and no retrieval tuning in the world fixes that because the damage happened before retrieval even ran.

Host: And that one’s especially nasty because the results still look plausible, right? It’s not obviously broken.

Guest: Exactly, it still returns neighbours that look topically right, so nobody suspects chunking. Fourth is naive score averaging — normalizing BM25 and cosine and adding them, which seems reasonable until you realize BM25 is an unbounded sum and cosine is bounded, so the blended number just gets dominated by whichever scale happens to be bigger for that particular query. It looks fine on whatever queries you tuned it on and falls apart everywhere else, which is why we fuse by rank instead. Fifth is undeleted content — a document removed at the source but still sitting in the vector index will keep getting retrieved and cited, and for a deletion that’s not a quality bug, that’s the system grounding answers in something it was legally required to forget. And the sixth one is the one that makes all the others possible: without a labelled set and real metrics, ‘retrieval got worse’ is just a feeling, it’s unfalsifiable, so every failure on this list stays invisible until someone builds the harness that can actually see it.

5. Why rank fusion beats score averaging, and other deliberate trade-offs

Host: Let’s stay on that rank fusion point for a second, because it sounds almost backwards — you’re throwing away information on purpose. Why is discarding the actual score values the right call?

Guest: Because the values were never comparable to begin with. BM25 is an unbounded sum that grows with query length and corpus statistics, cosine is squeezed into minus one to one, so a weighted average of the two is only calibrated for the exact queries you tuned it on. Reciprocal Rank Fusion just says a document ranked second is worth the same whether its BM25 score was four or four hundred, and that consistency is worth more than the precision you lose.

Host: So that’s one deliberate trade-off. Give me the others you’ve had to make peace with — chunk size, tenancy isolation, reranking cost.

Guest: Chunk size is precision versus context — small chunks retrieve cleanly but lose the surrounding meaning, large chunks keep context but dilute the embedding until the one relevant sentence drowns; structure-aware chunking just cuts where the document already cuts instead of picking an arbitrary length. Tenancy is filters versus separate indexes — filters are cheap and keep one index, but isolation becomes a property of query construction where one missing predicate is a breach, whereas separate indexes make isolation structural at a real cost multiplier. And reranking is only affordable because it’s two-stage — a cross-encoder scoring query and chunk jointly is far better than any heuristic, but it’s a model call per candidate, so you tune the candidate-set size, you don’t debate whether to rerank at all.

6. Access control as a retrieval-time concern

Host: Let’s talk about the part that keeps security people up at night — permissions. Where exactly does that check happen, and why does the order matter so much?

Guest: You filter by permission inside the index query itself, not by fetching results and discarding the ones the user shouldn’t see afterward — because the model has already read them by then, and a leaked sentence in a generated answer doesn’t announce itself the way a leaked document in a list view does. And you re-check at answer time too, not just at session start, because a long-running session can outlive a revocation — someone loses access mid-afternoon and their chat window shouldn’t still be quoting the document an hour later. On top of that, you have to treat every retrieved chunk as untrusted input, the same way you’d treat untrusted user input, because if anyone can write to the corpus, they can write instructions into a document and the model will read them with the same authoritative tone as legitimate content — that’s an injection vector wearing a lab coat.

Host: So it’s not just who can see what, it’s also not blindly trusting what comes back. What about after the fact — if something does go wrong, can you actually reconstruct what happened?

Guest: That’s what chunk-level provenance is for — every answer traces back to a specific document, a specific version, and the access decision that let it through, so an investigation isn’t archaeology. Deletion has to propagate through the whole pipeline too — source, chunks, both indexes, any cache — because a document deleted from the source but still sitting in a vector index is still fully retrievable, which defeats the point of deleting it. And you log what was retrieved per query, with redaction where needed, because “which documents grounded this answer” is always the first question anyone asks, and for genuinely per-tenant corpora you isolate at the index level rather than trusting a metadata filter, since a filter is one missing predicate away from a cross-tenant breach and a separate index simply can’t leak that way.

7. Scaling shapes and the cost of embeddings

Host: So permission scoping isn’t just an access control problem, it actually shapes how you build the index in the first place?

Guest: Right, if you’re pre-filtering by ACL, your index has to support that efficiently — metadata filters, per-tenant partitions, or fully separate indexes — and that’s a structural choice you make early and pay for forever. Switching from metadata filtering to separate indexes later isn’t a config change, it’s a rebuild. That’s also why reranking cost stays sane: it scales with candidate-set size, not corpus size, which is the entire reason a two-stage retrieve-then-rerank architecture exists instead of just reranking everything. A reranker run over the whole corpus is just a very slow retriever.

Host: And on the cost side, you’ve called out re-embedding as the single biggest expense this system will ever incur. Why does that dwarf everything else?

Guest: Because it’s not a config tweak, it’s a migration — you need dual-index operation and a real cutover, since old and new embedding spaces can’t be mixed in one index. Ingestion and query also scale independently and spikily, a bulk re-index is pure throughput with no latency pressure, query is the opposite, so sharing compute pools couples two problems that don’t belong together. And every chunk you retrieve is generation cost paid on every single turn, so pulling ten chunks when three would do isn’t a rounding error, it’s a recurring bill that compounds with traffic.

8. Instrumenting the invisible and the pre-traffic checklist

Host: So we’ve spent this whole episode cataloguing ways this can quietly break. How do you actually see it happening in production, before a user complains?

Guest: You watch six signals continuously, not as a one-time audit. Precision, recall, and MRR against a labelled set catch most of the retrieval failures we’ve discussed. Groundedness rate tells you when generation starts citing chunks it never actually retrieved. Staleness as an age distribution tells you if bounded freshness has quietly become unbounded. Permission-filter selectivity catches ACL bugs that move quality without touching retrieval code at all. Latency split by stage — lexical, vector, fusion, rerank — because a rerank regression and a vector-index regression need completely different fixes. And zero-result rate is your earliest warning of a corpus gap or a shift in what people are actually asking.

Host: So that’s the running instrumentation. What has to be true before you even let traffic hit this thing?

Guest: A checklist, basically, all provable rather than assumed. Query-side and index-side share one versioned contract, embedding model version is recorded and mismatches are refused, deletion is verified end to end across source, chunks, both indexes, and caches. Add chunk-level provenance, untrusted treatment of retrieved content in the prompt, staleness alerting, a documented dual-index migration path, and a labelled eval set running in CI with thresholds that actually fail a build — that’s the whole thing, and there’s a running lab implementing the retrieval half of it if you want to see it work rather than just hear about it.

9. The lab: running the retrieval half yourself

Host: So if someone’s listening to this and wants to actually touch it rather than just nod along, what are they going to find when they check out the lab?

Guest: It’s a real hybrid retrieval service you run locally — structure-aware chunking that respects section headings instead of chopping at a fixed character count, actual Okapi BM25 with IDF over the corpus, reciprocal rank fusion combining that with vector search, reranking scoped to just the fused candidates, and a groundedness checker that flags citations pointing at chunks the retriever never returned. You spin it up, hit a search endpoint, then hit an evaluate endpoint that gives you precision, recall, and MRR over a labeled set. Change the chunk size, disable reranking, rerun evaluate, and watch the numbers actually move — that loop is the entire point.

Host: And the one thing you don’t want people walking away thinking they’ve proven — what’s the honest caveat there?

Guest: The embedding function is deterministic feature hashing, not a real model — it matches on shared tokens, so a paraphrase scores near zero, and nothing in this lab demonstrates semantic similarity, full stop. What it does prove is that fusion and reranking and grounding work correctly as mechanisms, and it hands you an evaluation harness that’s fully transferable — swap in a real embedding model, and if precision and recall don’t move, you know the problem lives elsewhere in the pipeline. That’s the whole arc of this episode, really: stop assuming, build the instrument, let the numbers tell you where it’s actually broken.

Generated from this page by Claude Sonnet 5 on 2026-08-23, spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

Problem

RAG systems fail quietly. Retrieval gets slightly worse after a chunking change, answers get slightly less grounded, and nothing errors — no exception, no failed health check, no alert. The system that was demoed as working and the system that is now returning subtly wrong answers are indistinguishable from the outside.

Two things make this harder at organizational scale. First, the corpus is permissioned: a retrieval layer that ignores document ACLs will happily ground an answer in a document the asker may not read, and the model will paraphrase it convincingly. Second, ingestion and query are two pipelines that must agree — on tokenization, on embedding model, on chunk boundaries — and nothing structurally prevents them from drifting apart.

The architecture below treats retrieval quality as a measured property rather than an assumed one, and access control as a retrieval-time concern rather than an application-layer one.

Requirements

• Permission-aware retrieval. A document the asker cannot read is never a candidate, at any stage.
• Pipeline symmetry. Query-time processing matches index-time processing, and drift is detected rather than discovered.
• Hybrid retrieval. Lexical and semantic signals combined, since each fails where the other works.
• Reranking over a bounded candidate set, so a more expensive scorer stays affordable.
• Groundedness checking. An answer citing something never retrieved is detectable.
• Continuous evaluation. A labelled set and metrics, so a change produces a number that moved.
• Freshness. Index staleness is bounded and observable.

Constraints

• Access control cannot be a post-filter. Retrieving then discarding unauthorized documents leaks through result counts, latency, and any relevance score computed over the full corpus — and wastes the retrieval budget on documents that will be thrown away.
• The two pipelines share a contract nothing enforces. Change the embedding model on one side only and retrieval silently degrades to noise, because vectors from different models are not comparable.
• Scores from different retrievers are incommensurable. BM25 is an unbounded IDF-weighted sum; cosine similarity is bounded in [-1, 1]. Their distributions shift independently with query and corpus, so no fixed normalization makes them comparable across queries.
• Re-embedding the corpus is expensive. Changing embedding model is a migration, not a config change, which makes that choice unusually sticky.
• Chunk boundaries are a retrieval decision. They determine what can ever be retrieved as a unit, and are fixed at ingestion time.

Request Flow

flowchart TB
    D["Document"] --> CH["Structure-aware chunking<br/>respects headings, never splits a<br/>paragraph unless it alone is oversized"]
    CH --> IDX1["BM25 lexical index<br/>real IDF over the corpus"]
    CH --> IDX2["Vector index<br/>hashed-feature stand-in"]

    Q["Query"] --> L["Lexical ranking"]
    Q --> V["Vector ranking"]
    IDX1 --> L
    IDX2 --> V

    L --> RRF["Reciprocal Rank Fusion<br/>combines by POSITION, because the<br/>two score scales are incomparable"]
    V --> RRF

    RRF --> RR["Rerank<br/>rescores only the fused candidates:<br/>normalized signals + exact-phrase bonus"]
    RR --> R["Ranked results"]

    R --> G["Groundedness check<br/>flags citations pointing at chunks<br/>the retriever never returned"]
    R --> E["Evaluation harness<br/>precision@k, recall@k, MRR<br/>over a labeled query set"]

A document is chunked with structure awareness at ingestion and indexed into both a lexical index and a vector index. At query time the same processing is applied to the query, both indexes are searched within the asker’s permission scope, and the two rankings are fused by position rather than by score. The fused candidates — and only those — go to the reranker. Results feed both the answer path and, separately, the evaluation harness.

Failure Modes

✕Permission filtering applied after retrieval

Retrieving from the full corpus and filtering afterwards leaks through side channels even when no unauthorized text is returned: result counts reveal how many restricted documents matched, and latency correlates with corpus-wide work. It also silently degrades quality, because the top-k budget is spent on documents that get discarded.

✕Embedding model drift between the two pipelines

Upgrade the query-side embedding model without re-indexing and every similarity becomes meaningless — vectors from different models occupy different spaces. There is no error; relevance simply collapses toward random, which reads as “the model got worse” rather than “the index is incompatible”.

✕Chunking that splits the answer

Fixed-size chunking cuts mid-sentence, mid-table, and mid-list. The chunk containing the question’s keywords and the chunk containing its answer become different chunks, and no amount of retrieval tuning recovers what chunking destroyed. It is invisible because retrieval still returns plausible-looking neighbours.

✕Score averaging across incommensurable retrievers

Normalizing BM25 and cosine scores and averaging them produces a number dominated by whichever scale happens to be larger for that query. It appears to work on the queries used to tune it and behaves unpredictably elsewhere, because the two distributions shift independently.

✕Stale index presented as current

A document updated or deleted at the source but still indexed will be retrieved and cited. For a deletion this is a compliance problem, not a quality one — the system is grounding answers in content it was required to forget.

✕Quality regression with no detector

Without a labelled set and metrics, “retrieval got worse” is unfalsifiable and therefore undetectable. Every other failure mode on this list is silent by default; the evaluation harness is the only thing that makes any of them visible.

Scaling

• Permission-scoped retrieval constrains the index design. Pre-filtering by ACL requires the index to support it efficiently — metadata filters, per-tenant partitions, or separate indexes — and this is a decision that is expensive to change later.
• Reranking cost is bounded by candidate-set size, not corpus size, which is the entire reason a two-stage architecture exists. A reranker over the whole corpus is just a slow retriever.
• Ingestion and query scale independently and spikily. A bulk re-index is a throughput problem with no latency requirement; query is the opposite. Sharing a pool couples them badly.
• Re-embedding is a migration, needing dual-index operation and a cutover, because the two vector spaces cannot be mixed.
• Lexical and vector indexes have different growth profiles — inverted index size tracks vocabulary, vector index tracks chunk count times dimensionality — and they hit limits at different times.

Security

• Filter by permission at retrieval time, in the index query, never after.
• Re-check permissions at answer time, since a long-running session can outlive a revocation.
• Treat retrieved content as untrusted input. Documents can contain instructions; grounding a model in a corpus that anyone can write to is an injection vector with an authoritative tone.
• Store chunk-level provenance, so any answer can be traced to a document, a version, and an access decision.
• Honour deletion through the whole pipeline — source, chunks, both indexes, and any cache. A deleted document still in a vector index is still retrievable.
• Log what was retrieved per query, subject to redaction, since “which documents grounded this answer” is the first question of any investigation.
• Isolate tenants at the index level where the corpus is genuinely per-tenant; a metadata filter is one bug away from a cross-tenant leak, where separate indexes are not.

Trade-offs

⇄Rank fusion vs. score fusion

Reciprocal Rank Fusion discards magnitude and uses only position, which throws away real information — a document ranked first by a wide margin scores the same as one that barely won. In exchange it behaves consistently across queries, which normalized score averaging does not, because BM25 and cosine distributions shift independently with query length and corpus statistics. Consistency wins here because the alternative is only valid for the queries it was tuned on.

⇄Chunk size

Small chunks retrieve precisely and lose the context needed to interpret what was retrieved. Large chunks carry context and dilute the embedding, so a single relevant sentence is drowned by surrounding text. Structure-aware chunking sidesteps the worst of the trade by cutting where the document already cuts, rather than at an arbitrary length.

⇄Metadata filtering vs. per-tenant indexes

Filters keep one index, are cheap to operate, and make cross-tenant isolation a property of query construction — one missing predicate is a breach. Separate indexes make isolation structural and multiply operational cost and per-tenant capacity floor. The deciding question is whether a cross-tenant leak is survivable.

⇄Cross-encoder reranking vs. heuristic reranking

A cross-encoder scores query and chunk jointly and is substantially better than any heuristic. It also costs a model call per candidate, adding latency proportional to candidate-set size. The two-stage design exists to make that affordable; the tuning question is candidate-set size, not whether to rerank.

Cost

• Embedding cost is paid twice: once per chunk at ingestion, once per query. Ingestion dominates initially; query cost dominates at steady state and scales with traffic.
• Re-embedding the corpus is the largest single cost event in this architecture’s lifetime, which is why the embedding model choice deserves more scrutiny than its config-line appearance suggests.
• Reranking cost scales with candidate-set size, making that number the primary quality-per-dollar dial.
• Retrieved context is generation cost. Every chunk passed to the model is tokens paid for on every turn, so retrieving ten chunks where three suffice is a recurring bill, not a one-off.
• Storing both indexes roughly doubles index storage, which is the price of hybrid retrieval.

Observability

• Retrieval metrics on a labelled set — precision@k, recall@k, MRR — run continuously, not once. This is the only detector for most failures above.
• Groundedness rate: the proportion of answers whose citations point at chunks actually retrieved. A drop means the generation step is inventing sources.
• Index staleness, as the age distribution of indexed content against the source. Bounded staleness is a design choice; unbounded staleness is a defect.
• Permission-filter selectivity per query. A sudden change means an ACL bug or a permission model change, and it moves quality without touching retrieval code.
• Latency split by stage — lexical, vector, fusion, rerank — since a rerank regression and a vector-index regression need entirely different responses.
• Zero-result and low-confidence rate, the earliest signal of a corpus gap or a query distribution shift.

Production Deployment

☑Before real traffic

• Permission filtering happens inside the index query, and a test proves an unauthorized document is never a candidate.
• Query-side and index-side processing share one code path or one versioned contract, and a mismatch fails loudly.
• The embedding model and its version are recorded in the index, and a mismatch is refused rather than silently served.
• A labelled evaluation set exists and runs in CI, with thresholds that fail a regression.
• Deletion propagates to source, chunks, both indexes, and caches, verified end to end.
• Chunk-level provenance is stored, so any answer traces to document, version, and access decision.
• Retrieved content is treated as untrusted in the prompt path.
• Index staleness is measured and alerted against a stated bound.
• Re-embedding has a documented dual-index migration path.

⚙Hands-on Lab

A running implementation of the retrieval half: structure-aware chunking, BM25 and vector retrieval fused by rank, reranking over the fused set, groundedness checking, and an evaluation harness reporting precision@k, recall@k, and MRR. The embedding function is a deterministic hashing stand-in with no semantic similarity, which the lab is explicit about. Read the lab documentation →

labs/hybrid-retrievalproduction-shaped

Interview Questions

?Why must permission filtering happen during retrieval rather than after it?

Post-filtering leaks through side channels even when no unauthorized text is returned — result counts reveal how many restricted documents matched, and latency correlates with corpus-wide work. It also degrades quality, because the top-k budget is spent on documents that are then discarded, so an authorized user gets worse results the more restricted content exists.

?Why fuse lexical and vector results by rank instead of normalizing and averaging their scores?

BM25 is an unbounded sum of IDF-weighted contributions; cosine similarity is bounded in [-1, 1]. They are different kinds of quantity, and their distributions shift independently with query length and corpus statistics, so any fixed normalization is only valid for the query it was fitted on. Rank fusion discards magnitude — a real loss — in exchange for behaviour that is consistent across queries.

?You upgraded the embedding model on the query side only. What happens?

Relevance collapses toward random, with no error anywhere. Vectors from different models occupy different spaces, so similarity between them is meaningless. It reads as a quality regression rather than an incompatibility, which is why the index should record its embedding model and version and refuse a mismatch.

?How do you detect that retrieval quality has regressed?

A labelled evaluation set with precision@k, recall@k, and MRR, run continuously with thresholds that fail a build. Without it, every failure in this architecture is silent — nothing throws, nothing alerts, and the only signal is users gradually trusting answers less. Groundedness rate and zero-result rate are useful secondary detectors that need no labels.

?A document is deleted for compliance reasons. What has to happen?

Deletion has to propagate to the source, the chunk store, the lexical index, the vector index, and any cache — and be verified end to end. A deleted document still present in a vector index is still retrievable and will still be cited, which converts a quality concern into a compliance breach. Chunk-level provenance is what makes the deletion traceable.

?Why rerank a small candidate set instead of the whole corpus?

Because a cross-encoder scores each query-chunk pair jointly, so its cost is linear in candidates. Running it corpus-wide is just an expensive retriever. The two-stage design exists to make a better scorer affordable by only paying for it where cheap retrieval already indicated relevance.