RAG Architecture Deep Dive

Separate the offline and online systems

RAG has two connected paths. The offline path turns source documents into searchable units. The online path turns a user question into an evidence-backed answer. Mixing their responsibilities makes failures difficult to diagnose.

Keep ownership explicit

• Source owners define authoritative records, retention, and access policy.
• The ingestion pipeline preserves structure, lineage, policy metadata, and index freshness.
• The retrieval service owns query planning, filtering, candidate recall, and ranking.
• The answer service owns context assembly, citation mapping, refusal, and response policy.
• The evaluation system measures retrieval and generation separately so one cannot hide the other's failure.

Add complexity only for a measured retrieval failure

Agentic retrieval is not a quality switch. Planning calls add latency, cost, and new failure modes. Start with the smallest pipeline that passes a representative evaluation set, then add a stage for a named failure such as low exact-term recall, missing multi-hop facts, or poor ranking.

Design chunks as retrievable evidence units

A chunk should be large enough to preserve one useful idea and small enough to rank independently. There is no universal token count or overlap percentage. Tune the strategy against labeled questions from the actual corpus.

Preserve meaning before optimizing size

• Split on document structure such as headings, paragraphs, list boundaries, tables, and code blocks before falling back to token limits.
• Store stable source and chunk IDs, document version, section path, effective time, tenant, and access-control attributes with every chunk.
• Keep the original text available for display and citation even if cleaned text is used for embedding.
• Avoid copying headers or overlap into every chunk when it causes near-duplicate candidates to crowd out distinct evidence.
• Re-index deterministically so a source update can replace its previous chunks rather than creating silent duplicates.

Use parent context without flooding the prompt

Retrieve the smallest answer-bearing passage, then attach a bounded parent heading or neighboring passage only when it resolves an ambiguous reference. Parent-child retrieval can recover structure without ranking an entire long document as one unit.

Treat retrieval as a funnel, not one score

Lexical and vector retrieval produce candidate sets; reranking only reorders the candidates it receives. A reranker cannot recover a relevant passage that initial retrieval missed.

1. 1

   Policy

   Apply hard filters

   Constrain tenant, identity, document state, language, and effective time before candidates can enter the evidence path.

2. 2

   Candidate set

   Retrieve for recall

   Use lexical search for exact identifiers, vectors for semantic similarity, or both when the workload contains both query shapes.

3. 3

   Evidence set

   Rerank for precision

   Spend the expensive relevance model on a bounded candidate pool, then remove duplicates and passages that do not add a required fact.

4. 4

   Token budget

   Assemble the prompt

   Reserve answer and instruction tokens, order evidence deliberately, and retain source-to-passage mappings for citations.

Do not confuse these signals

• Similarity estimates how closely a chunk matches a query representation; it does not prove factual correctness.
• Reranker score orders the supplied candidates; score scales are model-specific and should not be treated as universal confidence.
• Citation presence shows that a source was attached; claim-level support still needs evaluation.
• Model confidence cannot repair missing, unauthorized, or stale evidence.

Build the context as an evidence contract

The final prompt should contain only what the answer service is allowed to use. Retrieved text is external data, not trusted instructions.

Before generation

• Preserve the system and application instructions outside the retrieved-data boundary.
• Label every passage with a stable source ID and delimit retrieved content from instructions.
• Drop unauthorized, deleted, expired, malformed, or untraceable passages.
• Detect conflicts across authoritative sources and route them to a refusal or resolution policy.
• Reserve enough tokens for the answer, citations, and safety instructions instead of filling the window with retrieval output.

After generation

• Map each material claim to supporting passage IDs.
• Reject citations that do not entail the attached claim.
• Return an explicit insufficient-evidence state when required facts are absent.
• Log source IDs, index version, query plan, filters, candidate ranks, selected passages, and policy version for replay.

Evaluate the two systems separately

An end-to-end answer score is useful, but it cannot tell whether retrieval failed or generation ignored good evidence. Maintain a versioned evaluation set with questions, required facts, acceptable source IDs, access context, and time context.

Retrieval evaluation

• Recall at k: did the candidate set contain an acceptable answer-bearing source?
• Precision at k: how much of the selected evidence was actually useful?
• Fact coverage: did the final evidence packet contain every fact required by the question?
• Policy leakage rate: did any unauthorized or expired passage enter candidates, selected evidence, or logs?
• Freshness lag: how long after a source change until retrieval reflects the approved version?

Generation evaluation

• Groundedness: are material claims supported by the supplied passages?
• Answer completeness: does the response cover the required facts without inventing missing ones?
• Citation correctness: does each citation point to the passage that supports its claim?
• Refusal correctness: does the system stop when evidence is absent, conflicting, or forbidden?

Primary documentation behind the design

• Microsoft's RAG overview separates classic and agentic retrieval, explains token and security constraints, and recommends measuring retrieval choices against the workload.
• Azure AI Search chunking guidance explains why large documents must be partitioned for embedding and generation input limits.
• Azure hybrid ranking documentation describes rank fusion and keeps hybrid and semantic reranker scores conceptually separate.
• Amazon Bedrock reranking documentation defines reranking as reordering an existing candidate list rather than searching the corpus again.
• NIST's adversarial machine-learning taxonomy includes indirect prompt injection through external sources used by RAG applications.

Product defaults and APIs change. Treat these sources as implementation references and verify the current limits of the service and models you deploy.