Advanced RAG Patterns

Start from the failure, not the pattern name

Advanced RAG is not a maturity ladder where every system should eventually become adaptive or agentic. Begin with a reproducible single-query baseline, slice its failures, and add the smallest mechanism that addresses one persistent gap.

Name the baseline before changing it

• Record the query, identity scope, index version, filters, candidate IDs, and selected evidence.
• Label the expected source IDs or required facts for a representative evaluation set.
• Measure retrieval recall and evidence coverage separately from answer quality.
• Slice results by query shape; a global average can hide a broken low-volume route.
• Preserve a simple branch as a control so additional calls must prove their value.

Understand what each pattern actually changes

1. 1

   Query

   Represent the intent

   Keep the original question, then create only the bounded decomposition, reformulation, or hypothetical representation the selected pattern requires.

2. 2

   Recall

   Retrieve independent lists

   Run policy-filtered lexical, dense, summary-level, or leaf-level searches. Each list has its own ranking contract.

3. 3

   Precision

   Fuse and select evidence

   Merge ranks, deduplicate lineage, attach bounded parent context, and reserve room for independently useful sources.

4. 4

   Control

   Release, retry once, or stop

   Check required facts, authority, freshness, and contradictions. A typed abstention is safer than an unbounded search loop.

Four useful pattern families

1. Multi-query fusion creates a small set of query variants or subqueries, retrieves for each, and combines the resulting ranks.
2. Hierarchical retrieval first selects a document or section, then retrieves detailed passages inside the selected parents.
3. HyDE generates a hypothetical answer-shaped passage as a search representation, embeds it, and retrieves real corpus documents near that representation.
4. Adaptive retrieval classifies the request, chooses a bounded branch, evaluates the evidence contract, and either runs one defined fallback or abstains.

These families can be composed, but every composition multiplies the trace, evaluation, and failure states an operator must understand.

Preserve document hierarchy without flooding context

Long documents contain relationships that flat chunks erase: a paragraph belongs to a section, an exception modifies a rule, and a definition may govern many later clauses. Hierarchical retrieval makes those relationships queryable.

Keep hierarchy observable

• Store stable document, section, and leaf IDs with explicit parent links.
• Version summaries with the source snapshot and treat generated summaries as derived data, not source evidence.
• Measure parent recall, leaf recall, and final evidence coverage separately.
• Permit a controlled broad-search fallback when parent routing confidence is low.
• Cite the real source span, not only the generated summary that helped route retrieval.

Tree-organized methods such as RAPTOR show one way to retrieve across levels of abstraction. They are design evidence, not a universal instruction to summarize every corpus into a tree.

Use generated representations as search hints, never as evidence

HyDE and contextual retrieval both change the text used for retrieval, but they make different trust promises.

Use either technique only after evaluation demonstrates a persistent representation gap. Generated context can improve matching while also introducing model cost, nondeterminism, stale derived text, and a new artifact that must be versioned.

Bound adaptive retrieval as a state machine

An adaptive pipeline should not be an open-ended agent that keeps searching until a model feels confident. Define routes, budgets, evidence requirements, and terminal outcomes in application code.

1. 1

   Route

   Classify the query

   Choose from a closed set such as exact lookup, hybrid search, multi-query fusion, hierarchy, or HyDE. Record the reason and router version.

2. 2

   Budget

   Run one branch

   Apply authorization first, then enforce maximum query variants, candidate counts, retrieval calls, and elapsed time.

3. 3

   Gate

   Evaluate the evidence

   Check required-fact coverage, source authority, freshness, contradictions, and citation spans without asking generation to repair missing evidence.

4. 4

   Outcome

   Terminate deliberately

   Answer, ask a clarifying question, run one named fallback, or return a typed insufficient-evidence result.

Evaluate the extra stage that you added

An end-to-end answer score cannot tell whether a new pattern fixed retrieval or merely changed the model's prose. Keep a frozen baseline and compare the advanced branch on the query slice it was designed to improve.

Retrieval evidence

• Candidate recall: did any active list retrieve every acceptable source or required fact?
• Parent recall: did hierarchical routing keep the correct document and section reachable?
• Fusion contribution: which ranked list promoted each selected document?
• Evidence coverage: does the final packet contain every fact needed to answer?
• Source diversity: did duplicate chunks displace an independently useful source?

Operational evidence

• Search executions, model calls, reranker pairs, input and output tokens, and elapsed time by stage.
• Router distribution, fallback rate, abstention reasons, and budget-exhaustion rate by query class.
• Index, embedding, query-rewriter, router, reranker, generator, and policy versions.
• Authorization failures, stale-source selections, contradictory evidence, and citation mismatches.

Release discipline

1. Replay the same labeled queries through the simple and advanced branches.
2. Compare the target slice and guardrail slices, not only an overall average.
3. Inspect regressions caused by off-topic rewrites, missed parents, or router mistakes.
4. Canary the new route with kill switches and a known-good fallback.
5. Remove the stage when it adds work without durable, measured benefit.

Operate advanced RAG for replay, rollback, and refusal

• Log the original query fingerprint, generated variants, route decision, filters, ranked lists, fusion contributions, selected evidence, and gate verdict.
• Keep derived summaries, contextualized chunks, and hypothetical representations versioned and linked to their source snapshot.
• Bound fan-out, candidate depth, reranking, model calls, retries, context tokens, and total request time.
• Define degradation per route: skip an optional reranker, fall back to a simple retriever, or return search results without generation.
• Test prompt injection, poisoned summaries, cross-tenant retrieval, stale parents, duplicate evidence, and contradictory sources.
• Return the missing evidence class to the caller when the release gate fails; do not hide abstention behind a generic model error.

Primary sources behind the patterns

• RAG-Fusion describes multi-query generation followed by reciprocal rank fusion and also reports that weak query variants can drift off topic.
• Azure AI Search RRF documentation documents rank-based fusion as 1 / (rank + k) and keeps fused scores separate from semantic reranker scores.
• HyDE defines the hypothetical-document retrieval pivot and explicitly treats the generated document as unreal rather than evidence.
• RAPTOR presents tree-organized retrieval across different levels of abstraction.
• Self-RAG studies adaptive retrieval and reflection rather than indiscriminately retrieving a fixed number of passages.
• Anthropic's contextual retrieval write-up describes adding chunk-specific context to both embedding and lexical representations.
• Lost in the Middle shows why merely supplying a longer context does not guarantee robust use of relevant evidence.

Research results are workload-specific. Reproduce the relevant comparison on your corpus, identities, query distribution, and release policy before adopting a pattern.