RAG Systems Foundation

Separate the knowledge loop from the answer loop

A RAG system contains two connected loops. The knowledge loop turns source material into a reproducible, searchable snapshot. The answer loop uses one snapshot to retrieve, assemble, generate, verify, and either serve or contain a response.

1. 1

   Offline

   Ingest sources

   Discover authorized documents, parse structure, normalize text, and attach source, revision, owner, access, and freshness metadata.

2. 2

   Offline

   Publish an index

   Chunk content, compute versioned representations, write searchable records, verify counts, then atomically publish one snapshot.

3. 3

   Online

   Retrieve candidates

   Authorize the request before search, encode or rewrite the query, and collect lexical, dense, or structured candidates.

4. 4

   Online

   Rerank evidence

   Deduplicate, apply metadata and safety filters, and spend a bounded second-stage budget on the strongest candidates.

5. 5

   Online

   Assemble and answer

   Fit source-labeled evidence into the prompt budget and require supported claims, claim-linked citations, and abstention.

6. 6

   Online

   Verify and contain

   Check policy, support, freshness, citations, latency, and cost before serving; otherwise abstain, degrade, or route to review.

Preserve identity across both loops

• Source identity: document ID, revision, location, owner, tenant, access policy, effective time, and deletion state.
• Transformation identity: parser, normalizer, tokenizer, chunker, embedding model, dimensions, normalization, and schema versions.
• Index identity: snapshot or generation, distance metric, filter schema, build time, validation result, and rollback target.
• Request identity: query and rewrite versions, authorized principal, candidate and rerank settings, prompt version, model version, and response policy.
• Evidence identity: chunk IDs, source spans, scores by stage, filter decisions, packed order, and citations actually shown to the user.

Build ingestion as a reproducible data product

Ingestion is the pipeline that converts source-of-truth content into searchable records. It is not a one-time file upload. Production ingestion must handle retries, revisions, deletions, partial failures, access changes, and a delayed index without creating duplicate or stale evidence.

Choose chunk boundaries from answer shape

A chunk is the smallest independently indexed evidence unit. Chunk size is a retrieval decision, not a universal constant.

• Small chunks can improve focus and reduce irrelevant prompt tokens.
  • They can split a definition from its exception, a table header from its row, or a procedure step from its prerequisite.
• Large chunks preserve more local structure and continuity.
  • They can dilute similarity, repeat irrelevant text, and consume context that could hold other evidence.
• Overlap protects facts near boundaries.
  • It also repeats embedding work and increases index size, refresh time, and duplicate candidates.
• Structure-aware chunking should preserve semantic units such as headings, clauses, code symbols, table rows, and speaker turns before applying token limits.

For fixed token chunks, start with stride = chunk_tokens - overlap_tokens. Measure the resulting answer-span coverage, duplicate-token ratio, retrieval recall, and prompt precision on representative queries. Do not select a chunk size from one average document.

Bind embeddings and indexes into one contract

An embedding maps an input into a numeric vector. A vector index can compare vectors quickly, but the numbers are meaningful only under the exact representation contract that produced them.

Reject incompatible writes and queries

• Fail closed when vector dimensions, normalization, distance metric, or contract fingerprints differ.
• Never encode documents with one model and queries with another unless that asymmetric pairing was explicitly trained and evaluated.
• Re-embed into a new index when the model or preprocessing contract changes; do not silently mix versions.
• Publish a source checkpoint with the index so operators can answer which revision was searchable for a request.
• Test deletion propagation through source storage, queues, index replicas, caches, and prompt assembly.

Retrieve candidates, then rank evidence

Retrieval finds a broad candidate set. Reranking spends more computation to order a smaller set for the actual question. Keeping these stages separate lets the first stage protect recall while the second stage improves prompt focus.

• Sparse retrieval uses lexical signals. It is valuable for names, exact identifiers, quoted phrases, and rare terms.
• Dense retrieval uses embedding similarity. It is valuable for paraphrases and semantically related language.
• Hybrid retrieval fuses sparse and dense rankings so one representation does not erase the other's strengths.
• Metadata filtering enforces authorization, locale, time, product, and document-state constraints inside the retrieval boundary.
• Reranking scores query-chunk pairs with richer features or a model, but its candidate depth creates a direct latency and cost budget.

Use labeled production-shaped queries to measure recall at the first-stage candidate depth. Then measure precision and answer support after filtering, deduplication, and reranking. The heterogeneous BEIR benchmark showed that retrieval methods behave differently across tasks, so an average from another corpus is not a local release argument (Thakur et al., 2021).

Read the workbench in causal order

1. Check whether chunks cover the answer span before increasing search effort.
2. Match lexical and semantic retrieval to the query distribution, including exact identifiers.
3. Increase candidate depth only while measured recall gains justify added tail latency.
4. Rerank a bounded pool and verify that prompt precision improves without dropping required evidence.
5. Treat the synthetic model as a planning aid; the release decision must use labeled queries and measured infrastructure.

Assemble a prompt that preserves evidence boundaries

Prompt assembly selects and formats the evidence that the generator may use. Grounding means the answer's factual claims are supported by that evidence. A citation is the user-visible link from a claim to the supporting source span.

Start with a budget, not with every retrieved chunk:

evidence_budget = context_limit - system_policy - conversation - query - output_reserve - safety_margin

Then apply a deterministic assembly policy:

1. Keep only authorized, current, non-quarantined chunks from the declared index snapshot.
2. Deduplicate near-identical text and resolve superseded or conflicting revisions.
3. Reserve evidence for distinct sub-questions instead of letting one long source consume the budget.
4. Order and delimit evidence as untrusted data, preserving chunk ID, source URI, revision, and location.
5. Require claim-linked citations and an explicit abstention response when support is missing or contradictory.
6. Verify the generated structure, citations, and high-risk claims before display or downstream use.

Make the answer earn permission to serve

An answer gate is a deterministic policy that evaluates evidence and operational conditions before a generated response reaches the user. Generation success is not permission to serve.

Contain each failure deliberately

• No relevant evidence: abstain, ask a clarifying question, or route to a bounded specialist source.
• Stale evidence: refresh or select a known-current snapshot; do not relax the freshness gate to make the answer pass.
• Conflicting revisions: identify the authoritative owner or route to review; do not ask the generator to guess.
• Suspicious retrieved instructions: quarantine the chunk, preserve the trace, and prevent tool or secret access.
• Retriever timeout: use an explicit no-evidence response or a separately evaluated degraded path, never an invisible parametric-only fallback.
• Model or verifier timeout: stop within the request deadline and return a bounded failure; do not retry until the latency budget collapses.
• Budget pressure: reduce candidate or output work according to a tested policy, or reject early with a retry signal.

Evaluate the stages before evaluating the whole answer

RAG evaluation measures retrieval, evidence use, answer behavior, and operations separately, then checks the end-to-end user task. One blended score cannot identify the owner of a failure.

Build a stage-attributed evaluation set

• Retrieval labels: relevant chunk IDs or source spans, hard negatives, exact identifiers, paraphrases, and multi-hop evidence.
• Generation labels: expected claims, acceptable variants, required citations, prohibited claims, and abstention behavior.
• Safety labels: unauthorized sources, indirect prompt injections, poisoned content, secrets, and policy-sensitive requests.
• Freshness labels: updates, superseded revisions, deletions, delayed connectors, and index cutover windows.
• Operational labels: corpus size, candidate depth, cache state, concurrency, dependency failure, and latency or cost ceiling.

The RAGAS research separates context relevance, answer faithfulness, and answer relevance, illustrating why retrieval and generation need distinct evidence (Es et al., 2023). Automated evaluators can accelerate iteration, but calibrate them against qualified human judgment and keep deterministic checks for exact properties.

Diagnose by metric pattern

• Low retrieval recall with good groundedness usually means the generator behaves when evidence arrives, but the retriever misses too often.
• High recall with low prompt precision means assembly or reranking admits too much distracting evidence.
• High context quality with unsupported claims points to prompt, generation, citation, or verification behavior.
• Good averages with a failed tenant, language, freshness, or attack slice means the release is still blocked.
• Good offline quality with production latency or cost regressions means the operating point, not necessarily the model, must change.

Budget latency and cost by stage

A service budget assigns a measurable ceiling to each stage while preserving the end-to-end objective. For a sequential request, begin with:

p95_total ~= p95_queue + p95_query + p95_search + p95_rerank + p95_assembly + p95_generation + p95_verification

This sum is a planning approximation; shared queues, parallel calls, retries, and correlated tails require trace measurements and load tests. Cost should likewise include ingestion and online work:

monthly_cost = source_processing + embedding_updates + index_storage + retrieval + reranking + model_input + model_output + evaluation + observability

Control the dominant terms

• Re-embed only changed chunks and cache immutable transformation results by contract fingerprint.
• Cap candidate and rerank depth from measured recall curves rather than a fixed generous top-k.
• Remove duplicate and low-value evidence before model input; prompt tokens affect both latency and cost.
• Route simple, low-risk requests to a smaller evaluated model only when the same grounding contract still passes.
• Apply admission control, bounded retries, deadlines, and backpressure before queues consume the entire tail budget.
• Track cost per answered request, per grounded answer, and per abstention; a cheap unsupported answer is not an efficiency win.

Observe one request from query to decision

RAG observability connects one request to the exact data, versions, stage timings, decisions, and user-visible outcome that produced it. Logs without this lineage cannot distinguish a model regression from an index cutover, stale connector, filter bug, or prompt change.

Trace at least these spans or equivalent events:

1. authorize: principal, tenant, policy version, allowed corpus, and decision without secret-bearing payloads.
2. encode_query: representation contract, dimensions, cache state, duration, and error.
3. retrieve: index snapshot, mode, filters, candidate count, chunk IDs, stage scores, and duration.
4. rerank: reranker version, input and output counts, score distribution, and duration.
5. assemble: prompt version, evidence IDs, token allocation, exclusions, conflicts, and quarantine decisions.
6. generate_and_verify: model version, input and output token counts, citations, groundedness decision, finish reason, and latency.
7. serve_or_contain: answer, abstain, degrade, review, or error outcome plus the gate and policy version that chose it.

OpenTelemetry's current GenAI registry includes retrieval document IDs and scores while warning that query and message attributes can contain sensitive information; the GenAI conventions are also moving between repositories and include development-status fields (OpenTelemetry GenAI attributes). Treat that evolving standard as input, pin your own telemetry contract, and default to IDs, counts, hashes, and sampled redacted content.

Alert on user impact and broken contracts

• Retrieval empty rate, recall proxy shifts, score-distribution drift, filter denials, and reranker drop-off by slice
• Source-to-index lag, failed or stuck checkpoints, stale-answer rate, deletion objective breaches, and snapshot age
• Grounded claim rate, citation coverage and validity, abstention rate, conflict rate, and safety gate failures
• Queue delay, stage p95 and p99, timeout and retry rate, context tokens, output tokens, and cost per outcome
• Release identity, canary cohort, error-budget burn, fallback use, and rollback success

Carry the foundation into deeper design

The foundation is one end-to-end evidence contract: reproducible ingestion, compatible representations, authorized retrieval, bounded reranking, evidence-aware prompting, claim-linked citations, stage-level evaluation, and tested containment.

Continue with RAG architecture deep dive to study more advanced ranking and context strategies, then vector database implementation for index internals and operational consistency.

Primary sources used in this lesson

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• RAGAS: Automated Evaluation of Retrieval Augmented Generation
• OWASP LLM01:2025 Prompt Injection
• NIST AI 600-1: Generative Artificial Intelligence Profile
• OpenTelemetry GenAI semantic convention registry