Design a RAG System
Design a RAG system like Perplexity: vector databases, embedding models, and LLM integration.
What is a RAG system?
A retrieval-augmented generation (RAG) system searches an approved knowledge collection before a language model writes an answer. Retrieval gives the model current, private, or domain-specific evidence that is not reliably available in its training data.
For this interview, design an enterprise assistant over 10 million documents for 100,000 daily users and 1 million queries per day. The answer path should stay under 3 seconds at p95, retrieval recall should exceed 85%, and every factual answer must be grounded in evidence the caller is allowed to access.
Core mechanism
Retrieve before writing
Use sparse and semantic search to find candidate passages, then rerank the best evidence for the question.
Security boundary
Authorization is an invariant
Filter by tenant and document permissions before retrieved text reaches ranking, prompts, caches, or logs.
Degraded mode
Abstention is a valid answer
When evidence is missing, conflicting, private, or unsafe, explain the gap instead of substituting model confidence.
Define the answer contract
The product accepts a natural-language question and returns either a cited answer or an explicit statement that the available evidence is insufficient. The evidence set is constrained by the caller's identity, tenant, role, region, and document permissions.
Functional requirements
- Ingest text documents, PDFs, web pages, code, and structured tables.
- Search by exact terms and semantic meaning across permitted content.
- Generate concise answers with claim-level source citations.
- Support multi-turn questions without allowing conversation history to bypass access policy.
- Reflect approved document changes in search within ten minutes.
Non-functional requirements
- Serve 1 million daily queries and design for an explicit 2,000 QPS peak scenario.
- Keep end-to-end p95 response latency under three seconds.
- Achieve Recall@10 above 85% and grounded-answer faithfulness above 90%.
- Isolate tenants, detect sensitive data, preserve audit trails, and honor data residency.
- Degrade to search results or abstention when generation or validation is unhealthy.
Ask questions that change the architecture
- Who is the caller? Employees, customers, and external partners imply different identity and policy systems.
- Which sources are authoritative? A policy portal may outrank an old wiki even when the wiki is more textually similar.
- What is the cost of a wrong answer? High-consequence domains require stronger support thresholds and human escalation.
- How fresh must evidence be? A ten-minute indexing target needs an incremental ingestion path, not nightly rebuilds.
- Which requests are out of scope? Open-ended creative writing and unsupported advice should not be disguised as grounded lookup.
State the invariant first. A model must never see or cite a passage the caller cannot access. Post-generation redaction is too late because private evidence has already crossed the model boundary.
RAG token & cost budget
Each query retrieves the top 5 chunks (~500 tokens each), plus a ~500-token system prompt, a ~100-token question, and a ~500-token answer. LLM cost is $5 per 1M tokens. Estimate the budget.
Retrieval strategy for the RAG system
Your RAG system answers questions where queries mix natural-language intent ("how do I reset a locked account") with exact tokens like product codes and proper names ("error E-4471", "Dr. Okafor"). Which retrieval approach should back the index?
Constraint: Answers must stay relevant for BOTH semantic paraphrase queries AND exact-keyword lookups (names, codes), within a tight latency budget.