Design Search Engine
Design a search engine: crawling, indexing, ranking, and serving search results at scale.
What is a web search engine?
A web search engine discovers public documents, builds a searchable representation of them, and returns a ranked result list for a user's query. Discovery and indexing happen asynchronously. Query serving reads a published index snapshot under a strict latency deadline.
The core invariant is stronger than "return relevant links": every served result must come from a valid index snapshot and pass current eligibility policy. A stale result may be tolerable during a failure. A removed, malicious, or access-restricted result is not.
This walkthrough uses an illustrative target of 10 billion canonical text documents and 4 billion queries per day. Those are design assumptions for practicing arithmetic, not claims about any existing search provider.
Crawl
Discover and revisit
Maintain a prioritized URL frontier, respect host capacity and crawl policy, fetch documents, and decide when each URL deserves another visit.
Index
Build searchable evidence
Parse and canonicalize content, detect duplicates, build compressed posting lists, and publish immutable index segments through a versioned manifest.
Serve
Retrieve, rank, and filter
Retrieve a broad candidate set, score it with lexical and quality signals, enforce hard policy, generate snippets, and return within the request deadline.
New to indexes or capacity math? Review database indexing, caching, and back-of-the-envelope estimation before the deeper sections.
Turn "search the web" into a bounded contract
Requirements are the user-visible actions and guarantees that the architecture must support. Clarify them before choosing shards, queues, models, or vendors because one change in freshness, corpus scope, or policy can reshape the whole system.
Functional
Core user actions
- Submit keyword, phrase, and filtered text queries.
- Receive ten ranked links with titles, URLs, and generated snippets.
- Correct obvious spelling errors and offer query suggestions.
- Discover HTML and PDF documents from seeds, links, and submitted sitemaps.
Non-functional
Service promises
- Query-serving p95 below 200 ms in supported regions.
- Query-serving availability of 99.99%; indexing availability of 99.9%.
- Publish high-priority updates within five minutes and most accepted changes within 24 hours.
- Keep partial-result and policy-violation rates explicitly measurable.
Invariant
Correctness and safety
- Never serve a document that current policy marks ineligible.
- Do not mix postings, metadata, and ranking features from incompatible index versions.
- Make deletions and legal or security removals override cached result pages.
- Treat fetched documents, links, and structured metadata as untrusted input.
Out of scope
First-version boundary
- No image, video, shopping, advertisement, or generated-answer vertical.
- No per-user behavioral personalization.
- No guarantee that every discovered URL is crawled or indexed.
- No requirement to synchronously rebuild the full index after each document change.
Questions that change the design
- Corpus: Is this a public-web engine, a fixed enterprise collection, or a vertical search product? Which languages and document formats are in scope?
- Freshness: Which sources need minute-level updates, and which can be revisited over days? Is freshness measured from publication, discovery, fetch, or index visibility?
- Quality: Which query intents matter most, and who provides graded relevance judgments for offline evaluation?
- Policy: Which crawler rules, removal requests, malware decisions, and regional restrictions can exclude a document?
- Degradation: May the service return partial shard coverage, an older complete snapshot, cached head-query results, or an explicit error?
- Economics: What are the storage, network, and ranking-compute budgets per query and per accepted document?
robots.txt is a crawler coordination protocol, not authorization. RFC 9309 defines how compliant crawlers interpret the Robots Exclusion Protocol; private content still needs real access control.