Google Search Crawler Architecture
Build scalable web crawlers: distributed architecture, rate limiting, politeness policies, and data processing.
What is a distributed web crawler?
A distributed web crawler is a fleet of programs that discovers web addresses, decides which address is safe and useful to visit next, downloads a response, and uses any accepted links to continue the loop. Search engines use crawlers to build the collection of documents that an index can later search.
In plain language, a crawler is a careful map maker. Its central invariant is: a URL may be fetched only after it is normalized, deduplicated, allowed by the host's policy, and eligible under that host's crawl budget. More workers can increase global throughput, but they must never turn one busy or failing website into a victim of the crawl.
This case study designs a crawl system for one billion pages each month. You will move from the crawl contract and capacity math to the frontier state machine, polite failure paths, operations, and the trade-offs behind a responsible crawler.
Define what the crawler is allowed to do
A crawler does not own the sites it visits. The first design decision is therefore a behavioral contract, not a worker count: discover useful public documents while placing an explicit, bounded load on each remote host.
Functional scope
- Start from supplied seed URLs and discover links from accepted HTML documents.
- Fetch HTML and selected documents such as PDFs; treat rendering-heavy pages as an explicit later capability.
- Normalize URLs, record redirects, suppress repeat work, and schedule refreshes.
- Read and cache
robots.txt; honor applicable allow/disallow rules and configured crawl delays.
Non-functional guardrails
- Scale: crawl one billion pages per month, with headroom for daily spikes.
- Politeness: a host's rules and health always override a URL's global priority.
- Safety: bound redirects, response size, URL length, traversal depth, parser time, and retries.
- Recoverability: isolate a bad host or failed worker so unrelated hosts continue making progress.
robots.txt is a crawl-policy signal, not an authorization system. A production crawler still needs a recognizable User-Agent, a contact channel, retention policy, and legal review appropriate to its data use.