Web Crawler System Design Interview: Crawling the Web at Scale
The indexed web runs into the hundreds of billions of pages, and the full web including uncrawled and dynamic pages is far larger. Googlebot is reported to crawl tens of billions of pages, and a serious crawler fleet needs to fetch thousands of pages per second sustained to cover a large corpus in a reasonable window. Common Crawl publishes monthly snapshots of a few billion pages each. Exact live numbers are not published by search engines, so treat per-second and per-fleet figures as estimates.
A web crawler starts from a set of seed URLs, fetches each page, extracts the links inside it, and enqueues the new URLs to fetch later. The hard part is not fetching one page, it is doing this across billions of pages while staying polite to each server, never crawling the same page twice, and keeping the crawled copy fresh. The heart of the design is the URL frontier, a large distributed queue that decides what to fetch next and enforces per-domain rate limits. Around it sit a DNS resolver with a cache, a pool of fetcher workers, a parser that extracts links and content, a dedup layer built from a URL bloom filter and content hashing, and a content store for the raw pages. Get the frontier and the politeness right and the rest of the system follows.
Asked at: Asked at Google, Amazon, Microsoft, and most companies that run search, price monitoring, or large-scale data collection.
Why this question is asked
Interviewers like the web crawler because it looks simple and then explodes into hard problems the moment you push on scale. Fetching one page is trivial, but fetching billions forces you to reason about the URL frontier that orders and schedules work, about politeness so you do not hammer a single site into an outage, about duplicate detection so you do not waste the fleet re-fetching the same content, about freshness so the crawled copy does not go stale, and about crawler traps and infinite URL spaces that can trap a naive crawler forever. It also touches queues, sharding, bloom filters, DNS, and back pressure in one problem, so it is a good test of whether a candidate can hold a whole distributed system in their head.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Start crawling from a configurable set of seed URLs.
- Fetch the HTML content of each URL over HTTP and HTTPS.
- Parse fetched pages and extract outbound links.
- Normalize and enqueue newly discovered URLs into the frontier.
- Respect robots.txt and crawl-delay directives for each host.
- Avoid re-crawling URLs and pages already seen, including near-duplicate content.
- Re-crawl pages on a schedule so the stored copy stays fresh.
- Store the raw page content and metadata for downstream indexing.
- Support filtering so the crawler stays within an allowed scope, for example specific domains or content types.
Non-functional requirements
- High throughput, on the order of thousands of pages per second across the fleet (estimated, tunable to hardware and bandwidth).
- Politeness: a strict per-domain request rate so no single site is overloaded.
- Horizontal scalability by adding fetcher workers and frontier shards.
- Fault tolerance: a worker or node crash must not lose queued URLs or corrupt progress.
- Extensibility so new parsers, filters, and content types can be added.
- Avoidance of crawler traps and infinite URL spaces.
- Efficient use of network bandwidth and storage.
- Observability into crawl rate, error rate, and frontier depth per host.
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Pages to crawl
About 30 to 100 billion pages for a broad crawl
The indexed web is in the hundreds of billions of pages and a broad general crawler targets tens of billions. Derived, exact figures are not published by search engines.
Fetch throughput
About 5,000 pages per second sustained
Crawling 40 billion pages in about 90 days needs roughly 40e9 / (90 * 86400) which is close to 5,100 pages per second. Derived estimate.
Storage for raw HTML
About 2 to 4 PB per full crawl (compressed)
At an average of about 100 KB of raw HTML per page, 40 billion pages is roughly 4 PB uncompressed. Text compresses well, so 2 to 4 PB compressed is a reasonable estimate. Derived.
URL frontier size
Hundreds of billions of URLs known, tens of billions pending
Every crawled page yields on average tens of outbound links, so the set of discovered URLs grows much faster than the set fetched. Most are duplicates and get filtered. Derived.
Seen-URL set size
Roughly 100 billion URL fingerprints
One entry per unique URL ever seen. A bloom filter at a few bits per element keeps this in tens to low hundreds of GB of RAM across shards. Derived from the pages-to-crawl estimate.
Bandwidth
About 4 Gbps sustained inbound
5,000 pages per second times about 100 KB per page is roughly 500 MB per second which is about 4 Gbps before overhead. Derived estimate.
High-level architecture
The crawl begins with a set of seed URLs loaded into the URL frontier, the central data structure that decides what to fetch next. The frontier is not a single queue, it is a two-level structure of priority queues that rank URLs by importance and per-host politeness queues that enforce a rate limit for each domain. Fetcher workers pull URLs from the frontier, but before making a request they resolve the hostname through a DNS resolver backed by a local cache, since DNS lookups are slow and repeat constantly. Each worker checks the host's robots.txt rules, fetches the page, and hands the raw bytes to a content parser. The parser extracts outbound links, normalizes them, and passes each candidate through the dedup layer. Dedup has two parts: a URL-seen bloom filter that cheaply rejects URLs already known, and content hashing using a fingerprint such as simhash to catch pages whose bodies are identical or near duplicates even when the URL differs. URLs that survive dedup are enqueued back into the frontier. The raw content and metadata are written to the content store for downstream indexing. A separate scheduler tracks how often each page changes and reinserts pages into the frontier for recrawl so the stored copy stays fresh. The whole fetcher tier is sharded by host using consistent hashing so politeness state for a domain lives on one place.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
URL Frontier
The distributed queue that holds URLs waiting to be crawled. It combines priority queues, which order URLs by estimated importance and refresh need, with per-host politeness queues that release URLs for a given domain no faster than that domain's rate limit allows.
Fetcher Workers
A horizontally scaled pool of workers that pull URLs, perform the HTTP fetch, handle redirects, timeouts, and retries, and pass raw responses downstream. Workers are sharded by host so all requests to one domain are coordinated in one place.
DNS Resolver and Cache
Resolves hostnames to IP addresses. Because a naive crawler would issue millions of repeat lookups, a local cache with sensible TTLs is essential and DNS is a common hidden bottleneck if left unmanaged.
Content Parser and Link Extractor
Parses fetched HTML, extracts outbound links, normalizes them into a canonical form, and pulls out the text and metadata needed for indexing. It also strips out URLs the crawl scope should exclude.
Dedup Layer
Two mechanisms working together. A bloom filter over URL fingerprints answers 'have we seen this URL' cheaply, and content hashing with simhash or MinHash detects exact and near-duplicate page bodies so mirrored or boilerplate content is not stored twice.
Content Store
Durable storage for raw page bytes plus metadata such as fetch time, HTTP status, and content hash. Usually a blob store or distributed file system, sized in petabytes, feeding the downstream indexer.
Recrawl Scheduler
Estimates how frequently each page changes and decides when to reinsert it into the frontier. High-change pages like news homepages get short intervals, static pages get long ones, so bandwidth is spent where freshness matters.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
url_frontierurl_fingerprinthostprioritynext_fetch_atshard_idWorking queue of pending URLs. Physically split into priority sub-queues and per-host politeness sub-queues. next_fetch_at enforces both politeness delay and scheduled recrawl.
seen_urlsurl_fingerprintfirst_seen_atlast_crawled_atThe set of every URL ever discovered. In practice fronted by a bloom filter in memory for fast membership tests, with the exact set persisted for auditing and recovery.
page_contenturl_fingerprintcontent_hashhttp_statusfetched_atraw_body_refOne row per fetched page. raw_body_ref points at the blob in the content store. content_hash and simhash support exact and near-duplicate detection.
domain_politenesshostcrawl_delay_msrobots_rulesrobots_fetched_atlast_request_atPer-domain crawl rules cached from robots.txt plus the timestamp of the last request so the frontier can compute the earliest legal next fetch.
link_graphfrom_url_fingerprintto_url_fingerprintanchor_textdiscovered_atThe directed graph of which page links to which. Feeds link-based ranking such as PageRank and helps prioritize the frontier.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Designing the URL frontier
The frontier has to do two competing jobs at once: fetch important pages sooner, and never exceed a domain's polite rate. The classic answer is the Mercator two-stage design. A front set of priority queues assigns each URL to a queue based on its estimated importance or refresh need, so higher-value URLs are drawn more often. A back set of queues holds one queue per active host, and every back queue maps to exactly one host at a time. A worker asking for work picks a back queue whose next-fetch time has arrived, fetches that host's next URL, and the release time is pushed forward by the host's crawl delay. This cleanly separates 'what is worth crawling' from 'when is it polite to crawl it'. The frontier is huge, so it is sharded across machines and the pending URLs spill to disk while only the heads of the active queues stay in memory.
Politeness and per-domain rate limiting
The fastest way to get a crawler banned is to hit one host too hard. Politeness means capping the request rate to each domain, usually one request at a time per host with a delay of one to several seconds between requests, and honoring any crawl-delay in robots.txt. The critical design rule is that all URLs for a single host must route to the same politeness queue and ideally the same worker, otherwise several workers each think they are being polite while together they overwhelm the site. This is why the fetcher tier is sharded by host rather than by URL. Politeness directly caps per-host throughput, so total throughput comes from crawling many hosts in parallel, not from hammering a few.
Duplicate detection with bloom filters and content hashing
There are two kinds of duplicates and they need two tools. URL duplicates are handled with a bloom filter over URL fingerprints. Before enqueuing a URL the crawler asks the filter 'seen this?', and a negative answer is exact while a positive answer may be a false positive, which only costs a skipped enqueue and never a wrong crawl. A bloom filter keeps tens of billions of URLs in a few GB of RAM at a small false-positive rate, far cheaper than an exact set. Content duplicates are different: two different URLs can serve the same or nearly the same body, common with mirrors, print versions, and session-id URLs. Here you compute a content fingerprint such as simhash or MinHash and compare it to fingerprints already stored. simhash produces similar values for similar documents, so near-duplicates cluster together and can be dropped before they waste storage and index space.
DNS as a hidden bottleneck
Every fetch needs the host's IP, and a live DNS lookup can take tens of milliseconds. At thousands of pages per second, uncached DNS becomes a wall long before bandwidth does, and it can also overload upstream resolvers. The fix is a local DNS cache with reasonable TTLs, often a dedicated caching resolver in front of the fetcher fleet, plus prefetching the DNS for hosts already queued in the frontier. Because many URLs share a host, cache hit rates are very high in practice. Some large crawlers run their own resolver infrastructure to control TTL behavior and avoid third-party rate limits.
Distributed coordination and sharding with consistent hashing
One machine cannot crawl the web, so the frontier and the fetchers are partitioned across many nodes. Sharding by host using consistent hashing sends all URLs for a domain to the same shard, which keeps politeness state and robots rules for that domain in one place and avoids cross-node coordination on every request. Consistent hashing matters because the fleet changes size: when a node is added or removed, only a small slice of hosts move, so most in-flight politeness state is preserved. Discovered URLs are routed to the owning shard for their host, and each shard runs its own slice of the frontier independently.
Freshness and recrawl scheduling
A crawl is never done, because pages change. The scheduler estimates a change frequency for each page from its history, a news site homepage might change many times a day while an archived article may not change for years. It then sets a recrawl interval per page and reinserts it into the frontier when due. The goal is to spend limited fetch budget where it improves freshness the most, so high-change and high-importance pages are revisited often and stable pages rarely. HTTP conditional requests using ETag and If-Modified-Since help here, because a 304 Not Modified response costs almost no bandwidth and confirms the stored copy is still current.
Crawler traps, infinite spaces, and robots.txt
The web contains structures that can trap a naive crawler: calendars that generate an endless chain of next-month links, faceted search that produces a combinatorial explosion of filter URLs, and session ids that make every visit look like a new URL. Defenses include capping crawl depth per host, limiting the number of URLs fetched per domain, normalizing away known junk parameters, and watching for hosts that keep producing new URLs without new content, detectable through content hashing. On top of this the crawler must obey robots.txt: fetch and cache each host's rules, honor Disallow paths and crawl-delay, and re-fetch robots.txt periodically since it can change. Ignoring these rules gets a crawler blocked and can create legal and reputational problems.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Priority-ordered crawl versus plain breadth-first
BFS from the seeds is simple and gives broad coverage, but it treats a spam page the same as a major news site. A priority frontier crawls important and fast-changing pages sooner, which improves the value of a fixed fetch budget, at the cost of a more complex ranking and scheduling layer.
Bloom filter for seen URLs versus an exact set
A bloom filter fits tens of billions of URLs in a few GB and answers membership in constant time, but it has false positives that cause a few real URLs to be skipped. An exact key-value set never skips a page but costs far more memory and lookups. Most crawlers accept the tiny miss rate for the huge memory saving.
Politeness versus throughput
Strict per-host limits protect the sites you crawl and keep you from being banned, but they cap how fast any single domain can be fetched. You recover total throughput by widening parallelism across many hosts rather than by relaxing politeness on a few.
Storing full raw HTML versus only extracted text
Keeping raw HTML lets you re-parse later with better extractors and supports auditing, at a large storage cost in the petabyte range. Storing only extracted text is far smaller but locks you into today's parser and loses structure you may need later. Many crawlers keep compressed raw HTML for a window and extracted text long term.
Central frontier versus per-shard frontiers
A single logical frontier makes global prioritization easy but becomes a coordination and scaling bottleneck. Independent per-shard frontiers scale cleanly and localize politeness state, at the cost of only approximate global ordering across shards.
Aggressive recrawl versus conservative recrawl
Frequent recrawling keeps the index fresh but spends bandwidth re-fetching pages that did not change. Conservative intervals save bandwidth but let content go stale. Per-page change estimation plus conditional requests is the middle path that spends effort only where pages actually move.
How Web Crawler actually does it
The reference design most interviewers have in mind is Mercator, described by Allan Heydon and Marc Najork, which introduced the two-stage frontier with priority and per-host queues that later crawlers reuse. Google's early architecture was laid out by Sergey Brin and Lawrence Page in their Anatomy of a Large-Scale Hypertextual Web Search Engine paper, which pairs the crawler with the URL server, the store, and PageRank. Modern production crawlers such as Googlebot are far larger and not fully documented publicly, so exact rates and fleet sizes are estimates. For a real, inspectable corpus, Common Crawl publishes monthly open crawls of a few billion pages that anyone can download and study, and open-source crawlers like Apache Nutch and StormCrawler show how the frontier, fetchers, and dedup fit together in practice.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.