Web Crawler System Design Template

System Design: [Web crawler name]

Reference design for a polite, distributed web crawler that fetches and refreshes [1 billion] pages a month for a search index or a research corpus. It covers the URL frontier with priority and per-host politeness queues (after the Mercator design), URL and content deduplication including simhash near-duplicate detection, robots.txt handling under the Robots Exclusion Protocol (RFC 9309, September 2022), sitemaps, recrawl scheduling with HTTP conditional requests, and defences against crawler traps and spam (after IRLbot). It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks. Crawling, storing and reusing content can raise copyright, privacy and terms-of-service questions; this is a design format, not legal advice, so check the current official text and a qualified professional.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: The search index is fed by a single-machine crawler that fetches 40 million pages a month, revisits pages on a fixed 60-day cycle and keeps its URL set in memory. Index freshness complaints are rising, the crawler restarted from scratch twice last quarter after crashes, and one site owner blocked our address range after we sent bursts of parallel requests to a small server.]

1.2 Proposed solution

[Example: A distributed crawler partitions hosts across crawl nodes by a hash of the host name. On each node a URL frontier holds priority queues in front and per-host politeness queues behind, so no host ever sees more than one request at a time from us and the gap between requests adapts to how fast the host answers. Fetchers check a robots.txt cache that follows RFC 9309, use conditional requests for recrawls and store raw responses in WARC files. Extracted links pass a URL-seen check before entering the frontier; content passes exact and near-duplicate checks before indexing. A scheduler recrawls each page at an interval learned from how often it changes, and per-domain budgets stop traps and link farms from eating the crawl.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer, one first-in first-out queue shared by all fetchers. Show that relative links make long runs of URLs from the same host, so many fetchers hit one server at once. Then introduce the back queues with one host each and a heap ordered by the next allowed contact time.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: rendering JavaScript-heavy pages in a headless browser (separate render queue, later release)]
• [Example: indexing, ranking and serving search results (see system-design/design-full-text-search)]
• [Example: crawling behind logins or paywalls (never done by this crawler)]
• [Example: images, video and PDF files (link recorded, not fetched)]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Fetch rate and bandwidth

Consequence: one data-centre uplink of a few Gbit/s is enough. Bandwidth is not the hard constraint.

3.3 Concurrency and politeness

By Little's law, fetches in flight at peak are $R_{peak} \times t = 772 \times 1 \ \text{s} \approx 772$. Because each host gets at most one connection, at least 772 different hosts must be ready to fetch at any moment. Following the Mercator rule of thumb of three back queues per fetch slot, each crawl cluster keeps about $3 \times 772 = 2\,316$ active host queues.

The per-host limit also caps what one site can get: at one request per second, a site of 10 million pages needs $10^{7} / 86\,400 \approx 116$ days for a full pass, longer than the 30-day cycle. At a 10 s gap (a response time of 1 s times 10) a host gets at most 8 640 pages a day. Consequence: large sites are crawled by priority and sitemaps, not exhaustively, and the frontier must hold a long backlog per host on disk.

3.4 Storage

Frontier: about 150 bytes per queued URL with metadata; a backlog of $10^{9}$ URLs is about 150 GB, so the frontier lives on disk with only the head of each queue in memory.

3.5 Deduplication sets

URL-seen set as exact 8-byte fingerprints: $10^{10} \times 8 \ \text{B} = 80 \ \text{GB}$, partitioned by host across nodes and kept on SSD.

A Bloom filter in front with a 1% false-positive rate needs $-\ln(0.01) / (\ln 2)^{2} \approx 9.59$ bits per URL and about 7 hash functions, so about 12 GB in memory. A false positive means a new URL is wrongly treated as seen and skipped, so the filter only short-cuts "seen" answers that the exact set then confirms (7.4).

Link checks: $386 \times 50 \approx 19\,300$ lookups/s on average (38 600 at peak), about 1.67 billion a day. Consequence: batch lookups per node (sort and merge against the on-disk set, as IRLbot's DRUM does) instead of one random disk read per link.

Near-duplicate fingerprints: 64-bit simhash per page, $10^{9} \times 8 \ \text{B} = 8 \ \text{GB}$ per table; four permuted tables for a Hamming distance of 3 need 32 GB (7.4).

3.6 Summary

4. API design

4.1 Control and output interfaces

4.2 Fetch request sent to sites

The crawler identifies itself and uses conditional headers on recrawl (RFC 9110):

GET /news/2026/10/a-story HTTP/1.1
Host: www.example.org
User-Agent: Mozilla/5.0 (compatible; ExampleBot/1.0; +https://example.com/bot)
Accept: text/html,application/xhtml+xml
Accept-Encoding: gzip, br
If-None-Match: "a7c3-5f1e"
If-Modified-Since: Tue, 27 Oct 2026 08:12:00 GMT

A 304 Not Modified answer costs a few hundred bytes instead of the page. A 429 or 503 with Retry-After pauses the host for at least that time.

4.3 Output event

Each stored response produces one event; the hash below is shortened for the example.

{"url": "https://www.example.org/news/2026/10/a-story", "canonical_url": "https://www.example.org/news/2026/10/a-story", "fetched_at": "2026-10-30T09:14:05Z", "status": 200, "content_type": "text/html", "warc_ref": {"file": "crawl-20261030-0912-n17.warc.gz", "offset": 18273645}, "content_sha256": "3b1f9c0d7e2a4b6c", "simhash": "9f3a5c7e01d2b4a6", "duplicate_of": null, "robots_meta": ["index", "follow"], "links_out": 47}

5. Data model

5.1 Entities

Host state drives politeness; URL state drives scheduling; content records point to raw bytes in WARC files.

5.2 Storage choices

5.3 Access patterns

6. High-level design

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Frontier

• Front queues: one first-in first-out queue per priority level (Example: 5 levels). Priority comes from source (sitemap, seed, discovered link), estimated page importance and whether it is a recrawl.
• Back queues: each holds URLs of exactly one host. A host-to-queue table maps hosts to queues; a heap orders queues by the time their host may next be contacted.
• Getting a URL: a fetch slot pops the heap root (waiting if its time is in the future), takes the head URL of that queue and fetches it. When the fetch ends, the host's next allowed time is computed (7.3) and the queue goes back into the heap. When a back queue empties, it is refilled from the front queues with a bias toward high priority.
• Partitioning: host name hash picks the crawl node. A link to another host is sent to its owner node in batches.

7.2 Robots.txt under RFC 9309

RFC 9309 also notes that robots.txt is not a form of access authorisation. Respecting it is the crawler's policy and commitment, so log every decision.

7.3 Politeness and adaptive delay

With a 0.4 s response the next allowed time is 4 s later; with a 0.1 s 304 response it is the 1 s minimum.

7.4 Deduplication

• URL normalisation before the seen check: lowercase scheme and host, remove default ports and fragments, resolve dot segments, apply known safe rules per host (for example removing session-ID parameters that the host is known to ignore). Normalise only in ways that keep the meaning of the URL, and keep the original URL for fetching.
• URL-seen: the Bloom filter answers "definitely new" cheaply; a "maybe seen" answer and every new URL go into a per-node batch that is sorted and merged against the exact fingerprint files every few minutes. New URLs go to the frontier; the files get the new fingerprints.
• Exact content: SHA-256 of the normalised body; a hit links the URL to the existing content record and skips indexing.
• Near-duplicate content: 64-bit simhash of the page text. Split the 64 bits into four 16-bit blocks; if two fingerprints differ in at most 3 bits, at least one block is identical, so four tables, each sorted by one block, find every candidate. With $10^{9}$ pages each table returns about $10^{9} / 2^{16} \approx 15\,259$ candidates per probe, about 61 000 for four, checked with XOR and bit counting in well under a millisecond.

7.5 Recrawl scheduling

• Estimate a change rate per URL from past fetches (changed or not, by content hash). Set the next interval so the expected number of missed changes stays small, bounded between 1 day and 90 days.
• Raise priority for URLs that sitemaps list with a recent lastmod; treat changefreq and priority as hints only. Sitemaps hold at most 50 000 URLs and 50 MB uncompressed per file.
• Recrawls always send If-None-Match and If-Modified-Since when the previous response had an ETag or Last-Modified.
• URLs that return 404 or 410 twice are removed and a crawl.removed event is emitted; a newly disallowed URL is removed at once.

7.6 Traps, spam and budgets

7.7 How real systems do it

How real systems do it (paraphrased, not copied). Heydon and Najork's Mercator papers (World Wide Web journal, 1999, and the Compaq SRC report "High-Performance Web Crawling", 2001) describe a frontier of prioritised front queues feeding back queues that each hold one host, with a heap ordered by when each host may be contacted again. The 2001 report says the team first allowed only one thread per host, still received complaints about unbroken request streams, and moved to waiting about ten times the last download time, with three times as many back queues as threads. It also reports that DNS lookups took 70% of each thread's elapsed time until a custom parallel resolver cut it to 14%, and that about 80% of links are relative, so host-based partitioning keeps most URLs on the node that found them. Lee, Leonard, Wang and Loguinov's IRLbot paper (WWW 2008, extended in ACM Transactions on the Web, 2009) reports 6.3 billion valid HTML pages in 41 days from a single server, at about 1 789 pages per second. Its DRUM structure batches URL-seen checks through disk buckets, and its STAR and BEAST methods give each domain a page budget based on links from other domains, so spam cannot snowball. Manku, Jain and Das Sarma (WWW 2007) report that 64-bit simhash fingerprints with up to 3 differing bits work for about 8 billion pages. Read the originals in 12.3; this box keeps only the general idea.

8. Scalability and performance

Per-fetch time budget (target: a fetch slot is busy for about 1 s on average):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security, privacy and legal questions

Questions to answer with legal and privacy reviewers (not legal advice): Which jurisdictions' data protection laws apply to the pages we store? Does the purpose (search, archive, AI training) require an opt-out mechanism beyond robots.txt? How long may raw copies be kept?

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Raise the global fetch rate in steps of 2x per week while watching complaints and error mix. Any step can be undone by lowering the global rate limit; per-host blocks apply at once.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on the Robots Exclusion Protocol (RFC 9309, September 2022), HTTP Semantics (RFC 9110), the sitemaps.org protocol and the published crawler research cited in 12.3. RFC section numbers are cited and the rules are paraphrased; no specification or paper text was copied, and all capacity numbers are illustrative examples. ISO 28500:2017 (WARC) is a paid standard and is named only; official text: ISO, purchase or licensed access at https://www.iso.org/standard/68004.html (the IIPC publishes WARC 1.1 openly). Check the current versions before relying on them. This is not legal advice; check the current official text and a qualified professional. Original text written for this collection. Template licence: MIT.