# System Design: [Web crawler name]

> Reference design for a polite, distributed web crawler that fetches and
> refreshes [1 billion] pages a month for a search index or a research
> corpus. It covers the URL frontier with priority and per-host politeness
> queues (after the Mercator design), URL and content deduplication
> including simhash near-duplicate detection, robots.txt handling under the
> Robots Exclusion Protocol (RFC 9309, September 2022), sitemaps, recrawl
> scheduling with HTTP conditional requests, and defences against crawler
> traps and spam (after IRLbot). It follows the structure of the System
> Design Document template, adds a 45-minute interview path,
> access-pattern and failure-mode tables and a cited "How real systems do
> it" box, and is pre-filled with a worked design to adapt. Replace every
> `[bracketed]` value and every row marked *Example*; values marked
> *Example* are assumptions, not benchmarks. Crawling, storing and reusing
> content can raise copyright, privacy and terms-of-service questions; this
> is a design format, not legal advice, so check the current official text
> and a qualified professional.

<!--
Guidance: How to use this template
- State the purpose of the crawl first (search index, archive, price
  monitoring, AI training corpus): it decides scope, freshness, what you
  store and which legal reviews you need.
- Write the politeness policy (7.3) before sizing anything; politeness, not
  bandwidth, limits how fast you can crawl any one site.
- Publish a crawler information page and a stable user-agent token before
  the first fetch, so site owners can identify and control your crawler.
- Plan for the web being adversarial: infinite calendars, session IDs in
  URLs and link farms will find you on day one (7.6).
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-CRAWL-001] |
| System / feature | [Web crawler for the search index] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Intermediate |
| User-agent product token | [ExampleBot] |
| Crawler information page | [https://example.com/bot] |
| Related | [Search indexing pipeline], [Legal review of crawl purpose], [ADR: URL-seen store] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Legal and privacy | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced fixed 1 s host delay with adaptive delay based on response time |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added per-domain crawl budgets after a link-farm incident |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* The search index is fed by a single-machine crawler that
fetches 40 million pages a month, revisits pages on a fixed 60-day cycle
and keeps its URL set in memory. Index freshness complaints are rising,
the crawler restarted from scratch twice last quarter after crashes, and
one site owner blocked our address range after we sent bursts of parallel
requests to a small server.]

### 1.2 Proposed solution

[*Example:* A distributed crawler partitions hosts across crawl nodes by a
hash of the host name. On each node a URL frontier holds priority queues
in front and per-host politeness queues behind, so no host ever sees more
than one request at a time from us and the gap between requests adapts to
how fast the host answers. Fetchers check a robots.txt cache that follows
RFC 9309, use conditional requests for recrawls and store raw responses in
WARC files. Extracted links pass a URL-seen check before entering the
frontier; content passes exact and near-duplicate checks before indexing.
A scheduler recrawls each page at an interval learned from how often it
changes, and per-domain budgets stop traps and link farms from eating the
crawl.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* How are URLs partitioned across nodes? | By hash of host name, so each host's politeness state lives on one node | 7.1 |
| Q-02 | *Example:* How fast may we crawl one host? | One connection per host; wait max(1 s, 10 x last response time) | 7.3 |
| Q-03 | *Example:* Exact or approximate URL-seen set? | Exact fingerprint set on disk with a Bloom filter in front | 7.4 |
| Q-04 | *Example:* What happens when robots.txt returns 5xx? | Treat the site as fully disallowed, as RFC 9309 requires | 7.2 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: purpose, HTML only or more, freshness, politeness; below-the-line items (2.3) | Purpose and three constraints |
| 5 to 10 | Estimates: pages per second, bandwidth, storage, URL-seen set size (section 3) | "Politeness, not bandwidth, is the limit" consequence |
| 10 to 15 | Interfaces: seed and control API, output to the indexer (section 4) | Two interfaces |
| 15 to 25 | High-level design: frontier, fetcher, robots cache, parser, dedup, store (section 6) | The crawl loop diagram |
| 25 to 40 | One deep dive: frontier and politeness (default, 7.1 and 7.3), or deduplication (7.4), or recrawl scheduling (7.5) | Options, choice, reason |
| 40 to 45 | Failures (node loss, DNS outage, trap), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer, one first-in first-out
queue shared by all fetchers. Show that relative links make long runs of
URLs from the same host, so many fetchers hit one server at once. Then
introduce the back queues with one host each and a heap ordered by the
next allowed contact time.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Crawl HTML pages over HTTP and HTTPS starting from seed lists and sitemaps | Must |
| FR-02 | *Example:* Obey robots.txt as specified in RFC 9309 for our product token | Must |
| FR-03 | *Example:* At most one open connection per host; adaptive delay between requests | Must |
| FR-04 | *Example:* Never fetch the same URL twice in one crawl cycle unless it is due for recrawl | Must |
| FR-05 | *Example:* Detect exact and near-duplicate content and keep one canonical copy | Must |
| FR-06 | *Example:* Recrawl pages at an interval based on their observed change rate | Must |
| FR-07 | *Example:* Store raw responses and extracted text for the indexer | Must |
| FR-08 | *Example:* Honour `noindex` and `nofollow` robots meta tags and the `X-Robots-Tag` header in the indexer | Should |
| FR-09 | *Example:* Let operators block a host or path within 10 minutes | Must |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Throughput | *Example:* 1 billion successful page fetches per 30 days | Fetch counter |
| NFR-02 | Politeness | *Example:* Zero hosts with two concurrent connections from us | Connection audit per host |
| NFR-03 | Freshness | *Example:* Median age of the indexed copy of top 10 million pages under 3 days | Sampled recrawl comparison |
| NFR-04 | Recoverability | *Example:* A node crash loses at most 10 minutes of crawl progress | Chaos test |
| NFR-05 | Duplicate waste | *Example:* Under 5% of fetches return content already stored | Content-hash hit rate |
| NFR-06 | Complaints | *Example:* Every site-owner complaint answered within 1 business day | Ticket system |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* rendering JavaScript-heavy pages in a headless browser (separate render queue, later release)]
- [*Example:* indexing, ranking and serving search results (see `system-design/design-full-text-search`)]
- [*Example:* crawling behind logins or paywalls (never done by this crawler)]
- [*Example:* images, video and PDF files (link recorded, not fetched)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Pages fetched per 30 days | $N$ | *Example:* 1 billion | Index plan |
| Peak to average ratio | $f$ | *Example:* 2 | Scheduler smoothing |
| Average HTML size, uncompressed | $s$ | *Example:* 100 KB | Sample crawl |
| Stored size after compression | $s_{c}$ | *Example:* 25 KB | Sample crawl, 4:1 |
| Overhead for robots.txt, redirects, retries | $o$ | *Example:* 20% | Sample crawl |
| Links extracted per page | $l$ | *Example:* 50 | Sample crawl |
| Known URLs (seen set) | $U$ | *Example:* 10 billion | Link discovery estimate |
| Average fetch time including connection | $t$ | *Example:* 1 s | Sample crawl |
| Parse CPU per page | $p$ | *Example:* 5 ms | Profiling |
| Months of history kept | $h$ | *Example:* 6 | Index requirement |

### 3.2 Fetch rate and bandwidth

$$
R_{avg} = \frac{10^{9}}{30 \times 86\,400} \approx 386 \ \text{pages/s}, \qquad R_{peak} = 2 \times 386 \approx 772 \ \text{pages/s}
$$

$$
B_{avg} = 386 \times 100 \ \text{KB} \times 1.2 \approx 370 \ \text{Mbit/s}, \qquad B_{peak} \approx 741 \ \text{Mbit/s}
$$

Consequence: one data-centre uplink of a few Gbit/s is enough. Bandwidth is
not the hard constraint.

### 3.3 Concurrency and politeness

By Little's law, fetches in flight at peak are $R_{peak} \times t = 772
\times 1 \ \text{s} \approx 772$. Because each host gets at most one
connection, at least 772 different hosts must be ready to fetch at any
moment. Following the Mercator rule of thumb of three back queues per fetch
slot, each crawl cluster keeps about $3 \times 772 = 2\,316$ active host
queues.

The per-host limit also caps what one site can get: at one request per
second, a site of 10 million pages needs $10^{7} / 86\,400 \approx 116$
days for a full pass, longer than the 30-day cycle. At a 10 s gap (a
response time of 1 s times 10) a host gets at most 8 640 pages a day.
Consequence: large sites are crawled by priority and sitemaps, not
exhaustively, and the frontier must hold a long backlog per host on disk.

### 3.4 Storage

$$
S_{month} = 10^{9} \times 25 \ \text{KB} = 25 \ \text{TB}, \qquad S_{total} = 25 \times 6 = 150 \ \text{TB} \rightarrow 225 \ \text{TB with erasure coding at 1.5x}
$$

Frontier: about 150 bytes per queued URL with metadata; a backlog of
$10^{9}$ URLs is about 150 GB, so the frontier lives on disk with only the
head of each queue in memory.

### 3.5 Deduplication sets

URL-seen set as exact 8-byte fingerprints: $10^{10} \times 8 \ \text{B} =
80 \ \text{GB}$, partitioned by host across nodes and kept on SSD.

A Bloom filter in front with a 1% false-positive rate needs
$-\ln(0.01) / (\ln 2)^{2} \approx 9.59$ bits per URL and about 7 hash
functions, so about 12 GB in memory. A false positive means a new URL is
wrongly treated as seen and skipped, so the filter only short-cuts "seen"
answers that the exact set then confirms (7.4).

Link checks: $386 \times 50 \approx 19\,300$ lookups/s on average (38 600
at peak), about 1.67 billion a day. Consequence: batch lookups per node
(sort and merge against the on-disk set, as IRLbot's DRUM does) instead of
one random disk read per link.

Near-duplicate fingerprints: 64-bit simhash per page, $10^{9} \times 8 \
\text{B} = 8 \ \text{GB}$ per table; four permuted tables for a Hamming
distance of 3 need 32 GB (7.4).

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Fetch rate | ~386 pages/s average, ~772 peak | A small fetcher fleet with asynchronous I/O |
| Bandwidth | ~370 Mbit/s average, ~741 peak | Not the bottleneck |
| Hosts in flight | at least ~772, ~2 316 active queues | Per-host back queues with a ready-time heap |
| Large sites | ~116 days for 10 million pages at 1 request/s | Priority and sitemaps, not full coverage |
| Storage | ~25 TB/month, ~225 TB for 6 months stored | Compressed WARC in object storage |
| URL-seen | 80 GB exact, 12 GB Bloom filter | Host-partitioned, batched checks |
| Parse CPU | ~3.9 cores at peak | Parsing is cheap; scale on I/O |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Control and output interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Control API | `POST /v1/seeds` | Add seed URLs or sitemap URLs with a priority |
| Control API | `PUT /v1/hosts/{host}/policy` | Block, slow down or set a budget for a host |
| Control API | `GET /v1/hosts/{host}/status` | Queue length, last fetch, robots state for support |
| Control API | `POST /v1/recrawl` | Force recrawl of a URL list (operator only) |
| Output stream | `crawl.fetched` events | One event per stored response for the indexer |
| Output stream | `crawl.removed` events | URL gone (404, 410) or newly disallowed |

### 4.2 Fetch request sent to sites

The crawler identifies itself and uses conditional headers on recrawl
(RFC 9110):

```http
GET /news/2026/10/a-story HTTP/1.1
Host: www.example.org
User-Agent: Mozilla/5.0 (compatible; ExampleBot/1.0; +https://example.com/bot)
Accept: text/html,application/xhtml+xml
Accept-Encoding: gzip, br
If-None-Match: "a7c3-5f1e"
If-Modified-Since: Tue, 27 Oct 2026 08:12:00 GMT
```

A `304 Not Modified` answer costs a few hundred bytes instead of the page.
A `429` or `503` with `Retry-After` pauses the host for at least that time.

### 4.3 Output event

Each stored response produces one event; the hash below is shortened for
the example.

```json
{"url": "https://www.example.org/news/2026/10/a-story", "canonical_url": "https://www.example.org/news/2026/10/a-story", "fetched_at": "2026-10-30T09:14:05Z", "status": 200, "content_type": "text/html", "warc_ref": {"file": "crawl-20261030-0912-n17.warc.gz", "offset": 18273645}, "content_sha256": "3b1f9c0d7e2a4b6c", "simhash": "9f3a5c7e01d2b4a6", "duplicate_of": null, "robots_meta": ["index", "follow"], "links_out": 47}
```

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

Host state drives politeness; URL state drives scheduling; content records
point to raw bytes in WARC files.

```mermaid
erDiagram
  HOST {
    string host PK
    int partition
    string robots_state
    datetime robots_fetched_at
    int delay_ms
    datetime next_allowed_at
    int budget_pages
  }
  URL {
    bigint url_fp PK
    string url
    string host FK
    int priority
    datetime last_fetched_at
    datetime next_due_at
    float change_rate
    string etag
  }
  CONTENT {
    string content_sha256 PK
    bigint simhash
    string warc_file
    bigint warc_offset
  }
  FETCH {
    bigint fetch_id PK
    bigint url_fp FK
    datetime fetched_at
    int status
    string content_sha256 FK
  }
  HOST ||--o{ URL : "has"
  URL ||--o{ FETCH : "fetched as"
  CONTENT ||--o{ FETCH : "stored for"
```

### 5.2 Storage choices

| Data | Store | Key | Main queries | Why |
|---|---|---|---|---|
| Frontier queues | Disk-backed queues per node, heads in memory | Priority level; host | Next URL for the next ready host | 150 GB backlog cannot live in memory |
| Host state and robots cache | In-memory map per node, checkpointed | `host` | Is this path allowed, when may we fetch next | Needed on every fetch |
| URL-seen set | Sorted fingerprint files on SSD per node, Bloom filter in memory | `url_fp` (8 bytes) | Batch check and insert | 80 GB exact set, batched access |
| URL state for recrawl | *Example:* Distributed key-value store | `url_fp` | Due URLs by time, change history | Survives node moves |
| Raw responses | WARC files in object storage | File and offset | Indexer reads, reprocessing | Standard archive format, cheap storage |
| Content and simhash index | Key-value store and in-memory permuted tables | `content_sha256`; simhash blocks | Exact and near-duplicate checks | Fast lookups |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Pick the next URL to fetch | ~772/s at peak | Heap of host queues by next allowed time | Hash of host: one node owns each host, so politeness needs no coordination |
| Check robots rules for a URL | Every fetch | Host map, rules cached up to 24 h | Same host partition |
| Check if a link is new | ~19 300/s average | Bloom filter, then batched merge against `url_fp` files | Link is routed to the node that owns its host; most links are same-host |
| Find duplicate content | Every 200 response | `content_sha256`, then simhash tables | Global, sharded by hash prefix |
| Find URLs due for recrawl | Continuous | `next_due_at` per host | Host partition keeps due URLs next to their politeness state |
| Block a host on request | Rare | Host map plus control API | Push to the owning node within minutes |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  seeds([Seeds and sitemaps]) --> fr[URL frontier]
  sched[Recrawl scheduler] --> fr
  fr --> fetch[Fetchers]
  fetch --> robots[Robots cache]
  fetch --> dns[DNS cache and resolver]
  fetch --> web([Web servers])
  web --> fetch
  fetch --> warc[(WARC object storage)]
  fetch --> parse[Parser and link extractor]
  parse --> cdedup[Content dedup: hash and simhash]
  parse --> filt[URL filter and normaliser]
  filt --> split[Host splitter]
  split --> seen[URL-seen check]
  split --> peers[[Other crawl nodes]]
  seen --> fr
  cdedup --> out[[crawl.fetched events]]
  out --> indexer([Indexer])
```

| Component | Responsibility | Scales by |
|---|---|---|
| URL frontier | Priority front queues, per-host back queues, ready-time heap | Hosts per node |
| Fetchers | Asynchronous HTTP, conditional requests, timeouts | Connections per node |
| Robots cache | Fetch, parse and cache robots.txt per host (7.2) | With host partition |
| DNS resolver | Asynchronous resolution with a local cache | Resolver instances |
| Parser and link extractor | Extract links, canonical tags, robots meta | CPU cores |
| URL filter and normaliser | Normalise, drop out-of-scope and trap-like URLs | CPU cores |
| Host splitter | Route each URL to the node that owns its host | Nodes |
| URL-seen check | Bloom filter and batched exact set | Disk per node |
| Content dedup | Exact hash, simhash near-duplicate search | Memory for tables |
| Recrawl scheduler | Compute next due time per URL (7.5) | With URL state store |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Frontier and partitioning | One global queue; partition by URL hash; partition by host hash | Host hash, front and back queues per node | Politeness state local to one node; most links are same-host | 7.1 |
| Politeness | Fixed delay; delay from robots extensions; adaptive delay | Adaptive: max(1 s, 10 x last response time), back off on errors | Fast sites crawl faster, slow sites are protected | 7.3 |
| URL-seen | In-memory set; Bloom filter only; exact disk set with batching | Bloom filter plus batched exact disk set | No silent loss of new URLs; bounded memory | 7.4 |
| Recrawl | Fixed cycle; sitemap only; change-rate model | Change-rate model plus sitemap hints and conditional requests | Spend fetches where content changes | 7.5 |

### 7.1 Frontier

- **Front queues**: one first-in first-out queue per priority level (*Example:*
  5 levels). Priority comes from source (sitemap, seed, discovered link),
  estimated page importance and whether it is a recrawl.
- **Back queues**: each holds URLs of exactly one host. A host-to-queue
  table maps hosts to queues; a heap orders queues by the time their host
  may next be contacted.
- **Getting a URL**: a fetch slot pops the heap root (waiting if its time is
  in the future), takes the head URL of that queue and fetches it. When the
  fetch ends, the host's next allowed time is computed (7.3) and the queue
  goes back into the heap. When a back queue empties, it is refilled from
  the front queues with a bias toward high priority.
- **Partitioning**: host name hash picks the crawl node. A link to another
  host is sent to its owner node in batches.

### 7.2 Robots.txt under RFC 9309

| Situation | Behaviour in this design | RFC 9309 basis |
|---|---|---|
| Before the first fetch on a host | Fetch `/robots.txt` for that scheme, host and port | Section 2.3 |
| Group selection | Match our product token case-insensitively; fall back to the `*` group | Section 2.2.1 |
| Allow and disallow conflict | The rule with the longest matching path wins; on a tie, allow wins | Section 2.2.2 |
| Special characters | Support `*` (any sequence) and `$` (end of path); `#` starts a comment | Section 2.2.3 |
| Redirects for robots.txt | Follow at least 5; more than 5 means treat as unavailable | Section 2.3.1.2 |
| 4xx (unavailable) | Crawl allowed | Section 2.3.1.3 |
| 5xx or network error (unreachable) | Treat as complete disallow; keep a cached copy if one exists and retry | Section 2.3.1.4 |
| Caching | Reuse for at most 24 hours unless the file is unreachable | Section 2.4 |
| File size | Parse at least the first 500 KiB | Section 2.5 |
| `Crawl-delay` and `Sitemap` lines | Not defined by RFC 9309; `Sitemap` used as a hint, `Crawl-delay` respected as a minimum gap by policy | Section 2.2.4 allows other records |

RFC 9309 also notes that robots.txt is not a form of access
authorisation. Respecting it is the crawler's policy and commitment, so
log every decision.

### 7.3 Politeness and adaptive delay

| Rule | *Example* value |
|---|---|
| Connections per host | 1 |
| Gap after a fetch | max(1 s, 10 x the last response time), the factor that Mercator reported using |
| On 429 or 503 | Wait for `Retry-After` if present, else double the gap up to 1 hour |
| On repeated timeouts | Double the gap; pause the host for 1 day after 10 failures in a row |
| Per-IP limit | Hosts that share one IP address share one politeness budget |
| Operator override | Block or set a slower gap per host within 10 minutes (FR-09) |

```mermaid
sequenceDiagram
  participant H as Ready-time heap
  participant F as Fetch slot
  participant R as Robots cache
  participant W as Web server
  participant P as Parser
  H->>F: queue for host www.example.org, ready now
  F->>R: allowed? /news/2026/10/a-story
  R-->>F: allowed, rules cached 3 h ago
  F->>W: GET with If-None-Match
  alt 200 OK
    W-->>F: page, 0.4 s
    F->>P: parse and extract links
  else 304 Not Modified
    W-->>F: not modified, 0.1 s
    F->>F: update last check only
  else 429 or 503
    W-->>F: Retry-After 120
    F->>F: set gap to 120 s
  end
  F->>H: requeue host with next allowed time
```

With a 0.4 s response the next allowed time is 4 s later; with a 0.1 s 304
response it is the 1 s minimum.

### 7.4 Deduplication

- **URL normalisation** before the seen check: lowercase scheme and host,
  remove default ports and fragments, resolve dot segments, apply known
  safe rules per host (for example removing session-ID parameters that the
  host is known to ignore). Normalise only in ways that keep the meaning of
  the URL, and keep the original URL for fetching.
- **URL-seen**: the Bloom filter answers "definitely new" cheaply; a
  "maybe seen" answer and every new URL go into a per-node batch that is
  sorted and merged against the exact fingerprint files every few minutes.
  New URLs go to the frontier; the files get the new fingerprints.
- **Exact content**: SHA-256 of the normalised body; a hit links the URL
  to the existing content record and skips indexing.
- **Near-duplicate content**: 64-bit simhash of the page text. Split the
  64 bits into four 16-bit blocks; if two fingerprints differ in at most 3
  bits, at least one block is identical, so four tables, each sorted by
  one block, find every candidate. With $10^{9}$ pages each table returns
  about $10^{9} / 2^{16} \approx 15\,259$ candidates per probe, about 61 000
  for four, checked with XOR and bit counting in well under a millisecond.

### 7.5 Recrawl scheduling

- Estimate a change rate per URL from past fetches (changed or not, by
  content hash). Set the next interval so the expected number of missed
  changes stays small, bounded between 1 day and 90 days.
- Raise priority for URLs that sitemaps list with a recent `lastmod`;
  treat `changefreq` and `priority` as hints only. Sitemaps hold at most
  50 000 URLs and 50 MB uncompressed per file.
- Recrawls always send `If-None-Match` and `If-Modified-Since` when the
  previous response had an `ETag` or `Last-Modified`.
- URLs that return 404 or 410 twice are removed and a `crawl.removed`
  event is emitted; a newly disallowed URL is removed at once.

### 7.6 Traps, spam and budgets

| Problem | Signal | Defence |
|---|---|---|
| Infinite calendars and faceted URLs | Many URLs per host with the same path pattern | Cap URLs per path pattern; cap query parameters |
| Session IDs in URLs | Same content under many URLs | Learned parameter removal per host; content hash dedup |
| Very long or deep URLs | Length, depth from seed | Drop URLs over 2 048 characters or 16 levels deep |
| Link farms and spam domains | Many hosts under one domain with few links from outside | Budget per registered domain in proportion to links from other domains (IRLbot's approach) |
| Soft 404 pages | 200 status with "not found" template | Compare with a probe for a random URL on the host |

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). Heydon and
> Najork's Mercator papers (World Wide Web journal, 1999, and the Compaq
> SRC report "High-Performance Web Crawling", 2001) describe a frontier of
> prioritised front queues feeding back queues that each hold one host,
> with a heap ordered by when each host may be contacted again. The 2001
> report says the team first allowed only one thread per host, still
> received complaints about unbroken request streams, and moved to waiting
> about ten times the last download time, with three times as many back
> queues as threads. It also reports that DNS lookups took 70% of each
> thread's elapsed time until a custom parallel resolver cut it to 14%, and
> that about 80% of links are relative, so host-based partitioning keeps
> most URLs on the node that found them. Lee, Leonard, Wang and Loguinov's
> IRLbot paper (WWW 2008, extended in ACM Transactions on the Web, 2009)
> reports 6.3 billion valid HTML pages in 41 days from a single server, at
> about 1 789 pages per second. Its DRUM structure batches URL-seen checks
> through disk buckets, and its STAR and BEAST methods give each domain a
> page budget based on links from other domains, so spam cannot snowball.
> Manku, Jain and Das Sarma (WWW 2007) report that 64-bit simhash
> fingerprints with up to 3 differing bits work for about 8 billion pages.
> Read the originals in 12.3; this box keeps only the general idea.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Fetch rate | Asynchronous fetchers, many hosts in flight | *Example:* ~2 000 concurrent connections per node |
| Host diversity | Enough ready hosts in the heap | Crawls focused on few sites are politeness-bound, not hardware-bound |
| URL-seen | Batched merges per node | Merge time grows with the set; add nodes and re-split hosts |
| DNS | Local caching resolver, asynchronous lookups | Upstream resolver rate limits |
| Storage | Append-only WARC in object storage | Cost, not technology |

Per-fetch time budget (target: a fetch slot is busy for about 1 s on
average):

| Step | Budget |
|---|---|
| Heap pop and robots check (memory) | 0.1 ms |
| DNS (cached in most cases) | 5 ms average |
| TCP and TLS connection (reused when possible) | 150 ms |
| Server response and transfer | 800 ms |
| Write WARC record and hand off to parser | 10 ms |
| Headroom | about 35 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Crawl node | Crash | Heartbeat | Hosts owned by that node pause | Frontier and seen files on durable disk, checkpoint every 10 min; hosts reassigned by consistent hashing | *Example:* 10 min |
| DNS resolver | Upstream outage | Resolution error rate | All new hosts | Serve from cache with extended TTL; pause hosts that cannot resolve | *Example:* 30 min |
| Robots cache | robots.txt unreachable (5xx) | Status counters | One host | Full disallow per RFC 9309; retry with backoff; use cached copy if present | Host-dependent |
| Object storage | Write failures | Error rate | Fetch results not stored | Local spill buffer for 1 h; pause fetchers after that | *Example:* 1 h |
| Trap or spam domain | Explosive URL growth on one domain | URLs per domain per hour | Crawl budget wasted | Domain budget, path-pattern caps, operator block | *Example:* 1 h |
| Site owner complaint | Overload or unwanted crawl | Complaint channel | Reputation, IP blocks | Block or slow the host in 10 min; review the delay policy | *Example:* 1 business day |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security, privacy and legal questions

| Threat or duty | Control |
|---|---|
| Server-side request forgery through crafted links (internal IP ranges, metadata endpoints) | Resolve then block private, loopback and link-local addresses; fetch from an isolated network |
| Malicious content (parser exploits, decompression bombs) | Size limits before and after decompression, sandboxed parser, timeouts |
| Personal data in crawled pages | Purpose limitation, retention limits and deletion on request, reviewed with privacy counsel |
| Copyright and terms of service | Legal review of the crawl purpose and reuse of stored content before launch |
| Impersonation of our crawler by others | Publish verification method (reverse DNS or published IP ranges) on the crawler page |
| [Threat] | [Control] |

Questions to answer with legal and privacy reviewers (not legal advice):
Which jurisdictions' data protection laws apply to the pages we store?
Does the purpose (search, archive, AI training) require an opt-out
mechanism beyond robots.txt? How long may raw copies be kept?

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Throughput | Successful fetches per hour | Below 80% of plan for 1 h |
| Politeness | Max concurrent connections per host | Any value above 1 |
| Error mix | Share of 4xx, 5xx, timeouts | 5xx share doubling vs. 7-day baseline |
| Duplicate waste | Exact plus near-duplicate share | Above 5% |
| Frontier health | Ready hosts in the heap | Below 3 x fetch slots |
| Complaints | Tickets per week | Any unanswered after 1 business day |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Crawl nodes (fetch, frontier, seen set) | [$/node-month] | *Example:* 8 | [$] |
| Bandwidth | [$/TB] | *Example:* about 120 TB transferred per month (1 billion x 100 KB x 1.2) | [$] |
| Object storage | [$/TB-month] | *Example:* 225 TB at steady state | [$] |
| Key-value store for URL state | [$/month] | [Nodes] | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Partitioning | By URL hash; by host hash | By host hash | Politeness without coordination; local links stay local | Uneven load when one host is huge |
| URL-seen | Bloom only; exact in memory; exact on disk with batching | Bloom plus batched exact disk set | No silent loss, bounded memory | Minutes of delay before new URLs enter the frontier |
| Politeness | Fixed delay; adaptive | Adaptive with backoff | Fair to slow servers, fast on fast ones | Harder to predict per-host throughput |
| Rendering | Render all pages; HTML only | HTML only now, render queue later | 10x to 100x cheaper per page | Misses content built by JavaScript |
| Storage | Extracted text only; raw WARC | Raw WARC plus text | Reprocessing without refetching | 4x more storage than text alone |
| What changes at 10x / 0.1x | 10x (10 billion pages a month, about 3 860 pages/s average): about 7 700 hosts in flight at peak, an 800 GB exact seen set and about 2.25 PB stored for 6 months with erasure coding, so more nodes, a dedicated URL-seen service and tiered storage; politeness still caps each host. 0.1x (100 million a month, about 39 pages/s): one or two nodes with an in-memory seen set (8 GB at 1 billion URLs) are enough | Review at each doubling of the target | Host diversity and the seen set grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Single-node crawler with frontier, robots, WARC output; crawler page published | RFC 9309 test suite passes; 1 million pages without a complaint | YYYY-MM-DD |
| M2 | Host-partitioned cluster, batched seen set | 100 million pages in 30 days; node crash drill passes | YYYY-MM-DD |
| M3 | Content dedup and recrawl scheduler | Duplicate waste under 5%; freshness target met for top 10 million | YYYY-MM-DD |
| M4 | Full rate, budgets and trap defences | 1 billion pages in 30 days | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Raise the global fetch rate in steps of 2x per week while
watching complaints and error mix. Any step can be undone by lowering the
global rate limit; per-host blocks apply at once.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* IP ranges blocked by large hosting providers after complaints | Medium | High | Conservative delays, fast response to complaints, published verification | [@name] |
| R-02 | *Example:* Legal challenge to storing or reusing content | Low | High | Legal review before M1; retention and deletion process | [@name] |
| R-03 | *Example:* Trap consumes a large share of the crawl | Medium | Medium | Domain budgets and caps from M2 | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we honour AI-specific opt-out signals beyond robots.txt, and which ones? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which share of pages needs JavaScript rendering to be useful? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Frontier | The set of URLs waiting to be fetched, with their order |
| Politeness | Limits on how often and how concurrently we contact one host |
| Back queue | A queue that holds URLs of one host only |
| URL-seen test | Check whether a URL has been discovered before |
| Simhash | Fingerprint where similar documents get fingerprints that differ in few bits |
| WARC | Web ARChive file format for raw HTTP responses (WARC 1.1, ISO 28500:2017) |
| Crawler trap | URL space that generates unlimited distinct URLs |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), Robots Exclusion Protocol, Koster, Illyes, Zeller and Sassman, September 2022 (Standards Track) | 7.2 |
| RFC 9110 HTTP Semantics | Conditional requests, `Retry-After`, 4.2 |
| [Sitemaps XML format](https://www.sitemaps.org/protocol.html), sitemaps.org | Limits and hints in 7.5 |
| Heydon and Najork, "Mercator: A scalable, extensible Web crawler", World Wide Web 2(4):219-229, 1999, [doi:10.1023/A:1019213109274](https://doi.org/10.1023/A:1019213109274) | 7.1, 7.7 |
| Najork and Heydon, "High-Performance Web Crawling", Compaq SRC Research Report 173, 26 September 2001 | Frontier details, delay factor, DNS, 7.7 |
| Lee, Leonard, Wang and Loguinov, "IRLbot: Scaling to 6 billion pages and beyond", WWW 2008, [doi:10.1145/1367497.1367556](https://doi.org/10.1145/1367497.1367556); extended in ACM TWEB 3(3), 2009, [doi:10.1145/1541822.1541823](https://doi.org/10.1145/1541822.1541823) | DRUM, budgets, 7.6, 7.7 |
| Manku, Jain and Das Sarma, "Detecting near-duplicates for web crawling", WWW 2007, [doi:10.1145/1242572.1242592](https://doi.org/10.1145/1242572.1242592) | Simhash parameters in 7.4 |
| Bloom, "Space/time trade-offs in hash coding with allowable errors", CACM 13(7):422-426, 1970, [doi:10.1145/362686.362692](https://doi.org/10.1145/362686.362692) | Bloom filter in 3.5 |
| IIPC, [The WARC Format 1.1](https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/); ISO 28500:2017, [ISO catalogue](https://www.iso.org/standard/68004.html) | Storage format |
| Full-Text Search Engine System Design (`system-design/design-full-text-search`) | Indexer that consumes the output |
| Job Scheduler System Design (`system-design/design-job-scheduler`) | Recrawl scheduling patterns |
| Distributed Key-Value Store System Design (`system-design/design-distributed-kv-store`) | URL state store |
| Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`) | Method for section 3 |

## Standard and licence note

Based on the Robots Exclusion Protocol (RFC 9309, September 2022), HTTP
Semantics (RFC 9110), the sitemaps.org protocol and the published crawler
research cited in 12.3. RFC section numbers are cited and the rules are
paraphrased; no specification or paper text was copied, and all capacity
numbers are illustrative examples. ISO 28500:2017 (WARC) is a paid standard and
is named only; official text: ISO, purchase or licensed access at
https://www.iso.org/standard/68004.html (the IIPC publishes WARC 1.1
openly). Check the current versions before relying on them. This is not
legal advice; check the current official text and a qualified
professional. Original text written for this collection. Template
licence: MIT.