# System Design: [Distributed file system name]

> Reference design for a GFS and HDFS style distributed file system: one
> replicated metadata service that owns the namespace and the map from files
> to large fixed-size chunks, many chunk servers that store the chunks on
> local disks, and a client library that talks to the metadata service for
> locations and streams data directly to and from chunk servers. It covers
> chunk size, the write pipeline with leases, record append, the consistency
> model, replica placement, re-replication, erasure coding for cold data,
> checksums and metadata high availability. It builds on "The Google File
> System" (Ghemawat, Gobioff and Leung, SOSP 2003) and "The Hadoop
> Distributed File System" (Shvachko et al., MSST 2010). It follows the
> structure of the System Design Document template, adds a 45-minute
> interview path, access-pattern and failure-mode tables and a cited "How
> real systems do it" box, and is pre-filled with a worked design to adapt.
> Replace every `[bracketed]` value and every row marked *Example*; values
> marked *Example* are assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Start from the workload: large sequential reads and appends by batch jobs
  suit this design; many small random writes or POSIX semantics do not
  (section 2.3 and 10).
- Fill the object counts in 3.1 before anything else. The number of files
  and chunks, not the number of bytes, decides whether one metadata server
  is enough.
- Draw the control path (client to metadata service) and the data path
  (client to chunk servers) as separate arrows; the whole design depends on
  keeping data off the metadata server.
- Rehearse a chunk server loss and a metadata failover before launch.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-DFS-001] |
| System / feature | [Cluster file system for analytics and logs] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| Chunk size | [128 MiB] |
| Related | [Object storage design], [Distributed KV store design], [ADR: replication versus erasure coding] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Storage operations | [@name] | [Decision] | YYYY-MM-DD |
| Data platform (main client) | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Raised chunk size from 64 MiB to 128 MiB after metadata memory estimate |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added erasure coding for data older than 30 days |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Batch analytics jobs read and write about 20 PB of logs and
derived tables. Today the data sits on network-attached storage that cannot
deliver the 2 Tbps of aggregate read bandwidth the nightly jobs need, and
jobs cannot run on the machines where their data lives. Disk and machine
failures are daily events at this size, so the storage layer must repair
itself without operators.]

### 1.2 Proposed solution

[*Example:* Files are split into 128 MiB chunks, each stored as a local file
on three chunk servers in at least two racks. A metadata service keeps the
whole namespace and the chunk map in memory, persists every change to a
replicated operation log, and hands out leases so that one replica orders
the writes to each chunk. Clients ask the metadata service where chunks are,
cache the answer and then stream data directly to chunk servers. Data older
than 30 days is converted from three replicas to Reed-Solomon (6,3) erasure
coding, which stores 1.5 bytes per byte instead of 3.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Chunk size? | 128 MiB, with guidance to compact small files | 3.3, 7.1 |
| Q-02 | *Example:* One metadata server or a sharded metadata layer? | One active plus hot standby now; namespace federation at about 1 billion objects | 3.3, 7.6 |
| Q-03 | *Example:* Consistency for concurrent appends? | At-least-once record append with checksummed, self-identifying records | 7.3 |
| Q-04 | *Example:* Replication or erasure coding? | 3 replicas for hot data, RS(6,3) after 30 days | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: large files, sequential reads, appends, batch clients; below-the-line items (2.3) | Workload profile and three requirements |
| 5 to 10 | Estimates: bytes, chunks, files, metadata memory, servers (section 3) | "Metadata fits in RAM on one machine, so one master" |
| 10 to 15 | Interface: open, read, append, the chunk location call (section 4) | Client calls and what each returns |
| 15 to 25 | High-level design: metadata service, chunk servers, client library (section 6) | Control path and data path drawn apart |
| 25 to 40 | One deep dive: the write path with leases and the consistency model (default, 7.2 and 7.3), or re-replication after failures (7.4), or metadata high availability (7.6) | Options, choice, reason |
| 40 to 45 | Failures (chunk server, disk, metadata server), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer: the client writes to
three replicas in parallel and waits for all. Then show the problems
(concurrent writers apply mutations in different orders on different
replicas) and introduce the lease: one replica, the primary, picks the order
and the others follow it.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Hierarchical namespace: create, delete, rename, list directories | Must |
| FR-02 | *Example:* Large sequential reads and reads at an offset | Must |
| FR-03 | *Example:* Append from many concurrent writers to one file | Must |
| FR-04 | *Example:* Expose chunk locations so schedulers can run tasks next to the data | Must |
| FR-05 | *Example:* Snapshots of a directory tree at low cost | Should |
| FR-06 | *Example:* Per-directory quotas and replication or coding policy | Should |
| FR-07 | *Example:* Random overwrite inside an existing file | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Durability | *Example:* No acknowledged data lost on loss of any one rack | Fault injection, block audit |
| NFR-02 | Aggregate read throughput | *Example:* 2 Tbps sustained during the nightly batch window | Cluster metrics |
| NFR-03 | Metadata latency | *Example:* p99 at or below 10 ms for open and chunk lookup | Client library timing |
| NFR-04 | Repair time | *Example:* Lost replicas of one failed server re-created within 1 h; its erasure-coded cells within 2 h | Failure drill |
| NFR-05 | Metadata availability | *Example:* Failover to standby within 60 s | Drill, quarterly |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* full POSIX semantics, file locking and low-latency small random writes]
- [*Example:* an S3-compatible object API (see the object storage design)]
- [*Example:* geo-replication between regions (handled by a copy service at the dataset level)]
- [*Example:* serving millions of tiny files to interactive users]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Logical data stored | $L$ | *Example:* 20 PB | Data inventory plus 18 months of growth |
| Files | $F$ | *Example:* 50 million (average 400 MB) | Namespace scan of current storage |
| Chunk size | $c$ | *Example:* 128 MiB ($134\,217\,728$ bytes) | HDFS default `dfs.blocksize` |
| Average chunk fill | $f$ | *Example:* 0.75 (last chunk of each file is partial) | Namespace scan |
| Hot share kept as 3 replicas | $h$ | *Example:* 0.3; the rest uses RS(6,3) at 1.5x | Access logs by age |
| Metadata memory per object (file or chunk) | $s$ | *Example:* 150 bytes | Measure on your build; GFS reported under 64 bytes per chunk |
| Chunk server raw disk | | *Example:* 12 disks x 20 TB = 240 TB, filled to 80% | Hardware standard |
| Chunk server network | | *Example:* 25 Gbps | Hardware standard |

### 3.2 Raw storage and servers

$$
S_{raw} = L \times \left(h \times 3 + (1 - h) \times 1.5\right) = 20 \times (0.9 + 1.05) = 39 \ \text{PB}
$$

$$
N_{cs} = \frac{39 \ \text{PB}}{240 \ \text{TB} \times 0.8} = \frac{39\,000}{192} \approx 203.1 \Rightarrow 204 \ \text{chunk servers}
$$

With about 8% spare capacity for failures and rebalancing, buy about 220
servers. Without erasure coding the same data would need 60 PB raw and about
313 servers, so coding cold data saves about a third of the hardware.

### 3.3 Metadata size

$$
B = \frac{L}{f \times c} = \frac{20 \times 10^{15}}{0.75 \times 134\,217\,728} \approx 198.7 \ \text{million chunks}
$$

$$
M = (F + B) \times s = (50 + 198.7) \times 10^{6} \times 150 \ \text{B} \approx 37.3 \ \text{GB}
$$

About 37 GB of metadata fits in the memory of one large server, so a single
active metadata server with a hot standby is enough at this size. The chunk
size is the main lever: at 64 MiB the chunk count doubles. Small files are the
main threat: 200 million extra files of 1 MB each would add only 200 TB of
data but about 400 million objects (one file and one chunk each), which is
about 60 GB more metadata, more than the whole 20 PB of large files needs.

### 3.4 Throughput and repair

- Read capacity: $204 \times 25 \ \text{Gbps} \approx 5$ Tbps of NICs against
  a 2 Tbps target, about 40% utilisation, which leaves room for repair
  traffic during the batch window.
- Repair: a failed server held up to 192 TB, of which about 89 TB are
  replicas (18 of the 39 PB raw) and about 103 TB are erasure-coded cells.
  If 200 surviving servers each give 5 Gbps (20% of their NIC) to repair,
  the cluster moves 1 Tbps. Replicated chunks repair first, in about
  $88.6 \times 10^{12} \times 8 / 10^{12} \approx 709$ s, about 12 minutes.
  Each lost coded cell needs six source cells (7.5), so the coded share
  reads about 620 TB and finishes after about 95 minutes in total: inside
  NFR-04, and the reason coded stripes, which still tolerate two more
  losses, wait behind replicated chunks.
- Heartbeats: about 220 servers every 3 s is about 73 messages/s, trivial
  for the metadata server; full chunk reports are spread over hours.
- Checksums: 32-bit checksums per 64 KiB block (the GFS scheme) cost 8 KiB
  per 128 MiB chunk, about 12 GB per fully used server, held on the chunk
  server, not in the metadata service.

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Raw storage | 39 PB | Erasure coding for cold data saves about 21 PB |
| Chunk servers | 204 needed, about 220 bought | Rack-aware placement across at least 10 racks |
| Chunks | About 199 million | 128 MiB chunks keep the count manageable |
| Metadata | About 37 GB | One active metadata server plus standby; federation later |
| Repair | About 12 min for replicated chunks, about 95 min including coded cells | Repair replicated chunks first; throttle repair to 20% of NIC |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Client library interface

Applications link a client library; they never call chunk servers or the
metadata service directly.

| Call | Semantics |
|---|---|
| `create(path, policy)` / `delete(path)` / `rename(src, dst)` | Namespace changes, atomic at the metadata service |
| `open(path) -> handle` | Returns file length and the first chunk locations |
| `read(handle, offset, length)` | Library maps offset to chunk index, reads from the closest replica |
| `append(handle, record) -> offset` | Record append, at least once, offset chosen by the system |
| `write(handle, offset, data)` | Overwrite at an offset (FR-07, limited support) |
| `snapshot(path, name)` | Copy-on-write snapshot of a directory tree |
| `locations(path, range)` | Chunk replicas for job schedulers |

### 4.2 Internal RPCs

| RPC | From, to | Content |
|---|---|---|
| `GetChunkLocations(file, chunk_index)` | Client to metadata | Chunk handle, version, replica list, primary if leased |
| `AllocateChunk(file)` | Client to metadata | New chunk handle and three target servers |
| `GrantLease(chunk)` | Metadata to primary | Lease with 60 s expiry, renewed with heartbeats |
| `PushData(data_id, bytes)` | Client to nearest replica, then along the chain | Data only, buffered, not yet applied |
| `ApplyWrite(chunk, data_id, serial)` | Primary to secondaries | Apply buffered data in the primary's order |
| `Heartbeat(stats, lease_renewals)` | Chunk server to metadata | Load, free space, lease extensions |
| `ChunkReport(handles, versions)` | Chunk server to metadata | Full list, spread over time |

```json
{"chunkHandle": "c_0004f1a9", "version": 12, "primary": "cs-r07-14", "replicas": ["cs-r07-14", "cs-r11-03", "cs-r02-22"], "leaseExpiresInMs": 41000}
```

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

The namespace and the file-to-chunk map are persistent metadata; chunk
locations are not persisted and are rebuilt from chunk server reports.

```mermaid
erDiagram
  INODE {
    bigint inode_id PK
    bigint parent_id FK
    string name
    string type
    string policy
    bigint length
    datetime mtime
  }
  CHUNK {
    bigint chunk_handle PK
    bigint inode_id FK
    int chunk_index
    int version
    string coding
  }
  REPLICA {
    bigint chunk_handle FK
    string server_id FK
    int version
  }
  CHUNK_SERVER {
    string server_id PK
    string rack
    bigint free_bytes
    datetime last_heartbeat
  }
  INODE ||--o{ INODE : contains
  INODE ||--o{ CHUNK : "split into"
  CHUNK ||--|{ REPLICA : "stored as"
  CHUNK_SERVER ||--o{ REPLICA : holds
```

### 5.2 Storage choices

| Data | Store | Why |
|---|---|---|
| Namespace and chunk map | In memory on the metadata server; operation log and checkpoints on local disk, replicated to standby and journal nodes | Every lookup is a memory read; the log gives durability and a total order |
| Chunk locations | In memory only, rebuilt from chunk reports | Chunk servers are the source of truth; nothing to keep in sync |
| Chunk data | Local files on chunk servers, one file per chunk | Sequential I/O on local disks, OS page cache for hot data |
| Checksums | Per 64 KiB block, kept on the chunk server with the chunk | Each replica verifies itself on every read |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Resolve a path to its inode | Every open | In-memory tree keyed by parent inode and name | Whole namespace in one process, so no cross-node lookup |
| Map file offset to chunk replicas | Every read of a new chunk, cached by clients | Inode to ordered chunk list, chunk handle to replica set | Clients cache for minutes, so metadata load follows opens, not bytes |
| Read a byte range | Continuous, about 2 Tbps at peak | Chunk handle on a chunk server, 64 KiB checksum blocks | Data path never touches metadata |
| Find under-replicated chunks | Continuous background scan | Priority queue by missing replica count | Chunks with one replica left repair first |
| List a directory | Interactive and job planning | Children index on parent inode | Same in-memory tree |
| Garbage collect deleted files | Background, daily | Hidden deleted names with deletion time | Deletion is a rename; space reclaimed lazily |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

The metadata service is on the control path only; data flows between clients
and chunk servers.

```mermaid
flowchart LR
  app[Application] --> lib[Client library]
  lib -->|"control: path, chunk index"| meta[Metadata server, active]
  meta -->|"handle, version, replicas"| lib
  meta -->|operation log| journal[(Replicated log)]
  journal --> standby[Metadata server, standby]
  lib -->|"data: read or push"| cs1[Chunk server rack A]
  cs1 -->|pipeline| cs2[Chunk server rack B]
  cs2 -->|pipeline| cs3[Chunk server rack B]
  cs1 -->|heartbeat, chunk report| meta
  cs2 -->|heartbeat, chunk report| meta
  meta --> repl[Re-replication and balancer]
  repl -->|copy instructions| cs3
  sched[Job scheduler] -->|locations| meta
```

| Component | Responsibility | Scales by |
|---|---|---|
| Client library | Path to chunk mapping, location cache, read and write pipelines, retries | With applications |
| Metadata server | Namespace, chunk map, leases, placement, garbage collection | Memory and operations per second (one active) |
| Replicated log and standby | Durable operation log, hot standby replaying it | Fixed size |
| Chunk servers | Store chunks, verify checksums, serve reads, forward writes | Disks and NICs |
| Re-replication and balancer | Restore replica counts, move chunks to even disk usage, convert to erasure coding | Background bandwidth budget |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Chunk size | 4 MiB, 64 MiB, 128 MiB, 256 MiB | 128 MiB | Metadata fits one server; reads stay sequential | 7.1 |
| Mutation ordering | Client fan-out, quorum writes, primary with lease | Primary with lease | One order per chunk without a consensus round per write | 7.2 |
| Concurrent appends | Client locking, single writer, record append | Record append, at least once | Many producers without coordination | 7.3 |
| Durability for cold data | 3 replicas, erasure coding | RS(6,3) after 30 days | 1.5x instead of 3x storage | 7.5 |
| Metadata availability | Single server with backups, hot standby on shared log, sharded metadata | Hot standby now, federation later | Simple until about 1 billion objects | 7.6 |

### 7.1 Chunk size

Large chunks reduce metadata, client interactions with the metadata server
and the number of TCP connections. GFS chose 64 MB; HDFS uses 128 MiB by
default (`dfs.blocksize`). The cost is internal fragmentation for small files
and hot spots when many clients read the same small file (one chunk on three
servers). Mitigations: raise the replication factor for hot small files
(executables, configuration), and run a compaction job that packs small
files into container files.

### 7.2 Write path with leases

The metadata server grants a lease on a chunk to one replica, the primary,
for 60 seconds, renewed while writes continue (GFS uses a 60 s initial
timeout with extensions carried on heartbeats). Data flow and control flow
are separated: the client pushes data along a chain of replicas ordered by
network distance, and only then asks the primary to apply it.

```mermaid
sequenceDiagram
  participant C as Client
  participant M as Metadata server
  participant P as Primary replica
  participant S1 as Secondary 1
  participant S2 as Secondary 2
  C->>M: which replicas hold chunk 7, who is primary?
  M-->>C: primary P, secondaries S1 and S2, version 12
  C->>P: push data (buffered)
  P->>S1: forward data
  S1->>S2: forward data
  S2-->>C: data received
  C->>P: write request, data id
  P->>P: assign serial number, apply
  P->>S1: apply in serial order
  P->>S2: apply in serial order
  S1-->>P: done
  S2-->>P: done
  P-->>C: success
```

If any secondary fails, the primary reports an error and the client retries;
the region may then be inconsistent between replicas until the retry
succeeds. Each lease grant raises the chunk version, so a replica that missed
mutations while down has an older version, is detected as stale in its next
chunk report and is garbage collected.

### 7.3 Record append and the consistency model

Many producers append to one log file without coordinating. The client sends
only the record; the primary picks the offset at the current end of the
chunk. If the record does not fit, the primary pads the chunk, all replicas
do the same, and the client retries on the next chunk. Records are limited to
one quarter of the chunk size so padding stays small.

Guarantees, stated so application developers can rely on them:

| Operation | Guarantee | What readers may see |
|---|---|---|
| Namespace change | Atomic, totally ordered by the operation log | Never a partial rename |
| Successful serial write | Defined: all replicas hold the same bytes, readers see the write | Exactly the write |
| Concurrent successful writes | Consistent but undefined: same bytes everywhere, possibly mixed fragments | Interleaved data |
| Record append | Each record written atomically at least once at an offset the system chooses | Duplicates and padding between records |
| Failed write | Inconsistent region | Different data on different replicas |

Applications therefore write self-identifying records (a unique ID per
record) with checksums, skip padding and drop duplicates on read. This moves
some work to clients in exchange for appends that scale with producers.

### 7.4 Replica placement and re-replication

- **Placement on create**: first replica on the writer's server or rack,
  second and third on two servers in a different rack. Losing a whole rack
  never loses all replicas, and the write pipeline crosses racks only once.
- **Priority**: chunks with one replica left repair before chunks with two;
  chunks blocking a running job are raised further.
- **Throttling**: repair traffic per server is capped (5 Gbps in 3.4) so
  the batch window keeps its read bandwidth.
- **Detection**: a server that misses heartbeats for long enough is declared
  dead. HDFS derives this timeout from the heartbeat interval (default 3 s)
  and `dfs.namenode.heartbeat.recheck-interval` (default 5 minutes) as two
  recheck intervals plus ten heartbeat intervals, 630 s (10.5 minutes) with
  defaults; check the source of your release. Too short, and a rolling restart starts petabytes of needless
  copying; too long, and durability drops during real failures.
- **Balancer**: moves chunks from full to empty servers in the background
  after capacity is added.

### 7.5 Erasure coding for cold data

With Reed-Solomon (6,3) a stripe of six data cells gets three parity cells
on nine servers in nine racks. Any three losses are tolerated, storage
overhead is 1.5x instead of 3x, but a read of a lost cell needs six other
cells (repair traffic is about six times the lost bytes) and small reads lose
locality. The policy: write new data with three replicas, convert directories
whose files are older than 30 days and unread for 7 days, and keep hot tables
replicated.

### 7.6 Metadata durability and availability

- Every namespace change is written to the operation log and replicated
  before the client is answered. The in-memory state is checkpointed
  periodically so recovery replays only the log tail.
- A hot standby replays the same log and holds the same in-memory state;
  failover is a fencing step (the old active can no longer write to the log)
  followed by promotion, within 60 s (NFR-05). Chunk locations are rebuilt
  from chunk reports, which the standby also receives.
- Read-only shadow replicas can serve listings and locations for jobs that
  tolerate slightly stale metadata.
- Beyond about 1 billion objects (about 150 GB of metadata at 150 bytes each)
  split the namespace by top-level directory across several metadata
  servers (federation), or move metadata into a sharded database, as
  Colossus did (7.7).

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). The GFS paper
> (Ghemawat, Gobioff and Leung, SOSP 2003) describes a single master holding
> all metadata in memory, 64 MB chunks stored as Linux files with three
> replicas by default, leases with a 60-second initial timeout so one primary
> orders mutations, data pushed along a chain of chunk servers separately
> from control messages, an operation log with checkpoints, shadow masters for
> read-only access, and 32-bit checksums on 64 KB blocks. It also explains
> the relaxed consistency model and record append with at-least-once
> semantics. HDFS (Shvachko et al., MSST 2010) follows the same split between
> a NameNode and DataNodes. Google's 2021 post on Colossus, the successor to
> GFS, describes moving metadata into Bigtable behind a scalable layer of
> "curators", which let clusters grow well beyond the largest GFS clusters,
> with background "custodians" handling durability and rebalancing. Read the
> originals in 12.3; this box keeps only the general idea.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Metadata memory | Large chunks, small-file compaction, quotas on object counts | *Example:* About 1 billion objects on one server |
| Metadata operations | Client location caching, batched chunk lookups, shadow servers for reads | [Operations/s measured on target hardware] |
| Aggregate bandwidth | Add chunk servers; data path never through metadata | Network fabric bisection bandwidth |
| Hot small files | Higher replication for those paths | Tens of replicas |
| Repair storms | Throttled, prioritised re-replication | Rack or power domain loss |

Read path budget for the first byte of a new chunk (target p99 30 ms):

| Step | Budget |
|---|---|
| Location lookup (cache miss) | 10 ms |
| Connect to nearest replica | 2 ms |
| Disk seek and first 64 KiB with checksum check | 15 ms |
| Headroom | 3 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Disk | Disk failure or silent corruption | I/O errors, checksum mismatch on read or scrub | Chunks on that disk; reads go to other replicas | Mark replica bad, re-replicate from healthy copy | *Example:* Under 30 min |
| Chunk server | Host loss | Missed heartbeats | About 1.5% of replicated chunks (3 of 204 servers) lose one replica; about 4.4% of coded stripes (9 of 204) lose one cell | Prioritised repair (3.4) | *Example:* About 12 min for replicated chunks, about 95 min for coded stripes |
| Rack | Power or switch loss | Many servers missing at once | One replica of many chunks | Rack-aware placement keeps another copy; delay repair a few minutes in case of a short blip | *Example:* Data served at once; redundancy within hours |
| Metadata server | Crash | Lost leadership heartbeat | All namespace operations; reads with cached locations continue | Fenced failover to standby | *Example:* 60 s |
| Operation log | Log replica loss | Journal health checks | None while a quorum remains | Replace node, re-sync | *Example:* 1 h |
| Client | Partial write then crash | Lease expires | One chunk region inconsistent | Readers skip by record checksum; lease recovery closes the file | *Example:* 60 s |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

| Threat | Control |
|---|---|
| Unauthorised reads | Permissions on paths at the metadata server; short-lived block access tokens checked by chunk servers |
| Impersonation of services | Mutual TLS or Kerberos between clients, metadata and chunk servers |
| Data exposure on decommissioned disks | Encryption at rest per encryption zone, keys from the KMS; crypto-erase |
| Accidental deletion | Deletes become hidden renames kept for 3 days; snapshots on critical paths |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Durability risk | Chunks with one replica, missing chunks | Any missing chunk; one-replica count rising for 10 min |
| Metadata health | Heap use, operation latency, log sync time | Heap above 75%, p99 above 10 ms |
| Capacity | Used share per server and cluster | Cluster above 80% |
| Corruption | Checksum failures per disk | More than a few per day on one disk |
| Repair | Bytes re-replicated per hour, queue length | Queue not shrinking for 1 h |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Chunk servers | [$/server-month] | *Example:* About 220 | [$] |
| Metadata servers and journal | [$/server-month] | *Example:* 2 large-memory plus 3 journal | [$] |
| Network fabric | [$/port-month] | [Ports] | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Metadata | Single in-memory server; sharded metadata on a database | Single active plus standby | Simple, fast, consistent namespace | A hard ceiling on object count |
| Chunk size | Small; 64 MiB; 128 MiB | 128 MiB | Fewer objects and lookups | Small-file waste, hot spots |
| Consistency | Strong POSIX semantics; relaxed with record append | Relaxed | Appends scale with producers | Applications handle duplicates and padding |
| Durability | 3 replicas; erasure coding; mixed | Mixed by age | 39 PB raw instead of 60 PB | Slower degraded reads and costlier repair for cold data |
| Build or adopt | Build; run an open-source HDFS distribution; use cloud object storage | [Choice] | [Reason] | [Cost] |
| What changes at 10x / 0.1x | 10x (200 PB, about 2.5 billion objects, about 373 GB of metadata): one metadata server no longer fits, so federate the namespace or move metadata to a sharded store, and code more data. 0.1x (2 PB): about 21 chunk servers; consider cloud object storage with a table format instead of running a file system | Review at each doubling of objects | Object count, not bytes, breaks first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Metadata server, chunk servers, client library, three replicas | 1 PB test cluster passes fault injection | YYYY-MM-DD |
| M2 | Hot standby and fenced failover | Failover within 60 s in 10 consecutive drills | YYYY-MM-DD |
| M3 | Migrate first batch pipelines | Nightly jobs meet their window for 30 days | YYYY-MM-DD |
| M4 | Erasure coding for cold data | 20 PB logical in under 40 PB raw | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Pipelines move one at a time with dual writes to old and new
storage for 14 days and daily checksum comparison. Rollback points the
pipeline back to the old storage, which stays authoritative until the
comparison has been clean for 14 days.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Small files exhaust metadata memory | High | High | Object quotas per team; compaction job; alerts at 75% heap | [@name] |
| R-02 | *Example:* Correlated failures (one power domain) exceed placement assumptions | Medium | High | Placement across power domains as well as racks | [@name] |
| R-03 | *Example:* Applications assume POSIX semantics | Medium | Medium | Client guide on record format and duplicates | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we need cross-region copies for any dataset, and at what lag? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which erasure code width suits our rack count (RS(6,3) needs 9 racks)? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Chunk (block) | Fixed-size piece of a file, the unit of placement and replication |
| Metadata server (master, NameNode) | Owner of the namespace and the chunk map |
| Chunk server (DataNode) | Stores chunks on local disks |
| Lease | Time-limited authority for one replica to order writes to a chunk |
| Record append | Append where the system chooses the offset; at least once |
| Defined region | All replicas hold the same bytes and they are one complete write |
| Erasure coding | Parity cells that let lost data be recomputed with less overhead than copies |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| Ghemawat, Gobioff and Leung, "The Google File System", SOSP 2003, pp. 29 to 43, [doi:10.1145/945445.945450](https://doi.org/10.1145/945445.945450) | 7.2, 7.3, 7.7 |
| Shvachko, Kuang, Radia and Chansler, "The Hadoop Distributed File System", MSST 2010, [doi:10.1109/MSST.2010.5496972](https://doi.org/10.1109/MSST.2010.5496972) | NameNode and DataNode design |
| Apache Hadoop, [`hdfs-default.xml`](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml) (stable release) | Defaults: 128 MiB block, replication 3, 3 s heartbeat, 5 min recheck |
| Hildebrand and Serenyi, ["Colossus under the hood: a peek into Google's scalable storage system"](https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system), Google Cloud blog, 20 April 2021 | 7.6, 7.7 |
| Distributed Key-Value Store Design (`system-design/design-distributed-kv-store`) | Replication and placement ideas |
| Distributed Lock and Coordination Service Design (`system-design/design-distributed-lock-coordination`) | Leader election and fencing for 7.6 |
| Object Storage Design (`system-design/design-object-storage`) | The alternative for small files and HTTP access |
| Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`) | Method for section 3 |

## Standard and licence note

Based on the published designs of the Google File System (SOSP 2003) and the
Hadoop Distributed File System (MSST 2010), the Apache Hadoop configuration
defaults and the Google Cloud post on Colossus, all cited in 12.3. These are
paraphrased in original words; no text, figures or tables were copied, and
all numbers marked *Example* are illustrative. Configuration defaults change
between releases; check the documentation of the version you run. Original
text written for this collection. Template licence: MIT.