Distributed File System Design Template (GFS and HDFS Style)

GFS/HDFS-style file system: metadata and chunk servers, chunk sizing, leases and write pipeline, record append consistency, re-replication, erasure coding

SystemdesignVollständigGFS (Ghemawat et al., SOSP 2003); HDFS (Shvachko et al., MSST 2010)design distributed file system mathAktualisiert 29.10.2026

Im Editor öffnen.md herunterladen

Anleitung: Use it when designing or reviewing a cluster file system for large sequential data, or preparing the GFS interview question. Fill the file and chunk counts in section 3 first, because object count, not bytes, decides whether one metadata server is enough. The main pitfall is ignoring small files, which exhaust metadata memory long before disks fill.

Vorschau

System Design: [Distributed file system name]

Reference design for a GFS and HDFS style distributed file system: one replicated metadata service that owns the namespace and the map from files to large fixed-size chunks, many chunk servers that store the chunks on local disks, and a client library that talks to the metadata service for locations and streams data directly to and from chunk servers. It covers chunk size, the write pipeline with leases, record append, the consistency model, replica placement, re-replication, erasure coding for cold data, checksums and metadata high availability. It builds on "The Google File System" (Ghemawat, Gobioff and Leung, SOSP 2003) and "The Hadoop Distributed File System" (Shvachko et al., MSST 2010). It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks.

Document control

Field	Value
Document ID	[SDD-DFS-001]
System / feature	[Cluster file system for analytics and logs]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Owner	[@owner]
Author(s)	[@author]
Reviewers	[@reviewer-1], [@reviewer-2]
Last updated	YYYY-MM-DD
Classification	[Internal]
Difficulty	Advanced
Chunk size	[128 MiB]
Related	[Object storage design], [Distributed KV store design], [ADR: replication versus erasure coding]

Approvals

Role	Name	Decision	Date
Tech lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Staff / principal engineer	[@name]	[Decision]	YYYY-MM-DD
Storage operations	[@name]	[Decision]	YYYY-MM-DD
Data platform (main client)	[@name]	[Decision]	YYYY-MM-DD
Security	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft for design review
0.2	YYYY-MM-DD	[@name]	Example: Raised chunk size from 64 MiB to 128 MiB after metadata memory estimate
0.3	YYYY-MM-DD	[@name]	Example: Added erasure coding for data older than 30 days
1.0	YYYY-MM-DD	[@name]	Example: Approved

Overview
Requirements
Capacity estimation
API design
Data model
High-level design
Detailed design
Scalability and performance
Reliability, security and operations
Trade-offs and alternatives
Delivery plan and risks
Appendices

1. Overview

1.1 Problem statement

[Example: Batch analytics jobs read and write about 20 PB of logs and derived tables. Today the data sits on network-attached storage that cannot deliver the 2 Tbps of aggregate read bandwidth the nightly jobs need, and jobs cannot run on the machines where their data lives. Disk and machine failures are daily events at this size, so the storage layer must repair itself without operators.]

1.2 Proposed solution

[Example: Files are split into 128 MiB chunks, each stored as a local file on three chunk servers in at least two racks. A metadata service keeps the whole namespace and the chunk map in memory, persists every change to a replicated operation log, and hands out leases so that one replica orders the writes to each chunk. Clients ask the metadata service where chunks are, cache the answer and then stream data directly to chunk servers. Data older than 30 days is converted from three replicas to Reed-Solomon (6,3) erasure coding, which stores 1.5 bytes per byte instead of 3.]

1.3 Decisions requested from reviewers

ID	Question	Recommended option	Section
Q-01	Example: Chunk size?	128 MiB, with guidance to compact small files	3.3, 7.1
Q-02	Example: One metadata server or a sharded metadata layer?	One active plus hot standby now; namespace federation at about 1 billion objects	3.3, 7.6
Q-03	Example: Consistency for concurrent appends?	At-least-once record append with checksummed, self-identifying records	7.3
Q-04	Example: Replication or erasure coding?	3 replicas for hot data, RS(6,3) after 30 days	7.5
[Q-nn]	[Question]	[Option]	[Section]

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Minutes	Cover	Output on the whiteboard
0 to 5	Scope: large files, sequential reads, appends, batch clients; below-the-line items (2.3)	Workload profile and three requirements
5 to 10	Estimates: bytes, chunks, files, metadata memory, servers (section 3)	"Metadata fits in RAM on one machine, so one master"
10 to 15	Interface: open, read, append, the chunk location call (section 4)	Client calls and what each returns
15 to 25	High-level design: metadata service, chunk servers, client library (section 6)	Control path and data path drawn apart
25 to 40	One deep dive: the write path with leases and the consistency model (default, 7.2 and 7.3), or re-replication after failures (7.4), or metadata high availability (7.6)	Options, choice, reason
40 to 45	Failures (chunk server, disk, metadata server), the 10x row in section 10	Two failure rows

Start the default deep dive with the naive answer: the client writes to three replicas in parallel and waits for all. Then show the problems (concurrent writers apply mutations in different orders on different replicas) and introduce the lease: one replica, the primary, picks the order and the others follow it.

2. Requirements

2.1 Core functional requirements

ID	Requirement	Priority
FR-01	Example: Hierarchical namespace: create, delete, rename, list directories	Must
FR-02	Example: Large sequential reads and reads at an offset	Must
FR-03	Example: Append from many concurrent writers to one file	Must
FR-04	Example: Expose chunk locations so schedulers can run tasks next to the data	Must
FR-05	Example: Snapshots of a directory tree at low cost	Should
FR-06	Example: Per-directory quotas and replication or coding policy	Should
FR-07	Example: Random overwrite inside an existing file	Could
[FR-nn]	[Requirement]	[Must / Should / Could]

2.2 Non-functional requirements

ID	Quality	Target	Measured as
NFR-01	Durability	Example: No acknowledged data lost on loss of any one rack	Fault injection, block audit
NFR-02	Aggregate read throughput	Example: 2 Tbps sustained during the nightly batch window	Cluster metrics
NFR-03	Metadata latency	Example: p99 at or below 10 ms for open and chunk lookup	Client library timing
NFR-04	Repair time	Example: Lost replicas of one failed server re-created within 1 h; its erasure-coded cells within 2 h	Failure drill
NFR-05	Metadata availability	Example: Failover to standby within 60 s	Drill, quarterly
[NFR-nn]	[Quality]	[Target]	[Measurement]

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

[Example: full POSIX semantics, file locking and low-latency small random writes]
[Example: an S3-compatible object API (see the object storage design)]
[Example: geo-replication between regions (handled by a copy service at the dataset level)]
[Example: serving millions of tiny files to interactive users]
[Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

Input	Symbol	Value	Source
Logical data stored	$L$	Example: 20 PB	Data inventory plus 18 months of growth
Files	$F$	Example: 50 million (average 400 MB)	Namespace scan of current storage
Chunk size	$c$	Example: 128 MiB ( $134\,217\,728$ bytes)	HDFS default `dfs.blocksize`
Average chunk fill	$f$	Example: 0.75 (last chunk of each file is partial)	Namespace scan
Hot share kept as 3 replicas	$h$	Example: 0.3; the rest uses RS(6,3) at 1.5x	Access logs by age
Metadata memory per object (file or chunk)	$s$	Example: 150 bytes	Measure on your build; GFS reported under 64 bytes per chunk
Chunk server raw disk		Example: 12 disks x 20 TB = 240 TB, filled to 80%	Hardware standard
Chunk server network		Example: 25 Gbps	Hardware standard

3.2 Raw storage and servers

S_{raw} = L \times \left(h \times 3 + (1 - h) \times 1.5\right) = 20 \times (0.9 + 1.05) = 39 \ \text{PB}

N_{cs} = \frac{39 \ \text{PB}}{240 \ \text{TB} \times 0.8} = \frac{39\,000}{192} \approx 203.1 \Rightarrow 204 \ \text{chunk servers}

With about 8% spare capacity for failures and rebalancing, buy about 220 servers. Without erasure coding the same data would need 60 PB raw and about 313 servers, so coding cold data saves about a third of the hardware.

3.3 Metadata size

B = \frac{L}{f \times c} = \frac{20 \times 10^{15}}{0.75 \times 134\,217\,728} \approx 198.7 \ \text{million chunks}

M = (F + B) \times s = (50 + 198.7) \times 10^{6} \times 150 \ \text{B} \approx 37.3 \ \text{GB}

About 37 GB of metadata fits in the memory of one large server, so a single active metadata server with a hot standby is enough at this size. The chunk size is the main lever: at 64 MiB the chunk count doubles. Small files are the main threat: 200 million extra files of 1 MB each would add only 200 TB of data but about 400 million objects (one file and one chunk each), which is about 60 GB more metadata, more than the whole 20 PB of large files needs.

3.4 Throughput and repair

Read capacity: $204 \times 25 \ \text{Gbps} \approx 5$ Tbps of NICs against a 2 Tbps target, about 40% utilisation, which leaves room for repair traffic during the batch window.
Repair: a failed server held up to 192 TB, of which about 89 TB are replicas (18 of the 39 PB raw) and about 103 TB are erasure-coded cells. If 200 surviving servers each give 5 Gbps (20% of their NIC) to repair, the cluster moves 1 Tbps. Replicated chunks repair first, in about $88.6 \times 10^{12} \times 8 / 10^{12} \approx 709$ s, about 12 minutes. Each lost coded cell needs six source cells (7.5), so the coded share reads about 620 TB and finishes after about 95 minutes in total: inside NFR-04, and the reason coded stripes, which still tolerate two more losses, wait behind replicated chunks.
Heartbeats: about 220 servers every 3 s is about 73 messages/s, trivial for the metadata server; full chunk reports are spread over hours.
Checksums: 32-bit checksums per 64 KiB block (the GFS scheme) cost 8 KiB per 128 MiB chunk, about 12 GB per fully used server, held on the chunk server, not in the metadata service.

3.5 Summary

Quantity	Estimate	Design consequence
Raw storage	39 PB	Erasure coding for cold data saves about 21 PB
Chunk servers	204 needed, about 220 bought	Rack-aware placement across at least 10 racks
Chunks	About 199 million	128 MiB chunks keep the count manageable
Metadata	About 37 GB	One active metadata server plus standby; federation later
Repair	About 12 min for replicated chunks, about 95 min including coded cells	Repair replicated chunks first; throttle repair to 20% of NIC

4. API design

4.1 Client library interface

Applications link a client library; they never call chunk servers or the metadata service directly.

Call	Semantics
`create(path, policy)` / `delete(path)` / `rename(src, dst)`	Namespace changes, atomic at the metadata service
`open(path) -> handle`	Returns file length and the first chunk locations
`read(handle, offset, length)`	Library maps offset to chunk index, reads from the closest replica
`append(handle, record) -> offset`	Record append, at least once, offset chosen by the system
`write(handle, offset, data)`	Overwrite at an offset (FR-07, limited support)
`snapshot(path, name)`	Copy-on-write snapshot of a directory tree
`locations(path, range)`	Chunk replicas for job schedulers

4.2 Internal RPCs

RPC	From, to	Content
`GetChunkLocations(file, chunk_index)`	Client to metadata	Chunk handle, version, replica list, primary if leased
`AllocateChunk(file)`	Client to metadata	New chunk handle and three target servers
`GrantLease(chunk)`	Metadata to primary	Lease with 60 s expiry, renewed with heartbeats
`PushData(data_id, bytes)`	Client to nearest replica, then along the chain	Data only, buffered, not yet applied
`ApplyWrite(chunk, data_id, serial)`	Primary to secondaries	Apply buffered data in the primary's order
`Heartbeat(stats, lease_renewals)`	Chunk server to metadata	Load, free space, lease extensions
`ChunkReport(handles, versions)`	Chunk server to metadata	Full list, spread over time

{"chunkHandle": "c_0004f1a9", "version": 12, "primary": "cs-r07-14", "replicas": ["cs-r07-14", "cs-r11-03", "cs-r02-22"], "leaseExpiresInMs": 41000}

5. Data model

5.1 Entities

The namespace and the file-to-chunk map are persistent metadata; chunk locations are not persisted and are rebuilt from chunk server reports.

5.2 Storage choices

Data	Store	Why
Namespace and chunk map	In memory on the metadata server; operation log and checkpoints on local disk, replicated to standby and journal nodes	Every lookup is a memory read; the log gives durability and a total order
Chunk locations	In memory only, rebuilt from chunk reports	Chunk servers are the source of truth; nothing to keep in sync
Chunk data	Local files on chunk servers, one file per chunk	Sequential I/O on local disks, OS page cache for hot data
Checksums	Per 64 KiB block, kept on the chunk server with the chunk	Each replica verifies itself on every read

5.3 Access patterns

Access pattern	Frequency	Key or index that serves it	Partition key rationale
Resolve a path to its inode	Every open	In-memory tree keyed by parent inode and name	Whole namespace in one process, so no cross-node lookup
Map file offset to chunk replicas	Every read of a new chunk, cached by clients	Inode to ordered chunk list, chunk handle to replica set	Clients cache for minutes, so metadata load follows opens, not bytes
Read a byte range	Continuous, about 2 Tbps at peak	Chunk handle on a chunk server, 64 KiB checksum blocks	Data path never touches metadata
Find under-replicated chunks	Continuous background scan	Priority queue by missing replica count	Chunks with one replica left repair first
List a directory	Interactive and job planning	Children index on parent inode	Same in-memory tree
Garbage collect deleted files	Background, daily	Hidden deleted names with deletion time	Deletion is a rename; space reclaimed lazily

6. High-level design

The metadata service is on the control path only; data flows between clients and chunk servers.

Component	Responsibility	Scales by
Client library	Path to chunk mapping, location cache, read and write pipelines, retries	With applications
Metadata server	Namespace, chunk map, leases, placement, garbage collection	Memory and operations per second (one active)
Replicated log and standby	Durable operation log, hot standby replaying it	Fixed size
Chunk servers	Store chunks, verify checksums, serve reads, forward writes	Disks and NICs
Re-replication and balancer	Restore replica counts, move chunks to even disk usage, convert to erasure coding	Background bandwidth budget

7. Detailed design

The deep dives in this section, each with options, choice and reason:

Deep dive	Options considered	Choice	Reason	Section
Chunk size	4 MiB, 64 MiB, 128 MiB, 256 MiB	128 MiB	Metadata fits one server; reads stay sequential	7.1
Mutation ordering	Client fan-out, quorum writes, primary with lease	Primary with lease	One order per chunk without a consensus round per write	7.2
Concurrent appends	Client locking, single writer, record append	Record append, at least once	Many producers without coordination	7.3
Durability for cold data	3 replicas, erasure coding	RS(6,3) after 30 days	1.5x instead of 3x storage	7.5
Metadata availability	Single server with backups, hot standby on shared log, sharded metadata	Hot standby now, federation later	Simple until about 1 billion objects	7.6

7.1 Chunk size

Large chunks reduce metadata, client interactions with the metadata server and the number of TCP connections. GFS chose 64 MB; HDFS uses 128 MiB by default (dfs.blocksize). The cost is internal fragmentation for small files and hot spots when many clients read the same small file (one chunk on three servers). Mitigations: raise the replication factor for hot small files (executables, configuration), and run a compaction job that packs small files into container files.

7.2 Write path with leases

The metadata server grants a lease on a chunk to one replica, the primary, for 60 seconds, renewed while writes continue (GFS uses a 60 s initial timeout with extensions carried on heartbeats). Data flow and control flow are separated: the client pushes data along a chain of replicas ordered by network distance, and only then asks the primary to apply it.

If any secondary fails, the primary reports an error and the client retries; the region may then be inconsistent between replicas until the retry succeeds. Each lease grant raises the chunk version, so a replica that missed mutations while down has an older version, is detected as stale in its next chunk report and is garbage collected.

7.3 Record append and the consistency model

Many producers append to one log file without coordinating. The client sends only the record; the primary picks the offset at the current end of the chunk. If the record does not fit, the primary pads the chunk, all replicas do the same, and the client retries on the next chunk. Records are limited to one quarter of the chunk size so padding stays small.

Guarantees, stated so application developers can rely on them:

Operation	Guarantee	What readers may see
Namespace change	Atomic, totally ordered by the operation log	Never a partial rename
Successful serial write	Defined: all replicas hold the same bytes, readers see the write	Exactly the write
Concurrent successful writes	Consistent but undefined: same bytes everywhere, possibly mixed fragments	Interleaved data
Record append	Each record written atomically at least once at an offset the system chooses	Duplicates and padding between records
Failed write	Inconsistent region	Different data on different replicas

Applications therefore write self-identifying records (a unique ID per record) with checksums, skip padding and drop duplicates on read. This moves some work to clients in exchange for appends that scale with producers.

7.4 Replica placement and re-replication

Placement on create: first replica on the writer's server or rack, second and third on two servers in a different rack. Losing a whole rack never loses all replicas, and the write pipeline crosses racks only once.
Priority: chunks with one replica left repair before chunks with two; chunks blocking a running job are raised further.
Throttling: repair traffic per server is capped (5 Gbps in 3.4) so the batch window keeps its read bandwidth.
Detection: a server that misses heartbeats for long enough is declared dead. HDFS derives this timeout from the heartbeat interval (default 3 s) and dfs.namenode.heartbeat.recheck-interval (default 5 minutes) as two recheck intervals plus ten heartbeat intervals, 630 s (10.5 minutes) with defaults; check the source of your release. Too short, and a rolling restart starts petabytes of needless copying; too long, and durability drops during real failures.
Balancer: moves chunks from full to empty servers in the background after capacity is added.

7.5 Erasure coding for cold data

With Reed-Solomon (6,3) a stripe of six data cells gets three parity cells on nine servers in nine racks. Any three losses are tolerated, storage overhead is 1.5x instead of 3x, but a read of a lost cell needs six other cells (repair traffic is about six times the lost bytes) and small reads lose locality. The policy: write new data with three replicas, convert directories whose files are older than 30 days and unread for 7 days, and keep hot tables replicated.

7.6 Metadata durability and availability

Every namespace change is written to the operation log and replicated before the client is answered. The in-memory state is checkpointed periodically so recovery replays only the log tail.
A hot standby replays the same log and holds the same in-memory state; failover is a fencing step (the old active can no longer write to the log) followed by promotion, within 60 s (NFR-05). Chunk locations are rebuilt from chunk reports, which the standby also receives.
Read-only shadow replicas can serve listings and locations for jobs that tolerate slightly stale metadata.
Beyond about 1 billion objects (about 150 GB of metadata at 150 bytes each) split the namespace by top-level directory across several metadata servers (federation), or move metadata into a sharded database, as Colossus did (7.7).

7.7 How real systems do it

How real systems do it (paraphrased, not copied). The GFS paper (Ghemawat, Gobioff and Leung, SOSP 2003) describes a single master holding all metadata in memory, 64 MB chunks stored as Linux files with three replicas by default, leases with a 60-second initial timeout so one primary orders mutations, data pushed along a chain of chunk servers separately from control messages, an operation log with checkpoints, shadow masters for read-only access, and 32-bit checksums on 64 KB blocks. It also explains the relaxed consistency model and record append with at-least-once semantics. HDFS (Shvachko et al., MSST 2010) follows the same split between a NameNode and DataNodes. Google's 2021 post on Colossus, the successor to GFS, describes moving metadata into Bigtable behind a scalable layer of "curators", which let clusters grow well beyond the largest GFS clusters, with background "custodians" handling durability and rebalancing. Read the originals in 12.3; this box keeps only the general idea.

8. Scalability and performance

Concern	Approach	Limit before redesign
Metadata memory	Large chunks, small-file compaction, quotas on object counts	Example: About 1 billion objects on one server
Metadata operations	Client location caching, batched chunk lookups, shadow servers for reads	[Operations/s measured on target hardware]
Aggregate bandwidth	Add chunk servers; data path never through metadata	Network fabric bisection bandwidth
Hot small files	Higher replication for those paths	Tens of replicas
Repair storms	Throttled, prioritised re-replication	Rack or power domain loss

Read path budget for the first byte of a new chunk (target p99 30 ms):

Step	Budget
Location lookup (cache miss)	10 ms
Connect to nearest replica	2 ms
Disk seek and first 64 KiB with checksum check	15 ms
Headroom	3 ms

9. Reliability, security and operations

9.1 Failure modes

Component	Failure	Detection	Blast radius	Mitigation	RTO
Disk	Disk failure or silent corruption	I/O errors, checksum mismatch on read or scrub	Chunks on that disk; reads go to other replicas	Mark replica bad, re-replicate from healthy copy	Example: Under 30 min
Chunk server	Host loss	Missed heartbeats	About 1.5% of replicated chunks (3 of 204 servers) lose one replica; about 4.4% of coded stripes (9 of 204) lose one cell	Prioritised repair (3.4)	Example: About 12 min for replicated chunks, about 95 min for coded stripes
Rack	Power or switch loss	Many servers missing at once	One replica of many chunks	Rack-aware placement keeps another copy; delay repair a few minutes in case of a short blip	Example: Data served at once; redundancy within hours
Metadata server	Crash	Lost leadership heartbeat	All namespace operations; reads with cached locations continue	Fenced failover to standby	Example: 60 s
Operation log	Log replica loss	Journal health checks	None while a quorum remains	Replace node, re-sync	Example: 1 h
Client	Partial write then crash	Lease expires	One chunk region inconsistent	Readers skip by record checksum; lease recovery closes the file	Example: 60 s
[Component]	[Failure]	[Signal]	[Scope]	[Mitigation]	[Time]

9.2 Security

Threat	Control
Unauthorised reads	Permissions on paths at the metadata server; short-lived block access tokens checked by chunk servers
Impersonation of services	Mutual TLS or Kerberos between clients, metadata and chunk servers
Data exposure on decommissioned disks	Encryption at rest per encryption zone, keys from the KMS; crypto-erase
Accidental deletion	Deletes become hidden renames kept for 3 days; snapshots on critical paths
[Threat]	[Control]

9.3 Observability

Signal	Metric or log	Alert threshold
Durability risk	Chunks with one replica, missing chunks	Any missing chunk; one-replica count rising for 10 min
Metadata health	Heap use, operation latency, log sync time	Heap above 75%, p99 above 10 ms
Capacity	Used share per server and cluster	Cluster above 80%
Corruption	Checksum failures per disk	More than a few per day on one disk
Repair	Bytes re-replicated per hour, queue length	Queue not shrinking for 1 h

9.4 Cost estimate

Item	Unit cost	Quantity	Monthly cost
Chunk servers	[$/server-month]	Example: About 220	[$]
Metadata servers and journal	[$/server-month]	Example: 2 large-memory plus 3 journal	[$]
Network fabric	[$/port-month]	[Ports]	[$]
Total			[$]

10. Trade-offs and alternatives

Decision	Options	Chosen	Why	What we give up
Metadata	Single in-memory server; sharded metadata on a database	Single active plus standby	Simple, fast, consistent namespace	A hard ceiling on object count
Chunk size	Small; 64 MiB; 128 MiB	128 MiB	Fewer objects and lookups	Small-file waste, hot spots
Consistency	Strong POSIX semantics; relaxed with record append	Relaxed	Appends scale with producers	Applications handle duplicates and padding
Durability	3 replicas; erasure coding; mixed	Mixed by age	39 PB raw instead of 60 PB	Slower degraded reads and costlier repair for cold data
Build or adopt	Build; run an open-source HDFS distribution; use cloud object storage	[Choice]	[Reason]	[Cost]
What changes at 10x / 0.1x	10x (200 PB, about 2.5 billion objects, about 373 GB of metadata): one metadata server no longer fits, so federate the namespace or move metadata to a sharded store, and code more data. 0.1x (2 PB): about 21 chunk servers; consider cloud object storage with a table format instead of running a file system	Review at each doubling of objects	Object count, not bytes, breaks first	[Note]
[Decision]	[Options]	[Chosen]	[Why]	[Cost]

11. Delivery plan and risks

11.1 Milestones

Milestone	Scope	Exit criterion	Target date
M1	Metadata server, chunk servers, client library, three replicas	1 PB test cluster passes fault injection	YYYY-MM-DD
M2	Hot standby and fenced failover	Failover within 60 s in 10 consecutive drills	YYYY-MM-DD
M3	Migrate first batch pipelines	Nightly jobs meet their window for 30 days	YYYY-MM-DD
M4	Erasure coding for cold data	20 PB logical in under 40 PB raw	YYYY-MM-DD

11.2 Rollout and rollback

[Example: Pipelines move one at a time with dual writes to old and new storage for 14 days and daily checksum comparison. Rollback points the pipeline back to the old storage, which stays authoritative until the comparison has been clean for 14 days.]

11.3 Risks

ID	Risk	Likelihood	Impact	Mitigation	Owner
R-01	Example: Small files exhaust metadata memory	High	High	Object quotas per team; compaction job; alerts at 75% heap	[@name]
R-02	Example: Correlated failures (one power domain) exceed placement assumptions	Medium	High	Placement across power domains as well as racks	[@name]
R-03	Example: Applications assume POSIX semantics	Medium	Medium	Client guide on record format and duplicates	[@name]
[R-nn]	[Risk]	[L / M / H]	[L / M / H]	[Mitigation]	[@name]

12. Appendices

12.1 Open questions

ID	Question	Owner	Due
OQ-01	Example: Do we need cross-region copies for any dataset, and at what lag?	[@name]	YYYY-MM-DD
OQ-02	Example: Which erasure code width suits our rack count (RS(6,3) needs 9 racks)?	[@name]	YYYY-MM-DD
[OQ-nn]	[Question]	[@name]	YYYY-MM-DD

12.2 Glossary

Term	Meaning
Chunk (block)	Fixed-size piece of a file, the unit of placement and replication
Metadata server (master, NameNode)	Owner of the namespace and the chunk map
Chunk server (DataNode)	Stores chunks on local disks
Lease	Time-limited authority for one replica to order writes to a chunk
Record append	Append where the system chooses the offset; at least once
Defined region	All replicas hold the same bytes and they are one complete write
Erasure coding	Parity cells that let lost data be recomputed with less overhead than copies
[Term]	[Meaning]

12.3 References

Reference	Use
Ghemawat, Gobioff and Leung, "The Google File System", SOSP 2003, pp. 29 to 43, doi:10.1145/945445.945450	7.2, 7.3, 7.7
Shvachko, Kuang, Radia and Chansler, "The Hadoop Distributed File System", MSST 2010, doi:10.1109/MSST.2010.5496972	NameNode and DataNode design
Apache Hadoop, `hdfs-default.xml` (stable release)	Defaults: 128 MiB block, replication 3, 3 s heartbeat, 5 min recheck
Hildebrand and Serenyi, "Colossus under the hood: a peek into Google's scalable storage system", Google Cloud blog, 20 April 2021	7.6, 7.7
Distributed Key-Value Store Design (`system-design/design-distributed-kv-store`)	Replication and placement ideas
Distributed Lock and Coordination Service Design (`system-design/design-distributed-lock-coordination`)	Leader election and fencing for 7.6
Object Storage Design (`system-design/design-object-storage`)	The alternative for small files and HTTP access
Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`)	Method for section 3

Standard and licence note

Based on the published designs of the Google File System (SOSP 2003) and the Hadoop Distributed File System (MSST 2010), the Apache Hadoop configuration defaults and the Google Cloud post on Colossus, all cited in 12.3. These are paraphrased in original words; no text, figures or tables were copied, and all numbers marked Example are illustrative. Configuration defaults change between releases; check the documentation of the version you run. Original text written for this collection. Template licence: MIT.