# System Design: [Object storage service name]

> Reference design for an S3-compatible object (blob) store: a strongly
> consistent metadata service, data placement across failure domains,
> packing of small objects into large volumes, replication for new data and
> erasure coding for data that has cooled, durability arithmetic, repair and
> deletion. It draws on the published Haystack (OSDI 2010) and f4 (OSDI 2014)
> papers and on the public Amazon S3 documentation for API behaviour. It
> follows the structure of the System Design Document template, adds a
> 45-minute interview path, access-pattern and failure-mode tables and a cited
> "How real systems do it" box, and is pre-filled with a worked design to
> adapt. Replace every `[bracketed]` value and every row marked *Example*;
> values marked *Example* are assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Start from the object size distribution, not the total bytes. Small
  objects stress metadata and disk seeks; large objects stress bandwidth.
- Write the durability target as a number and show the model behind it
  (7.3). Say which failures the model ignores, because those are the ones
  that lose data in practice.
- Decide the consistency contract for PUT, GET, LIST and DELETE before
  choosing the metadata store (7.1).
- Treat deletion as a feature with its own design: versioning, retention
  locks, garbage collection and crypto-shredding (7.6).
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-OBJ-001] |
| System / feature | [Object storage service] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| API compatibility target | [S3 REST API subset, listed in 4.1] |
| Related | [Backup policy], [Data retention schedule], [ADR: erasure code parameters] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| Hardware and capacity planning | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Changed erasure code from RS(10,4) in one zone to RS(9,6) across three zones |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added crypto-shredding for deletes in erasure-coded volumes |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Product teams store user uploads, backups and analytics files on
a network file system that holds 3 copies of everything. It costs three times
the logical bytes, listing a directory with millions of files takes minutes,
and two disk-shelf failures in one rack last year made 4 TB unavailable for
a day. The company needs an internal object store with an S3-compatible API,
lower storage overhead and a stated durability target.]

### 1.2 Proposed solution

[*Example:* A regional service across three availability zones (AZs).
Stateless front ends speak the S3 REST API. A metadata service, sharded by
bucket and key range on consensus-replicated groups, maps each object version
to its data location and gives strong read-after-write consistency. New
objects are written as three replicas, one per AZ, into large append-only
volumes; small objects are packed many to a volume so a read costs one disk
seek. After 30 days without change, volumes are converted to a Reed-Solomon
RS(9,6) code with five fragments per AZ, which survives the loss of a whole
AZ plus one more disk at 1.67 times the logical size. Background scrubbers
verify checksums and a repair service rebuilds lost fragments.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Replication or erasure coding? | Both: 3 replicas while hot, RS(9,6) across 3 AZs after 30 days | 7.3, 10 |
| Q-02 | *Example:* Consistency for GET and LIST after PUT? | Strong read-after-write for object operations; bucket configuration eventually consistent | 7.1 |
| Q-03 | *Example:* How are small objects stored? | Packed into 100 GB append-only volumes with an in-memory index | 7.2 |
| Q-04 | *Example:* How do deletes reclaim space in coded volumes? | Per-object key deleted at once (crypto-shredding); space reclaimed by periodic rewrite | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: PUT, GET, DELETE, LIST, multipart; consistency; durability target; below-the-line items (2.3) | Five operations and two numbers |
| 5 to 10 | Estimates: objects, bytes, request rates, raw capacity under replication and erasure coding (section 3) | "Metadata is 100 TB, data is 87 PB raw" |
| 10 to 15 | Interface: PUT and GET with conditional headers, multipart upload (section 4) | Three calls |
| 15 to 25 | High-level design: front end, metadata service, storage nodes, placement, repair (section 6) | One diagram |
| 25 to 40 | One deep dive: durability and erasure coding (default, 7.3), or metadata and LIST (7.1), or small-object packing (7.2), or deletes (7.6) | Options, choice, reason |
| 40 to 45 | Failures (disk, AZ, metadata shard, silent corruption), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer, three full copies, and
work out its cost (150 PB raw for 50 PB of data). Then introduce a
Reed-Solomon code, show why the fragments must be spread so one AZ holds no
more fragments than the code can lose, and only then compute durability.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* PUT, GET (with byte ranges), HEAD, DELETE objects in buckets | Must |
| FR-02 | *Example:* Multipart upload for large objects, with parts uploaded in parallel and resumed | Must |
| FR-03 | *Example:* LIST objects by prefix with pagination, in lexicographic key order | Must |
| FR-04 | *Example:* Conditional writes (`If-None-Match: *` to create only, `If-Match` on ETag) | Must |
| FR-05 | *Example:* Object versioning per bucket and retention locks (write once, read many) | Must |
| FR-06 | *Example:* Lifecycle rules: expire objects and versions after a set age | Should |
| FR-07 | *Example:* Asynchronous replication of a bucket to another region | Should |
| FR-08 | *Example:* Event notifications on create and delete | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Durability | *Example:* Design target of at least 11 nines per object per year, under the model in 7.3 | Model plus scrubber and repair metrics |
| NFR-02 | Availability | *Example:* 99.99% of GET and PUT requests succeed per month; survives loss of one AZ | Front end success ratio, AZ evacuation drill |
| NFR-03 | Consistency | *Example:* A GET or LIST started after a successful PUT or DELETE sees its effect | Linearizability tests on the metadata service |
| NFR-04 | Latency | *Example:* First byte p99 at or below 100 ms for objects under 1 MiB | Front end timing |
| NFR-05 | Cost | *Example:* Raw-to-logical ratio at or below 1.8 averaged over the fleet | Capacity reports |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* a POSIX file system interface or in-place partial updates of objects]
- [*Example:* content delivery to end users (a CDN sits in front)]
- [*Example:* tabular or vector bucket types and query in place]
- [*Example:* cross-region strong consistency (replication is asynchronous)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. Every number should end in a consequence. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Logical data stored | $L$ | *Example:* 50 PB | Current file systems x growth |
| Objects stored | $N$ | *Example:* 200 billion | Inventory of current files |
| Mean object size | $\bar{s} = L / N$ | 250 KB | Derived; median is far smaller |
| Peak PUT rate | $W$ | *Example:* 5 000/s (average 2 000/s) | Upload logs |
| Peak GET rate | $G$ | *Example:* 100 000/s | Access logs |
| Share of bytes in the replicated (hot) tier | $h$ | *Example:* 5% | Objects younger than 30 days |
| Drive size and target fill | $D$, $f$ | *Example:* 20 TB, 80% | Hardware plan |
| Metadata per object version | $m$ | *Example:* 500 bytes | Key, version, size, checksum, location, tags |
| Annual failure rate of a drive | AFR | *Example:* 2% | Fleet statistics |
| Time to detect and repair a lost fragment | $T$ | *Example:* 24 h | Repair service target |

### 3.2 Bandwidth

$$
B_{put} = W \times \bar{s} = 5\,000 \times 250 \ \text{KB} = 1.25 \ \text{GB/s}, \qquad
B_{get} = G \times \bar{s} = 100\,000 \times 250 \ \text{KB} = 25 \ \text{GB/s} \approx 200 \ \text{Gbit/s}
$$

Three replicas on write mean 3.75 GB/s of disk writes at peak, and the later
conversion to RS(9,6) writes another $1.25 \times 15/9 \approx 2.08$ GB/s on
average when it catches up with peak intake. At the average PUT rate the gross
intake is about 1.3 PB a month before deletes. Consequence: reads, not writes,
size the network; the front end tier is sized on 200 Gbit/s egress and a cache
absorbs the hottest objects.

### 3.3 Raw capacity

| Scheme | Overhead (raw / logical) | Raw for 50 PB | Survives |
|---|---|---|---|
| 3 replicas, one per AZ | 3.0 | 150 PB | Loss of one AZ plus one more copy |
| RS(10,4) inside one AZ | 1.4 | 70 PB | Four lost fragments, but not the loss of that AZ |
| RS(6,3) across 3 AZs (3 per AZ) | 1.5 | 75 PB | One AZ, and nothing more while it is down |
| RS(9,6) across 3 AZs (5 per AZ) | 1.67 | 83.3 PB | One AZ plus one more fragment |
| **Chosen blend: 5% at 3.0, 95% at RS(9,6)** | **1.73** | **86.7 PB** | As above |

$$
\text{drives} = \frac{86.7 \ \text{PB}}{20 \ \text{TB} \times 0.8} \approx 5\,417, \qquad
\text{servers} = \frac{5\,417}{24} \approx 226 \ (\text{about } 75 \text{ per AZ})
$$

Consequence: erasure coding saves about 63 PB of raw disk against three
replicas, which pays for the extra CPU and repair traffic many times over.

### 3.4 Metadata

$$
M = N \times m = 2 \times 10^{11} \times 500 \ \text{B} = 100 \ \text{TB}, \quad 300 \ \text{TB with three replicas}
$$

At 2 TB per metadata shard that is 50 shards, each a three-member consensus
group. Every GET needs one metadata lookup (100 000/s at peak) unless the
front end caches the location of immutable versions. Consequence: metadata
does not fit in memory on one machine and must be range-partitioned from day
one.

### 3.5 Disk operations

- Average GET load per drive: $100\,000 / 5\,417 \approx 18$ reads/s, well
  below the roughly 100 random reads a second an HDD sustains, but only if a
  GET costs one seek (7.2).
- Objects per drive: $2 \times 10^{11} / 5\,417 \approx 3.7 \times 10^{7}$.
  With a 32-byte in-memory index entry per object that is 1.2 GB of index per
  drive, about 28 GB per 24-drive server. Consequence: the storage-node index
  fits in memory, so no disk read is spent on finding the object.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Egress | 25 GB/s (200 Gbit/s) peak | Front ends sized on bandwidth; hot-object cache |
| Raw capacity | 86.7 PB (5 417 drives, about 226 servers) | RS(9,6) for cold data, 3 replicas for 30 days |
| Metadata | 100 TB per copy | 50 consensus-replicated range shards |
| Seeks per GET | 1 | Packed volumes and in-memory volume index |

## 4. API design

<!-- Guidance: Define the contract first: operations, request and response shapes, errors, idempotency, pagination and limits. Use RFC 9457 problem details for errors where the S3 XML error format is not required. -->

### 4.1 Interfaces

| Operation | Request | Notes |
|---|---|---|
| Put object | `PUT /{bucket}/{key}` with `Content-MD5` or a checksum header | Single request up to *Example:* 5 GiB; returns ETag and version ID |
| Get object | `GET /{bucket}/{key}?versionId=` with optional `Range` | Range reads touch only the needed fragments |
| Head object | `HEAD /{bucket}/{key}` | Metadata only |
| Delete object | `DELETE /{bucket}/{key}` | Adds a delete marker in versioned buckets |
| List objects | `GET /{bucket}?list-type=2&prefix=&delimiter=&continuation-token=` | Up to 1 000 keys per page, lexicographic order |
| Multipart upload | `POST ?uploads`, `PUT ?partNumber=&uploadId=`, `POST ?uploadId=` (complete), `DELETE ?uploadId=` (abort) | Parts uploaded in parallel and retried independently |
| Conditional create | `PUT` with `If-None-Match: *` | Fails with 412 if the key exists; makes "create once" safe |

For reference, the public Amazon S3 documentation (checked October 2026)
lists multipart parts of 5 MiB to 5 GiB, at most 10 000 parts and a maximum
object size of 48.8 TiB; choose your own limits and publish them, and check
the provider page if you need exact compatibility.

### 4.2 Multipart flow

```mermaid
sequenceDiagram
  participant C as Client
  participant F as Front end
  participant M as Metadata service
  participant S as Storage nodes
  C->>F: POST /photos/raw/v1.mov?uploads
  F->>M: create upload record
  M-->>F: uploadId
  F-->>C: uploadId
  loop each part, in parallel
    C->>F: PUT part n (8 MiB, checksum)
    F->>S: write part to 3 nodes in 3 AZs
    S-->>F: durable
    F->>M: record part n location and checksum
    F-->>C: 200, part ETag
  end
  C->>F: POST complete (list of part numbers and ETags)
  F->>M: atomically publish version pointing at parts
  M-->>F: version ID
  F-->>C: 200, ETag of the whole object
```

Parts become visible only when the completion commits the version in one
metadata transaction, so readers never see half an object. Incomplete
uploads expire by lifecycle rule (*Example:* 7 days) and their parts are
garbage-collected.

### 4.3 Error example

A conditional create that loses the race gets 412 and must not retry the
same create; the client reads the existing version instead.

```http
HTTP/1.1 412 Precondition Failed
Content-Type: application/problem+json

{"type": "https://storage.example.com/problems/precondition-failed", "title": "Object already exists", "status": 412, "detail": "If-None-Match: * was set and the key reports/2026/q3.parquet already has a current version.", "instance": "/analytics/reports/2026/q3.parquet"}
```

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

```mermaid
erDiagram
  BUCKET {
    string bucket_name PK
    string owner
    bool versioning
    string region
  }
  OBJECT_VERSION {
    string bucket_name PK
    string object_key PK
    string version_id PK
    bigint size
    string checksum
    bool delete_marker
    datetime created_at
  }
  EXTENT {
    string version_id FK
    int part_no
    string volume_id FK
    bigint offset
    bigint length
  }
  VOLUME {
    string volume_id PK
    string state
    string scheme
    int fragment_count
  }
  FRAGMENT {
    string volume_id FK
    int fragment_no
    string node_id
    string disk_id
  }
  BUCKET ||--o{ OBJECT_VERSION : holds
  OBJECT_VERSION ||--o{ EXTENT : "stored as"
  VOLUME ||--o{ EXTENT : contains
  VOLUME ||--o{ FRAGMENT : "placed as"
```

A volume is an append-only container (*Example:* 100 GB). While open it is
written as three replicas; once sealed and cold it is encoded as nine data
and six parity fragments. Objects point at volumes, not disks, so moving or
re-encoding a volume never touches object metadata.

### 5.2 Storage choices

| Data | Store | Key | Main queries | Why |
|---|---|---|---|---|
| Object versions and extents | *Example:* Range-partitioned key-value store on consensus groups | `(bucket, key, inverted version)` | Point lookup, prefix scan for LIST | Strong consistency and ordered scans |
| Volume and fragment map | Separate placement service, replicated | `volume_id` | Where are the fragments of volume V | Small (millions of volumes), changes on repair |
| Volume index on storage nodes | Memory, rebuilt from an index file on disk | `(volume_id, object offset)` | Offset and length of an object in a volume | One seek per read |
| Object bytes | Storage nodes, raw disks or large files | Volume, offset | Sequential append, positioned read | No file-per-object metadata overhead |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| GET latest version of a key | 100 000/s peak | Metadata point lookup on `(bucket, key)`, newest version first | Range partitions keep keys in order; hot ranges split automatically |
| PUT a new version | 5 000/s peak | Insert `(bucket, key, version)` after data is durable | Same partition as reads of that key, so read-after-write is local to one group |
| LIST by prefix | Thousands/s | Ordered scan of `(bucket, key)` from the prefix | Hash partitioning would scatter a prefix; range partitioning keeps it on one or few shards |
| Find fragments of a volume | Every GET miss in cache, every repair | Placement map by `volume_id` | Small map, cached in front ends |
| Expire objects by lifecycle | Daily batch | Per-bucket scan by creation time index | Batch work, never on the request path |
| Repair a lost disk | Rare, large | Placement map reverse index `disk_id` to fragments | Lets repair start in seconds after a failure |

Sequential key names (timestamps as prefixes) concentrate writes on the last
range. The metadata service splits a range when its write rate stays above a
threshold, and documentation tells clients that prefixes spread load.

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  client([Client or SDK]) --> lb[Load balancer]
  lb --> fe["Front end: auth, S3 API, checksums"]
  fe --> meta[("Metadata service: range shards on consensus groups")]
  fe --> place[(Placement service)]
  fe --> cache[Hot object cache]
  subgraph az1["AZ 1"]
    sn1[Storage nodes]
  end
  subgraph az2["AZ 2"]
    sn2[Storage nodes]
  end
  subgraph az3["AZ 3"]
    sn3[Storage nodes]
  end
  fe --> sn1
  fe --> sn2
  fe --> sn3
  enc["Encoder: replicas to RS(9,6)"] --> place
  enc --> sn1
  enc --> sn2
  enc --> sn3
  scrub[Scrubber] --> sn1
  repair[Repair service] --> place
  gc[Garbage collector and lifecycle] --> meta
```

| Component | Responsibility | Scales by |
|---|---|---|
| Front end | Authenticate, sign-check, stream data, compute checksums, call metadata | Bandwidth and connections |
| Metadata service | Object versions, extents, LIST order, conditional writes | Shards (key ranges) |
| Placement service | Map volumes to fragments and disks; choose disks for new volumes | Volumes (small) |
| Storage nodes | Append to open volumes, serve positioned reads, keep volume index in memory | Disks |
| Encoder | Convert sealed cold volumes from 3 replicas to RS(9,6) | Background throughput |
| Scrubber | Read every fragment periodically and verify checksums | Disk read budget |
| Repair service | Rebuild missing or corrupt fragments on other disks | Repair bandwidth |
| Garbage collector and lifecycle | Expire versions, delete keys, reclaim volume space | Batch |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Metadata and consistency | Eventually consistent hash-partitioned store, relational database, range-partitioned consensus groups | Range-partitioned consensus groups | Strong read-after-write and ordered LIST | 7.1 |
| Small objects | File per object, packed volumes | Packed append-only volumes | One seek per read, no per-file metadata | 7.2 |
| Redundancy | Replicas only, erasure code only, replicas then code | 3 replicas for 30 days, then RS(9,6) across AZs | Cheap writes and reads while hot, 1.67x when cold | 7.3 |
| Deletes | Rewrite volumes on every delete, tombstones and compaction, per-object keys | Per-object keys plus periodic rewrite | Immediate logical delete in coded volumes | 7.6 |

### 7.1 Metadata service and consistency

- Keys are ordered by `(bucket, key)` and split into ranges. Each range is a
  consensus group of three members in three AZs; a write commits when two
  members accept it.
- A PUT writes the data first, then commits the version in metadata. Only
  the commit makes the object visible, so a crash between the two leaves
  orphan data for garbage collection, never a visible broken object.
- Reads of the latest version go to the range leader (or to a follower with
  a read lease), so a GET after a successful PUT always sees it. LIST scans
  the same ordered ranges, so it also sees new keys at once.
- Two concurrent PUTs to the same key both succeed; the later commit becomes
  the latest version. `If-Match` and `If-None-Match` let clients avoid lost
  updates. There are no transactions across keys.
- Bucket configuration (policies, lifecycle, versioning state) is cached in
  front ends and is eventually consistent for a short interval; document it.

### 7.2 Packing small objects into volumes

A file per object costs inodes, directory lookups and extra seeks. Instead,
each storage node appends objects ("needles") to a large volume file and
keeps an in-memory map from object ID to offset and length. A GET is then
one metadata lookup (often cached) plus one disk read. Each needle carries a
header with its ID, size and checksum, so the in-memory index can be rebuilt
by scanning the volume, and a periodic index checkpoint keeps restarts fast.
Objects larger than *Example:* 16 MiB skip packing: they are split into
8 MiB extents and erasure-coded directly, because a large object gains
nothing from replication-first writes.

### 7.3 Redundancy and durability

**Placement.** For RS(9,6) the 15 fragments of a volume go to 15 different
servers in 15 different racks, five per AZ. Losing an AZ removes five
fragments; nine of the remaining ten are enough to read, so data stays
available and one more failure is still tolerated.

**Durability model.** With independent drive failures, the probability that
one drive holding a fragment fails during a repair window of $T$ hours is

$$
p = \text{AFR} \times \frac{T}{8\,760} = 0.02 \times \frac{24}{8\,760} \approx 5.48 \times 10^{-5}
$$

A stripe of $n$ fragments that tolerates $n-k$ losses is lost when more than
$n-k$ fragments fail within the same window:

$$
P_{window} = \sum_{j=n-k+1}^{n} \binom{n}{j} p^{j} (1-p)^{n-j}, \qquad
P_{year} \approx P_{window} \times \frac{8\,760}{T}
$$

| Scheme | $P_{window}$ | $P_{year}$ per stripe | Nines |
|---|---|---|---|
| 3 replicas | $1.6 \times 10^{-13}$ | $6.0 \times 10^{-11}$ | 10.2 |
| RS(6,3) | $1.1 \times 10^{-15}$ | $4.1 \times 10^{-13}$ | 12.4 |
| RS(10,4) | $9.9 \times 10^{-19}$ | $3.6 \times 10^{-16}$ | 15.4 |
| RS(9,6) | $9.5 \times 10^{-27}$ | $3.5 \times 10^{-24}$ | 23.5 |

Read this table with care. The model assumes independent failures and a
fixed repair window; real losses come from correlated events the model
ignores: a power or cooling failure in a rack, a bad firmware batch, a bug
in the encoder, an operator deleting the wrong volume. The design therefore
spreads fragments across racks and AZs, mixes drive batches, verifies every
encoded volume by decoding it before the replicas are dropped, and keeps
versioning and retention locks as the defence against logical deletion.
The extra margin of RS(9,6) also matters while an AZ is down: with five
fragments already gone, a remaining stripe is lost only if two of the other
ten fail in a window, about $1.35 \times 10^{-7}$ per 24-hour window, which is
why AZ outages trigger priority repair.

**Repair speed.** Rebuilding a 20 TB drive (16 TB used) onto one spare at
150 MB/s takes about 29.6 hours; spreading the rebuild across 100 drives at
50 MB/s each takes about 0.9 hours. Distributed repair is what keeps $T$
small. Rebuilding one RS(9,6) fragment reads nine fragments, so repair
traffic is nine times the lost bytes and must be throttled against client
traffic.

### 7.4 Write and read paths

```mermaid
sequenceDiagram
  participant C as Client
  participant F as Front end
  participant P as Placement
  participant S1 as Node AZ1
  participant S2 as Node AZ2
  participant S3 as Node AZ3
  participant M as Metadata
  C->>F: PUT /media/u/42/avatar.jpg (180 KB, checksum)
  F->>P: open volume for this size class
  P-->>F: volume v-9137 on S1, S2, S3
  par three replicas
    F->>S1: append needle
    F->>S2: append needle
    F->>S3: append needle
  end
  S1-->>F: synced, offset
  S2-->>F: synced, offset
  S3-->>F: synced, offset
  F->>M: commit version (bucket, key, v-9137, offset, length)
  M-->>F: committed, version ID
  F-->>C: 200, ETag, version ID
```

The front end acknowledges only after all three replicas have synced and the
metadata commit succeeded. If one replica fails, the front end writes the
object to a different open volume rather than accepting two copies. A GET
reads one replica (nearest AZ first) or, for coded volumes, the one data
fragment that holds the object; it falls back to a decode from any nine
fragments when that fragment is missing, and verifies the checksum before
returning bytes.

### 7.5 Scrubbing and silent corruption

Every fragment is read and checksummed at least once every *Example:* 30
days. A mismatch marks the fragment bad and queues a repair from the other
fragments. Front ends verify checksums on every read as well, so a corrupt
fragment is never served even before the scrubber finds it.

### 7.6 Deletion, versioning and space reclamation

- A DELETE in a versioned bucket adds a delete marker; older versions stay
  until lifecycle rules expire them. Retention locks block deletion and
  overwrite until a date, including for administrators in compliance mode.
- In replicated volumes, deleted needles are marked in the index and the
  volume is compacted by copying live needles to a new volume.
- In coded volumes, each object is encrypted with its own data key stored in
  metadata. Deleting the key makes the bytes unreadable at once
  (crypto-shredding). Space is reclaimed later by rewriting volumes whose
  live share has fallen below *Example:* 60%.

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). The Haystack paper
> (Beaver, Kumar, Li, Sobel and Vajgel, OSDI 2010) describes the photo store
> of a large social network that, at the time, served over a million images
> a second at peak. The authors found that a network file system spent disk
> operations on file metadata, so Haystack packs photos into large volume
> files (about 100 GB each) and keeps a compact in-memory map from photo to
> offset, using on average about 10 bytes of memory per photo, so a read
> needs at most one disk operation. The f4 paper (Muralidhar and co-authors,
> OSDI 2014) adds a warm tier: once data is older and read less, volumes move
> from Haystack's replicated layout (effective replication 3.6, three copies
> on RAID-6) to Reed-Solomon(10,4) inside a cell, with blocks of a stripe on
> different racks, and an XOR of two volumes in a third region for disaster
> tolerance, which brings the effective replication factor to 2.1. f4
> handles deletes by removing a per-object encryption key. This design keeps
> the same ideas: packed volumes (7.2), hot then cold redundancy (7.3) and
> key deletion (7.6). Read the originals in 12.3.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| GET rate | One seek per read, location cache in front ends, hot-object cache | Seek budget of about 100/s per HDD |
| Metadata rate | Range shards split on load | *Example:* about 20 000 writes/s per shard before a split |
| Sequential key hot spot | Automatic range split, guidance on prefixes | One key range accepting all new keys |
| Large objects | Direct erasure coding in 8 MiB extents, parallel parts | Front end bandwidth |
| Repair traffic | Distributed rebuild, throttled | Nine bytes read per byte rebuilt for RS(9,6) |
| Capacity growth | Add servers per AZ in balanced groups; placement prefers empty disks | Uneven AZ capacity breaks five-per-AZ placement |

Latency budget for a GET of a 200 KB object (target p99 100 ms to first byte):

| Step | Budget |
|---|---|
| Authentication and request parsing | 2 ms |
| Metadata lookup (cache miss) | 10 ms |
| Placement lookup (cached) | 1 ms |
| Storage node seek and read | 40 ms |
| Checksum and first byte to client | 5 ms |
| Headroom | 42 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Disk | Drive failure | I/O errors, missed heartbeats | Fragments on that drive; no data unavailable | Distributed repair from other fragments | *Example:* about 1 h of repair |
| Server or rack | Power or switch failure | Heartbeats per rack | One fragment per stripe (one rack per fragment) | Placement rule, repair after a grace period of 15 min to avoid needless rebuilds | *Example:* 15 min plus repair |
| AZ | Whole AZ lost | Multiple rack alarms, provider status | Reads decode from 10 of 15 fragments; writes go to two AZs | Priority repair; new hot writes use two AZs plus a later third copy | *Example:* reads continue; full redundancy restored in days |
| Metadata shard | Leader loss | Consensus heartbeats | Keys in one range for a few seconds | Leader election within the group | *Example:* 5 s |
| Encoder | Bug writes bad parity | Decode verification before replicas are dropped | Volumes encoded since the bug | Keep replicas until a full decode check passes | *Example:* none (replicas kept) |
| Fragment | Silent corruption | Read-path checksums, scrubber | One fragment | Repair from other fragments | *Example:* minutes |
| Operator | Wrong bucket or volume deleted | Audit log, delete-rate alarm | Data under that bucket | Versioning, retention locks, delayed physical deletion (7 days) | *Example:* hours, by undelete |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

| Threat | Control |
|---|---|
| Public exposure of private buckets | Buckets private by default; account-level block on public access |
| Stolen credentials | Short-lived signed requests, scoped keys, anomaly alerts on egress |
| Data at rest on stolen or returned disks | Per-object data keys wrapped by a key management service; crypto-shredding on decommission |
| Ransomware or malicious deletion | Versioning, retention locks, separate admin roles, delayed physical deletion |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Durability risk | Stripes with fewer than full fragments, by fragments missing | Any stripe missing 4 or more |
| Repair backlog | Bytes waiting for repair, oldest item age | Oldest above 24 h |
| Scrub coverage | Share of fragments verified in 30 days | Below 100% |
| Request health | Error and latency by operation and AZ | p99 first byte above 100 ms for 10 min |
| Capacity | Fill per AZ and per disk class | Above 80% |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Storage servers | [$/server-month] | *Example:* about 226 servers with 24 x 20 TB drives | [$] |
| Metadata servers | [$/server-month] | *Example:* 150 (50 shards x 3) | [$] |
| Front ends and caches | [$/server-month] | Sized on 200 Gbit/s | [$] |
| Network between AZs | [$/GB] | Replica writes plus repair traffic | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Redundancy | 3 replicas; RS(10,4) in one AZ; RS(9,6) across AZs | Replicas while hot, RS(9,6) when cold | AZ tolerance at 1.67x; fast hot reads | Encoding work, repair reads nine fragments |
| Metadata store | Eventually consistent hash store; consensus range shards | Consensus range shards | Read-after-write and ordered LIST | Cross-AZ latency on every write |
| Small objects | File per object; packed volumes | Packed volumes | One seek per read | Compaction and index rebuild logic |
| Deletes in coded volumes | Rewrite at once; per-object keys | Per-object keys | Immediate, cheap logical delete | Key management on the read path |
| Cross-region | Synchronous; asynchronous per bucket | Asynchronous, opt-in | No cross-region latency on PUT | Recovery point equals replication lag |
| What changes at 10x / 0.1x | 10x (500 PB, 2 trillion objects): about 54 000 drives and 1 PB of metadata per copy, so split into independent cells per region with buckets assigned to cells, and consider a wider code such as RS(12,6) to cut overhead to 1.5. 0.1x (5 PB): three replicas everywhere is simpler and the extra disk is affordable; erasure coding can wait | Review at each doubling of logical bytes | Metadata shards and repair bandwidth grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Front end, metadata, replicated volumes, PUT, GET, LIST | API conformance tests pass; consistency tests pass | YYYY-MM-DD |
| M2 | Multipart, versioning, lifecycle | One pilot team migrated | YYYY-MM-DD |
| M3 | Encoder, scrubber, distributed repair | AZ evacuation drill with no data loss | YYYY-MM-DD |
| M4 | Retention locks, crypto-shredding, cross-region replication | Security review signed off | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Teams copy data in with checksums verified on both sides and keep
the old file system read-only for 60 days. The encoder starts with
verification only (encode, decode, compare, keep replicas) for 30 days before
any replica is dropped. Rollback of the encoder means keeping replicas, which
costs capacity but no data.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Correlated drive failures from one firmware batch | Medium | High | Mix batches per stripe; firmware updates in waves | [@name] |
| R-02 | *Example:* Hot key ranges overload one metadata shard | Medium | Medium | Load-based splits; prefix guidance | [@name] |
| R-03 | *Example:* Repair traffic hurts client latency after an AZ loss | Medium | Medium | Repair throttles by priority; durability-first override | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we need a locally repairable code to cut repair reads below nine fragments? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which buckets need cross-region replication and in which regions (data residency)? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Volume | Large append-only container that holds many objects |
| Needle | One object inside a volume, with its own header and checksum |
| Stripe | The k data and n minus k parity fragments computed together |
| RS(k, m) in this document | Reed-Solomon code with k data and m parity fragments, written RS(9,6) for 9 plus 6 |
| Effective replication factor | Raw bytes stored divided by logical bytes |
| Scrubbing | Reading stored data to detect corruption before a client does |
| Crypto-shredding | Deleting data by destroying the key that encrypts it |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| Beaver, Kumar, Li, Sobel, Vajgel, ["Finding a needle in Haystack: Facebook's photo storage"](https://www.usenix.org/conference/osdi10/finding-needle-haystack-facebooks-photo-storage), OSDI 2010 | 7.2, 7.7 |
| Muralidhar and co-authors, ["f4: Facebook's Warm BLOB Storage System"](https://www.usenix.org/conference/osdi14/technical-sessions/presentation/muralidhar), OSDI 2014, pages 383 to 398 | 7.3, 7.6, 7.7 |
| [Amazon S3 data consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel), AWS documentation, checked October 2026 | Behaviour reference for 7.1 |
| [Amazon S3 multipart upload limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html), AWS documentation, checked October 2026 | Limits reference for 4.1 |
| RFC 9457 Problem Details for HTTP APIs | Error body (4.3) |
| Distributed Key-Value Store System Design (`system-design/design-distributed-kv-store`) | Consensus and partitioning for the metadata service |
| Photo Sharing System Design (`system-design/design-photo-sharing`) | A main client of this store |
| CDN System Design (`system-design/design-cdn`) | Delivery in front of the store |
| Backup Policy (`ops/backup-policy`) | What versioning does not replace |
| Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`) | Method for section 3 |

## Standard and licence note

Based on common industry practice for object storage, the published Haystack
(OSDI 2010) and f4 (OSDI 2014) papers and the public Amazon S3 documentation
cited in 12.3, used only as a behaviour reference. The papers are
paraphrased in 7.7 only; no text, figures or code were copied. The
durability figures come from the simple independent-failure model in 7.3 and
are illustrative, not a guarantee; all capacity numbers are examples computed
from the stated assumptions. Original text written for this collection.
Template licence: MIT.