File Sync System Design Template (Reference Design)

System Design: [File sync service name]

Reference design for a cloud file sync service in the style of consumer and business "sync folder" products: a desktop and mobile client keeps a local folder in step with the cloud and with the user's other devices. It covers content-defined chunking (as introduced by the Low-Bandwidth File System, SOSP 2001), content-addressed deduplication and its privacy side channel, the commit protocol between metadata and block storage, change journals and push notifications, and conflict handling with a three-tree model. It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box that paraphrases public engineering write-ups, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Our first sync client uploads whole files on every save and polls the server every 60 seconds. A one-line change to a 200 MB design file re-uploads 200 MB, laptops on hotel Wi-Fi fall hours behind, and when two devices edit the same file the later upload silently replaces the earlier one. Support logs about 400 "lost edit" tickets a month.]

1.2 Proposed solution

[Example: The client splits files into content-defined chunks (average 1 MB, bounded 256 KB to 4 MB) and identifies each chunk by its SHA-256 hash. A save becomes a commit of the file's ordered chunk list to the metadata service; the server answers with the chunks it does not already have in the account, the client uploads only those to the block service, and the commit completes with an optimistic-concurrency check on the file's parent revision. Each committed change is appended to a per-namespace journal, and a notification service tells the user's other online devices to pull changes since their cursor. Conflicting edits never overwrite: the losing version is saved beside the original as a conflicted copy.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer, upload the whole file on every change, and show the cost of a small edit to a large file. Then try fixed-size blocks and show that inserting one byte shifts every later block. Finish with content-defined chunking and the "commit, need chunks, upload, commit again" protocol.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: real-time co-editing inside documents (see the collaborative editing design)]
• [Example: full-text search over file contents]
• [Example: client-side end-to-end encryption (would disable server-side deduplication and previews)]
• [Example: the object storage system itself (see system-design/design-object-storage)]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Storage

Consequence: content belongs in an erasure-coded object store; the sync service never stores bytes itself, only chunk references.

3.3 Commit and transfer rate

Consequence: uploads and downloads go straight between clients and the block service in the nearest region; metadata servers never proxy bytes.

3.4 Metadata

With an average of 1.6 chunks per file (most files are one small chunk, large files have many), files reference about 128 billion chunks, or about 102 billion unique chunks after the 20% deduplication saving; a chunk index entry of 64 B makes the chunk index about 6.6 TB. A commit costs about 5 metadata writes (file row, version row, chunk references, journal entry, quota), so peak writes are about 13 900/s. Consequence: shard metadata by namespace from day one; 64 shards hold about 375 GB and take about 217 writes/s each at peak.

3.5 Connected devices

At 100 000 idle persistent connections per notification server that is 60 servers, or 90 with 1.5x headroom. Consequence: notifications are a separate tier from the metadata API.

3.6 Summary

4. API design

4.1 Interfaces

4.2 Commit request and responses

The client sends the path, the stable file ID, the revision it edited (its parent) and the chunk hashes in order. The commit is idempotent through the Idempotency-Key header so a retried commit never creates two versions.

{"file_id": "f_5Q2M8X", "path": "/Projects/brochure.indd", "parent_rev": 41, "size": 209715200, "chunks": ["9a1f3c", "77be02", "c4d815"], "client_mtime": "2026-10-30T08:12:44Z", "device_id": "dev_mac_02"}

If any chunk is unknown in the account, the server answers 409 with the missing hashes, and the client uploads them and retries:

{"type": "https://api.example.com/problems/need-chunks", "title": "Chunks missing", "status": 409, "missing": ["c4d815"]}

If parent_rev is no longer the latest revision, the server answers with a conflict problem that carries the current revision, and the client follows 7.4. The chunk hashes above are shortened for the example; real requests carry full 64-character SHA-256 values.

5. Data model

5.1 Entities

A namespace is a user's root or a shared folder; it is the unit of sharding, journaling and permission.

5.2 Storage choices

5.3 Access patterns

6. High-level design

Metadata and content take separate paths: small, consistent writes to the metadata service; large, idempotent transfers to the block service.

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Chunking

Content-defined chunking slides a rolling hash (for example a Rabin fingerprint) over a small window of bytes and declares a boundary wherever the hash matches a chosen bit pattern. Because boundaries depend on content, not on offsets, an insertion only affects the chunks around it. Minimum and maximum sizes (here 256 KB and 4 MB) stop pathological inputs, such as long runs of zeros, from producing tiny or huge chunks. Each chunk is hashed with SHA-256; the hash is its identity everywhere in the system.

7.2 Deduplication and its side channel

When chunks are deduplicated across all users, an upload that completes instantly tells the uploader that someone already stored that content. Harnik, Pinkas and Shulman-Peleg (IEEE Security and Privacy, 2010) showed that this lets an attacker test whether a specific file, or a file with a guessed secret in it, exists in other users' storage. This design deduplicates within one account only: versions of the same file, copies and moves are the bulk of the saving (the 20% in 3.1), and no answer about other users' content ever leaves the server. Shared folders deduplicate within the namespace owner's account.

7.3 Change journal and notifications

Every commit appends an entry with the next journal_id to its namespace's journal in the same transaction as the file update. A device keeps one cursor (the last journal_id applied) per namespace. The notification service only sends "namespace X changed"; the device then calls list changes since cursor. Notifications are therefore hints: a lost or duplicated signal costs at most one extra call, and correctness rests entirely on the journal. Clients that cannot keep a WebSocket open fall back to long polling, and every client also pulls on reconnect and every 10 minutes as a safety net.

7.4 Conflict handling with three trees

The client keeps three trees in its local database: the remote tree (what the server last reported), the local tree (what is on disk) and the synced tree (the last state where both agreed). The synced tree is the merge base: comparing each side with it shows which side changed.

Moves and renames use the stable file_id, so a rename on one device and an edit on another merge cleanly. The server enforces the same rule: a commit whose parent_rev is stale is rejected, never applied over the newer revision. The conflicted copy is named [name] (conflicted copy, [device], YYYY-MM-DD).[ext] so users can find it.

7.5 Commit protocol

The block service verifies that uploaded bytes hash to the name they were uploaded under, so a buggy or hostile client cannot poison the chunk store. Chunks uploaded but never committed are deleted by the garbage collector after 24 hours.

7.6 Testing sync correctness

Sync bugs are rare interleavings: a rename during an upload, a delete during a download, a crash between two steps. Run the planner with a deterministic scheduler against a simulated file system and server, generate random operation sequences from a seed, and check invariants after each step (no file lost, trees converge, no silent overwrite). A failing seed reproduces the bug exactly.

7.7 How real systems do it

How real systems do it (paraphrased, not copied). In "Streaming File Synchronization" (Dropbox tech blog, Nipunn Koorapati, 10 July 2014), Dropbox describes splitting files into 4 MB blocks identified by SHA-256 hashes. A client commits a file's list of block hashes to a metadata server, which replies with the blocks it still needs; the client uploads those to a block server and commits again. The post explains how letting other devices start fetching blocks before the commit finishes improved multi-device sync time. In "Rewriting the heart of our sync engine" (Sujay Jayakar, 9 March 2020), Dropbox explains why it rebuilt its desktop sync engine in Rust: files gained stable identifiers, moves became atomic, and most logic moved onto one deterministic control thread so that pseudorandom simulation tests could reproduce any failure from its seed. A companion post, "Testing sync at Dropbox" (Isaac Goldberg, 20 April 2020), describes the remote, local and synced trees used as a merge base, the model adopted in 7.4. This design differs on one point: it uses content-defined chunks instead of fixed blocks to cut re-uploads after insertions. Read the originals in 12.3.

8. Scalability and performance

Sync latency budget for a small change between two online devices (NFR-02, 5 s p95):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and privacy

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Release the client to 1% of users, then 10%, 50% and 100%, each stage held for 7 days and gated on the conflicted-copy rate and rescan rate. The server keeps accepting the old commit format for two client versions, so a client rollback needs no server change.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on common industry practice for file synchronisation and on the published papers and engineering write-ups cited in 12.3. The Dropbox posts are paraphrased in 7.7 only; no text, figures or code were copied, the fictional service and every number in this template are illustrative examples computed for this design, and nothing here describes any company's current system. Original text written for this collection. Template licence: MIT.