# System Design: [Photo sharing service name] > Reference design for an Instagram-style photo sharing service: resumable > upload straight to object storage, an asynchronous media pipeline that > produces thumbnails and display sizes, storage tiers from hot to cold, > profiles and a follower feed, a time-sortable 64-bit id scheme, sharded > PostgreSQL for metadata, celebrity fan-out, and CDN delivery. It follows > the structure of the System Design Document template, adds a 45-minute > interview path, and is pre-filled with a worked design to adapt. Replace > every `[bracketed]` value and every row marked *Example*; numbers marked > *Example* are assumptions to recompute. ## Document control | Field | Value | |---|---| | Document ID | [SDD-PHOTO-001] | | System / feature | [Photo upload, storage, profile and feed] | | Version | [0.1] | | Status | [Draft / In review / Approved / Implemented / Superseded] | | Owner | [@owner] | | Author(s) | [@author] | | Approvers / reviewers | [@reviewer-1], [@reviewer-2] | | Last updated | YYYY-MM-DD | | Classification | [Internal / Confidential] | | Personal data | [Photos, captions, location metadata, follower graph] | | Related | News Feed System Design (`system-design/design-news-feed`), Content Delivery Network System Design (`system-design/design-cdn`), [Content moderation policy], [ADR: id scheme] | ### Approvals | Role | Name | Decision | Date | |---|---|---|---| | Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD | | Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD | | SRE / storage | [@name] | [Decision] | YYYY-MM-DD | | Privacy and trust and safety | [@name] | [Decision] | YYYY-MM-DD | | Finance (storage and CDN) | [@name] | [Decision] | YYYY-MM-DD | ## Revision history | Version | Date | Author | Change summary | |---|---|---|---| | 0.1 | YYYY-MM-DD | [@name] | First draft for design review | | 0.2 | YYYY-MM-DD | [@name] | *Example:* Client-side resize before upload; originals capped at 1 080 px wide | | 0.3 | YYYY-MM-DD | [@name] | *Example:* Hybrid feed: pull for accounts above 200 000 followers | | 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved | ## Contents 1. [Overview](#1-overview) 2. [Requirements](#2-requirements) 3. [Capacity estimation](#3-capacity-estimation) 4. [API design](#4-api-design) 5. [Data model](#5-data-model) 6. [High-level design](#6-high-level-design) 7. [Detailed design](#7-detailed-design) 8. [Scalability and performance](#8-scalability-and-performance) 9. [Reliability, security and operations](#9-reliability-security-and-operations) 10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives) 11. [Delivery plan and risks](#11-delivery-plan-and-risks) 12. [Appendices](#12-appendices) ## 1. Overview ### 1.1 Problem statement [*Example:* The current service stores photos on application servers' disks and metadata in one PostgreSQL primary. Uploads time out on mobile networks, thumbnails are generated synchronously inside the upload request, the primary is at 80% CPU and 4 TB, and a celebrity post causes a write storm into millions of feed rows. Growth to 100 million uploads per day is planned within 18 months.] ### 1.2 Proposed solution [*Example:* Clients resize and compress photos, then upload them directly to object storage with a resumable, pre-authorised upload session. A finalize call creates the media record with a 64-bit time-sortable id and publishes an event; workers generate display sizes, strip sensitive metadata, run safety checks and mark the media ready. Metadata lives in PostgreSQL sharded by user id across thousands of logical shards. Feeds use fan-out on write into per-user timelines in a key-value store, except for accounts above a follower threshold, whose posts are merged at read time. Images are served through a CDN from versioned, immutable URLs; storage moves from replicated hot storage to erasure-coded warm storage after 30 days.] ### 1.3 Decisions requested from reviewers | ID | Question | Recommended option | Section | |---|---|---|---| | Q-01 | *Example:* How do clients upload? | Direct to object storage with resumable sessions; no bytes through app servers | 7.1 | | Q-02 | *Example:* Id scheme? | 64-bit: time, logical shard, sequence, generated inside the shard database | 7.2 | | Q-03 | *Example:* Feed fan-out for large accounts? | Pull above 200 000 followers, merged at read | 7.3 | | Q-04 | *Example:* Storage tiers? | Hot replicated for 30 days, then erasure-coded warm; originals of deleted media purged in 30 days | 7.4 | | [Q-nn] | [Question] | [Option] | [Section] | ### 1.4 45-minute interview path | Minutes | Cover | Output on the whiteboard | |---|---|---| | 0 to 5 | Core requirements (2.1) and below the line (2.3) | Upload, view, profile grid, follow, feed | | 5 to 10 | Estimates (3) with consequences | Storage per year, view egress, "CDN and tiering are mandatory" | | 10 to 15 | API (4) and data model (5) | Media, users, follows; access-pattern table | | 15 to 25 | High-level design (6) | Upload path and read path drawn separately | | 25 to 40 | One or two deep dives | 7.1 (upload pipeline) by default; 7.3 (celebrity fan-out) if the feed comes up; 7.2 (ids) if sharding comes up | | 40 to 45 | Failure modes and 10x (9.1, 10.1) | What breaks first and the fix | Separate the write path (upload) from the read path (view) early; they have different bottlenecks and the interviewer will want both. ## 2. Requirements ### 2.1 Core functional requirements | ID | Requirement | Priority | |---|---|---| | FR-01 | *Example:* Upload one or more photos with a caption; resumable on poor networks | Must | | FR-02 | *Example:* Show the photo in several display sizes, including a grid thumbnail | Must | | FR-03 | *Example:* Profile page with a reverse-chronological grid of a user's posts | Must | | FR-04 | *Example:* Follow and unfollow users | Must | | FR-05 | *Example:* Home feed of posts from followed accounts | Must | | FR-06 | *Example:* Like and comment on posts, with counts | Must | | FR-07 | *Example:* Delete a post; photo becomes unreachable within 1 minute | Must | | FR-08 | *Example:* Private accounts visible only to approved followers | Must | | [FR-nn] | [Requirement] | [Must / Should / Could] | ### 2.2 Non-functional requirements | ID | Quality | Target | Measured as | |---|---|---|---| | NFR-01 | Upload success | *Example:* 99.5% of started uploads complete on 3G-class networks | Client telemetry | | NFR-02 | Time to visible | *Example:* Post visible to its author within 2 s of finalize, to followers within 10 s at p95 | Synthetic probes | | NFR-03 | Image load | *Example:* First feed image p75 under 500 ms on mobile | Real user monitoring | | NFR-04 | Durability | *Example:* No loss of acknowledged uploads; 11 nines design target for object storage | Provider design figures and restore tests | | NFR-05 | Availability | *Example:* 99.95% for feed and profile reads | Error budget | | NFR-06 | Privacy | *Example:* GPS metadata removed from every served image | Pipeline tests | | [NFR-nn] | [Quality] | [Target] | [Measurement] | ### 2.3 Below the line (out of scope) - [*Example:* video, stories and live (see Video Streaming System Design (`system-design/design-video-streaming`))] - [*Example:* feed ranking models (this design delivers candidates; ranking is a separate service)] - [*Example:* direct messages (see Chat System Design (`system-design/design-chat-system`))] - [*Example:* search and hashtag pages] - [*Example:* image editing filters beyond what the client applies before upload] - [Item deferred, with the release that will take it] ## 3. Capacity estimation ### 3.1 Assumptions | Input | Value | Source | |---|---|---| | Daily active users | *Example:* 500 million | Growth plan | | Photos uploaded per day | *Example:* 100 million | Growth plan | | Stored bytes per photo, original after client resize | *Example:* 1.5 MB | Client telemetry | | Stored bytes per photo, all derived sizes | *Example:* 0.5 MB | Pipeline output | | Photo views per day | *Example:* 25 billion (50 per DAU) | Feed analytics | | Average bytes served per view | *Example:* 150 KB | Mix of grid and full-width sizes | | Peak to average ratio | *Example:* 3 | Traffic curve | ### 3.2 Uploads - Average: 100 million / 86 400 s = about 1 160 uploads/s; peak about 3 500/s. - Peak ingest bandwidth: 3 500 x 1.5 MB = 5.25 GB/s, about 42 Gbit/s. **Consequence:** bytes must not pass through application servers; clients upload directly to object storage (7.1). ### 3.3 Storage - Per day: 100 million x 2 MB = 200 TB. - Per year: 200 TB x 365 = 73 PB of logical data. - With 3 replicas for 30 days and erasure coding at 1.4x afterwards, the physical footprint after one year is about 200 TB x 30 x 3 + 200 TB x 335 x 1.4 = 18 PB + 93.8 PB, about 112 PB. Keeping 3 replicas for the whole year would need 219 PB. **Consequence:** tiering halves the storage bill; views drop sharply after the first days, so warm storage is acceptable for older media (7.4). ### 3.4 Views and egress - Average: 25 billion / 86 400 s = about 289 000 views/s; peak about 870 000/s. - Peak egress: 870 000 x 150 KB = about 130 GB/s, about 1 Tbit/s. **Consequence:** a CDN is mandatory. At a 95% CDN hit rate the origin serves about 6.5 GB/s, which still needs an origin shield tier in front of object storage. ### 3.5 Metadata - Media rows: 100 million/day x 1 KB = 100 GB/day, about 36.5 TB/year, plus likes and comments at several times that. **Consequence:** one PostgreSQL primary cannot hold this; metadata is sharded from the start, with logical shards so the physical layout can change later (7.2). ### 3.6 Feed fan-out - Average 200 followers per poster: 100 million posts x 200 = 20 billion timeline inserts/day, about 231 000/s on average. - An account with 50 million followers would add 50 million inserts for one post. **Consequence:** fan-out on write for most accounts, fan-out on read above a threshold (7.3). ### 3.7 Summary | Quantity | Estimate | Design consequence | |---|---|---| | Upload ingest | 42 Gbit/s at peak | Direct-to-storage resumable upload | | Storage growth | 73 PB/year logical | Tiering and erasure coding | | View egress | ~1 Tbit/s at peak | CDN with origin shield, immutable URLs | | Metadata | Tens of TB/year | Sharded PostgreSQL with logical shards | | Timeline inserts | ~231 000/s average | Hybrid push and pull feed | ## 4. API design ### 4.1 Endpoints | Method and path | Purpose | Notes | |---|---|---| | `POST /v1/uploads` | Create an upload session for one photo | Returns a pre-authorised resumable upload URL and an `upload_id` | | `PUT {upload_url}` | Upload bytes in chunks | Object storage endpoint, not the API | | `POST /v1/media` | Finalize: create the post from completed uploads | `Idempotency-Key` header required | | `GET /v1/media/{media_id}` | Post details and image URLs by size | URLs are CDN paths with content hash | | `DELETE /v1/media/{media_id}` | Delete a post | Purges CDN paths | | `GET /v1/users/{user_id}/media?cursor=` | Profile grid, newest first | Cursor is the last media id | | `GET /v1/feed?cursor=` | Home feed page | Cursor encodes position in the merged timeline | | `PUT /v1/users/{user_id}/followers/me` | Follow | Pending if the account is private | ### 4.2 Request and response examples Finalize after the upload completes. The id is a 64-bit integer returned as a string so JavaScript clients do not lose precision. ```http POST /v1/media Idempotency-Key: 8d4c2f0e-6a1b-4c55-9f2e-0b7c1d9a3e11 Content-Type: application/json {"upload_ids": ["up_7Hq2"], "caption": "Morning run", "audience": "followers"} ``` ```json {"media_id": "3477619814263193600", "status": "processing", "author_id": "1028", "created_at": "2026-10-22T07:12:44Z"} ``` ### 4.3 Errors, limits and pagination | Case | Behaviour | |---|---| | Upload larger than limit (*Example:* 30 MB) | Reject at session creation with 413 problem details | | Unsupported format | Reject at finalize with 415; supported: JPEG, HEIC, PNG, WebP | | Duplicate finalize with same key | Return the first response unchanged | | Processing failed | Media status `failed` with reason; client may retry finalize | | Paging | Cursor-based on media id; ids are time-sortable, so no offset paging | ## 5. Data model ### 5.1 Entities Metadata is relational; bytes live in object storage; timelines live in a key-value store. ```mermaid erDiagram USER { bigint user_id PK string handle bool is_private bigint follower_count } MEDIA { bigint media_id PK bigint author_id FK string caption string status string storage_key string content_hash datetime created_at } RENDITION { bigint media_id PK string size PK int width int height string object_key } FOLLOW { bigint follower_id PK bigint followee_id PK datetime created_at } TIMELINE_ENTRY { bigint owner_id PK bigint media_id PK } USER ||--o{ MEDIA : posts MEDIA ||--o{ RENDITION : "has sizes" USER ||--o{ FOLLOW : follows USER ||--o{ TIMELINE_ENTRY : "feed of" ``` ### 5.2 Storage choices | Data | Store | Shard key | Why | |---|---|---|---| | Users, media, renditions, likes, comments | *Example:* PostgreSQL, 8 192 logical shards on 64 hosts at launch | `user_id` (author) | Profile grid and author's posts on one shard | | Follow edges | Same PostgreSQL shards, both directions | `follower_id` for following list, `followee_id` for followers list | Each list on one shard | | Timelines | *Example:* Wide-column or Redis-compatible store | `owner_id` | Append and read newest N per user | | Photo bytes | Object storage, hot and warm classes | Content-addressed key | Cheap, durable, CDN origin | | Counters (likes) | Sharded counters with periodic roll-up | `media_id` | Avoid row contention on viral posts | ### 5.3 Access patterns | Query | Served by | Key or index | Partition rationale | |---|---|---|---| | Profile grid, newest first | `media` on author's shard | `(author_id, media_id desc)` | Author is the shard key; media id is time-sortable | | Single post by id | `media` | `media_id`, shard decoded from the id | No lookup service needed | | Home feed page | Timeline store, plus pull for celebrities | `owner_id` list, then `media` by id | Precomputed list per reader | | Followers of a user | `follow` inverse rows | `(followee_id, created_at)` on followee's shard | Fan-out workers read one shard | | Image bytes for a size | CDN, then object storage | `/{hash}/{size}.webp` | Immutable URL, cacheable forever | | Like count | Counter roll-up cached with post | `media_id` | Read-heavy, write-bursty | ## 6. High-level design ```mermaid flowchart LR app([Mobile app]) -->|1 create session| api[Media API] app -->|2 upload chunks| obj[(Object storage, hot)] app -->|3 finalize| api api --> pg[(Sharded PostgreSQL)] api --> bus[[Event bus]] bus --> proc[Media processing workers] proc --> obj proc --> pg bus --> fan[Fan-out workers] fan --> tl[(Timeline store)] app -->|feed, profile| read[Read API] read --> tl read --> pg app -->|images| cdn[CDN edge] cdn --> shield[Origin shield] shield --> obj obj -->|after 30 days| warm[(Object storage, warm)] ``` | Component | Responsibility | Scales by | |---|---|---| | Media API | Upload sessions, finalize, delete; id allocation | Stateless instances | | Media processing workers | Decode, resize, strip metadata, safety scan, write renditions | Queue depth | | Fan-out workers | Insert media ids into followers' timelines | Event rate x followers | | Read API | Profile, feed merge, hydration of posts | Stateless instances, caches | | Sharded PostgreSQL | Users, media, follows, counters | Logical shards moved between hosts | | Timeline store | Newest N media ids per user | Partitions by owner | | CDN and origin shield | Serve image bytes | Provider capacity; see Content Delivery Network System Design (`system-design/design-cdn`) | ## 7. Detailed design ### 7.1 Deep dive 1: upload and media pipeline **Problem.** Uploads must survive flaky mobile networks, must not tie up application servers, and the post must appear quickly even though resizing and safety checks take seconds. | Option | How it works | Drawback | |---|---|---| | A. Upload through the API, process synchronously | Simple | Long requests, timeouts, app servers move every byte | | B. Upload through the API, process asynchronously | Faster response | App servers still move every byte | | C. Direct resumable upload to object storage, process asynchronously | API only signs and records | Two-step client flow, orphaned uploads to clean up | **Choice.** C. **Reason.** The 42 Gbit/s peak ingest (3.2) goes straight to storage built for it. Resumable chunked upload survives network changes. The author sees the post immediately from a local copy while processing runs; followers get it once status becomes `ready`. ```mermaid sequenceDiagram participant A as App participant API as Media API participant S as Object storage participant Q as Event bus participant W as Processing worker participant DB as PostgreSQL shard A->>API: POST /v1/uploads API-->>A: upload_url (expires 1 h) A->>S: PUT chunks (resumable) A->>API: POST /v1/media (Idempotency-Key) API->>DB: insert media (status processing) API->>Q: media.created API-->>A: 201 media_id Q->>W: media.created W->>S: read original, write sizes W->>DB: renditions, status ready W->>Q: media.ready (triggers fan-out) ``` Processing steps, each idempotent so a retried message is safe: 1. Verify the file type by content, not by extension; reject anything the decoder does not recognise. 2. Decode in a sandboxed worker with memory and time limits. 3. Apply orientation, strip location and device metadata. 4. Produce display sizes (*Example:* 150, 320, 640, 1 080 px wide) in WebP and JPEG, keyed by content hash and size. 5. Run safety classification and known-harmful-content hash matching; hold for review when flagged. 6. Mark `ready` and emit `media.ready`. A daily job deletes uploads that were never finalized after 24 hours. ### 7.2 Deep dive 2: ids and sharding **Problem.** Ids must be unique across thousands of shards, sortable by time for paging, compact (64 bits), and must tell the router which shard holds the row. | Option | Unique | Time-sortable | Shard routing | Notes | |---|---|---|---|---| | A. UUIDv4 | Yes | No | Needs lookup | 128 bits, random index inserts | | B. UUIDv7 (RFC 9562) | Yes | Yes | Needs lookup | 128 bits | | C. Central ticket server | Yes | Roughly | Needs lookup | Extra dependency on every write | | D. 64-bit time + shard + sequence, generated in the shard database | Yes | Yes (to the millisecond) | Shard decoded from id | Clock discipline required | **Choice.** D, with this layout (*Example*): | Bits | Field | Range | |---|---|---| | 41 | Milliseconds since a custom epoch | About 69.7 years (2^41 ms); in a signed PostgreSQL `bigint` the id turns negative after about 34.8 years (2^40 ms), so plan the epoch or the migration for that date | | 13 | Logical shard id | 8 192 logical shards | | 10 | Per-shard sequence, modulo 1 024 | 1 024 ids per shard per millisecond | **Reason.** Paging by id is paging by time, so the profile grid and timeline need no separate time index. The shard is part of the id, so a post URL routes without a lookup. Generating the id inside the shard's database (a stored function using the shard's own sequence) keeps it unique without a new service. Logical shards are mapped to physical hosts through a small versioned map. Each logical shard is a separate schema, so moving it to a new host is a replication copy plus a map update, with no id changes. *Example:* start with 8 192 logical shards on 64 hosts (128 per host) and split hosts as they fill. > **How real systems do it** (paraphrased from the Instagram Engineering > post "Sharding & IDs at Instagram", 2011; inspiration only). The post > explains why the team kept PostgreSQL and sharded it instead of moving to > a different database, and how it generated unique ids once inserts were > spread across many databases. Its id goals were time sortability and a > 64-bit size, and its scheme composed a time component, a logical shard > id and a per-shard sequence, generated inside PostgreSQL. Data was split > into many logical shards placed on fewer physical servers, so shards > could later be moved without changing ids. The post describes the system > as it was in 2011; check it for the exact layout and reasoning. ### 7.3 Deep dive 3: celebrity fan-out **Problem.** Fan-out on write gives fast feed reads but explodes for accounts with millions of followers; fan-out on read is cheap to write but expensive for every feed load. | Option | Write cost per post | Read cost per feed page | Freshness | |---|---|---|---| | A. Push to all followers | O(followers) | One list read | Seconds, minutes for celebrities | | B. Pull at read time | O(1) | One query per followee | Immediate | | C. Hybrid: push below threshold, pull above | Bounded | List read plus a few celebrity lookups | Seconds | **Choice.** C with a threshold of *Example:* 200 000 followers. Pushes skip followers inactive for 30 days; they rebuild their timeline on next login. **Reason.** Most posters are small, so push covers most posts cheaply. A typical reader follows a handful of large accounts, so the read path merges fewer than *Example:* 20 celebrity lists, each served from a cache of that account's latest posts. See News Feed System Design (`system-design/design-news-feed`) for ranking and timeline storage detail. ### 7.4 Deep dive 4: storage tiers and CDN **Problem.** Storage grows by 73 PB a year while most views hit recent photos. | Option | Cost | Read latency on cold photos | |---|---|---| | A. Everything in hot replicated storage | Highest | Lowest | | B. Hot for 30 days, then erasure-coded warm | About half (3.3) | Slightly higher, hidden by CDN | | C. Add an archive tier for originals older than 1 year | Lowest | Minutes to hours; only for originals never served | **Choice.** B for all served sizes, C for originals not needed for serving. Served sizes are generated at upload, so the archive tier is read only when a new size must be produced. **Reason.** The CDN absorbs repeat views, and warm storage latency is paid only on CDN misses for old photos. Image URLs contain the content hash (`/{hash}/640.webp`), so they are cached with a long immutable lifetime; deletion removes the database row, revokes the path and purges the CDN. ### 7.5 Privacy and deletion Private accounts serve images through signed URLs with short expiry (*Example:* 1 hour) so a leaked URL stops working. Deletion marks the media row, removes timeline entries asynchronously, purges CDN paths, and deletes bytes from storage after a recovery window of *Example:* 30 days. ## 8. Scalability and performance | Concern | Approach | Limit before redesign | |---|---|---| | Metadata writes | More hosts, move logical shards | 8 192 hosts (one logical shard each) | | Hot author shard | Celebrity posts cached; counters sharded | One author above 100 000 reads/s on the shard | | Processing backlog | Autoscale workers on queue age | Decoder licence or GPU budget if added | | Timeline storage | Keep newest *Example:* 800 ids per user | Readers paging deeper fall back to pull | | CDN egress | Multiple CDN providers, steering by cost and quality | Provider capacity per region | Latency budget for the first feed image (target p75 500 ms on mobile): | Step | Budget | |---|---| | Feed API (timeline read, hydrate 10 posts) | 120 ms | | DNS and TLS to CDN (often reused) | 80 ms | | CDN edge hit, first bytes | 60 ms | | Download 150 KB on a typical mobile link | 200 ms | | Decode and render | 40 ms | ## 9. Reliability, security and operations ### 9.1 Failure modes | Component | Failure | Detection | Blast radius | Mitigation | RTO | |---|---|---|---|---|---| | Object storage region | Unavailable | Upload error rate | New uploads in the region | Upload sessions issued for a second region; reads served by CDN | *Example:* 5 min | | Processing workers | Backlog | Queue age above 60 s | New posts delayed for followers | Autoscale; author still sees local copy | *Example:* 15 min | | PostgreSQL shard host | Primary failure | Replication monitor | Users on that host's logical shards | Promote replica | *Example:* 2 min | | Fan-out workers | Lag | Event age | Followers see posts late | Readers fall back to pull for recent followees | *Example:* 10 min | | CDN provider | Outage in a region | Real user monitoring | Image loads in that region | Steer to second provider by DNS | *Example:* 5 min | | Id generation | Clock moves backwards | Clock skew monitor | Possible duplicate ids on one shard | Refuse to generate until clock passes last issued time; NTP monitoring | Seconds | ### 9.2 Security and abuse | Threat | Control | |---|---| | Malicious image files exploiting decoders | Content-type sniffing, sandboxed decoding, size and pixel limits, patched libraries | | Location leakage | Strip EXIF GPS and device identifiers before any rendition is served | | Scraping private photos | Signed short-lived URLs; follower check on the API | | Harmful content | Hash matching and classification before `ready`; reporting flow | | Enumeration of ids | Ids reveal creation time by design; access checks never rely on id secrecy | | [Threat] | [Control] | ### 9.3 Observability | Signal | Metric | Alert threshold | |---|---|---| | Upload completion | Finalized / started | Below 99% for 15 min | | Time to ready | p95 from finalize to `ready` | Above 10 s | | Feed latency | Read API p99 | Above 300 ms | | CDN hit rate | Per provider | Below 93% | | Shard load | CPU and replication lag per host | CPU above 70%, lag above 5 s | ### 9.4 Cost estimate | Item | Unit cost | Quantity | Monthly cost | |---|---|---|---| | Hot object storage | [USD per TB-month] | *Example:* 18 PB physical | [USD] | | Warm object storage | [USD per TB-month] | *Example:* growing to 94 PB | [USD] | | CDN egress | [USD per GB] | *Example:* about 3.75 PB per day (25 billion x 150 KB) | [USD] | | Processing compute | [USD per vCPU-hour] | [vCPU-hours] | [USD] | | PostgreSQL hosts | [USD per host-month] | *Example:* 64 primaries + 128 replicas | [USD] | | **Total** | | | **[USD]** | ## 10. Trade-offs and alternatives ### 10.1 Decisions | Decision | Options | Chosen | Why | What we give up | |---|---|---|---|---| | Upload path | Through API; direct to storage | Direct, resumable | Scales with storage, survives poor networks | Orphaned uploads to clean up | | Metadata store | Single database; NoSQL; sharded PostgreSQL | Sharded PostgreSQL | Team expertise, transactions per user | Cross-shard queries need fan-out | | Ids | UUID; ticket server; time+shard+sequence | Time+shard+sequence | 64-bit, sortable, routable | Ids reveal creation time | | Feed | Push; pull; hybrid | Hybrid | Bounded write cost | Two code paths to merge | | Image sizes | On upload; on demand at edge | On upload | Predictable latency and cost | Adding a new size needs a backfill | | **At 10x** (1 billion uploads/day) | | | Storage 730 PB/year: archive more aggressively; regional processing; more logical shards via id-space split | Id layout may need more shard bits | | **At 0.1x** (10 million uploads/day) | | | One PostgreSQL primary with replicas and a managed CDN suffice; push-only feed | Sharding effort is not worth it yet, but keep the id layout so you can shard later | | [Decision] | [Options] | [Chosen] | [Why] | [Cost] | ### 10.2 Alternatives considered - **On-demand resizing at the edge.** Fewer stored bytes, any size on request, but a cold cache costs decode time on the critical path and ties you to an edge compute product. Reasonable at 0.1x. - **A dedicated blob store with needle-style files.** Large photo platforms have built their own stores to cut per-file metadata overhead; object storage is chosen here until per-object request cost dominates. ## 11. Delivery plan and risks ### 11.1 Milestones | Milestone | Scope | Exit criterion | Target date | |---|---|---|---| | M1 | Direct upload and async pipeline | 99.5% upload completion in beta | YYYY-MM-DD | | M2 | New id scheme and sharded metadata for new posts | Shadow writes consistent for 14 days | YYYY-MM-DD | | M3 | Backfill old posts to shards | Old primary read-only | YYYY-MM-DD | | M4 | Hybrid feed and storage tiering | Feed p99 under 300 ms; warm tier live | YYYY-MM-DD | ### 11.2 Rollout and rollback [*Example:* Enable the new upload path per app version and per country. Rollback returns clients to the old upload endpoint, which remains deployed until M3. Metadata migration dual-writes until the backfill is verified.] ### 11.3 Risks | ID | Risk | Likelihood | Impact | Mitigation | Owner | |---|---|---|---|---|---| | R-01 | *Example:* Backfill of old ids breaks external links | Medium | High | Keep a legacy id lookup table; redirect old URLs | [@name] | | R-02 | *Example:* Decoder vulnerability in an image library | Medium | High | Sandboxing, rapid patching, fuzzing in CI | [@name] | | R-03 | *Example:* CDN cost exceeds forecast | Medium | Medium | Size mix review; second provider for price | [@name] | | [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] | ## 12. Appendices ### 12.1 Open questions | ID | Question | Owner | Due | |---|---|---|---| | OQ-01 | *Example:* Do we serve AVIF to supporting clients? | [@name] | YYYY-MM-DD | | OQ-02 | *Example:* Retention of originals for deleted accounts under local law? | [@name] | YYYY-MM-DD | | [OQ-nn] | [Question] | [@name] | YYYY-MM-DD | ### 12.2 Glossary | Term | Meaning | |---|---| | Rendition | A derived size or format of the original photo | | Logical shard | A unit of data placement that can be moved between hosts without changing ids | | Fan-out on write | Copying a new post's id into each follower's timeline when posted | | Fan-out on read | Fetching followees' recent posts when a reader opens the feed | | Origin shield | A cache tier between CDN edges and storage that collapses misses | | [Term] | [Meaning] | ### 12.3 References Inspiration only; no text, diagrams or figures are copied. | Reference | Use | |---|---| | Instagram Engineering, "Sharding & IDs at Instagram" (2011, instagram-engineering.com) | Logical shards and time-sortable 64-bit ids (paraphrased in 7.2) | | Beaver et al., "Finding a Needle in Haystack: Facebook's Photo Storage", USENIX OSDI 2010 | Background for 10.2 | | Muralidhar et al., "f4: Facebook's Warm BLOB Storage System", USENIX OSDI 2014 | Background for warm, erasure-coded tiers (7.4) | | RFC 9562 Universally Unique IDentifiers (UUIDs) | UUIDv7 alternative in 7.2 | | RFC 9457 Problem Details for HTTP APIs | Error format | | News Feed System Design (`system-design/design-news-feed`) | Feed ranking and timelines | | Content Delivery Network System Design (`system-design/design-cdn`) | Image delivery | ## Standard and licence note Based on common industry practice for photo sharing services and on the public write-ups and RFCs listed in 12.3, which are cited as inspiration only. This is a reference design to adapt, with illustrative numbers. Original text written for this collection; check the cited sources for the exact mechanisms they describe. Template licence: MIT.