# System Design: [Service name] multi-region active-active

> Reference design for running one service actively in several cloud regions
> at once: regional topology and capacity for the loss of a region, data
> classes and how each resolves conflicting writes (home-region ownership,
> CRDTs, consensus across regions), traffic routing and evacuation, split
> brain, and data residency. It draws on current provider guidance (AWS
> "Disaster Recovery of Workloads on AWS" whitepaper and DynamoDB global
> tables documentation, Azure Well-Architected Framework disaster recovery
> guide updated 2026, Google Cloud Well-Architected Framework reliability
> pillar updated October 2026). It follows the structure of the System Design
> Document template, adds a 45-minute interview path, access-pattern and
> failure-mode tables and a cited "How real systems do it" box, and is
> pre-filled with a worked design to adapt. Replace every `[bracketed]` value
> and every row marked *Example*; values marked *Example* are assumptions,
> not benchmarks. Section 7.6 names data-protection laws as dated examples;
> it is a documentation format, not legal advice: check the current official
> text and a qualified professional.

<!--
Guidance: How to use this template
- Write down why you want active-active before designing it: latency for
  distant users, surviving a region loss, or data residency. Each leads to a
  different topology, and many workloads are better served by a warm standby.
- Classify every table or store into the data classes of 7.1 before
  choosing databases; one conflict strategy for everything is the most
  common mistake.
- Size each region for the traffic it must absorb when a peer fails, not
  for its normal share (section 3).
- Rehearse region evacuation regularly; an untested failover is a hope.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-MRAA-001] |
| System / feature | [Service name, multi-region deployment] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| Regions in scope | [*Example:* eu-1, eu-2, eu-3, us-1, us-2, us-3] |
| Criticality tier | [*Example:* Tier 0 mission critical, SLO 99.99%] |
| Related | [Disaster recovery plan], [Transfer impact assessment], [ADR: conflict strategy per data class] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| Privacy / data protection | [@name] | [Decision] | YYYY-MM-DD |
| Finance (regional cost) | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Moved from two to three regions per geography after the capacity review |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added write leases to prevent split brain on home-region failover |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* A project-management SaaS runs in one region. A four-hour
regional outage last year stopped every customer, enterprise contracts now
ask for a 99.99% monthly SLO, users in the EU and the US see 150 ms or more of
extra round-trip time to the far region, and EU customers require that their
personal data is stored and processed in EU regions.]

### 1.2 Proposed solution

[*Example:* Two home geographies, EU and US, each with three regions that
all serve traffic. Every tenant has a home geography chosen at sign-up and a
home region inside it. User-owned data is written only in the tenant's home
region and replicated asynchronously to the other two regions of the same
geography (write partitioned). Shared, concurrently edited fields use CRDTs.
A small set of data that must never conflict (unique names, billing credits)
uses a consensus group spread over the three regions of the geography.
Global reference data has one writer and is replicated read-only everywhere.
A global traffic layer routes users to the nearest healthy region in their
home geography; a region that loses its write lease stops accepting writes
before another region takes over its tenants.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Two or three regions per geography? | Three: 1.5x capacity instead of 2x, and a majority survives one region loss | 3.3, 10 |
| Q-02 | *Example:* How are write conflicts resolved? | Per data class: home-region ownership, CRDTs, consensus, single writer | 7.1 |
| Q-03 | *Example:* Automatic or manual region evacuation? | Automatic for traffic, lease-guarded for writes; manual approval for failback | 7.4 |
| Q-04 | *Example:* May EU data be replicated to US regions for disaster recovery? | No; EU data stays in the three EU regions, including backups | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Why active-active (latency, region loss, residency), RPO and RTO targets; below-the-line items (2.3) | Two targets and one constraint |
| 5 to 10 | Estimates: traffic per geography, capacity per region for N+1, replication bandwidth, data at risk (section 3) | "Each region runs at under 50% normally" |
| 10 to 15 | Interface: region-aware routing, consistency hints in the API (section 4) | Routing rule and one header |
| 15 to 25 | High-level design: global routing, regions, replication, home directory (section 6) | One diagram |
| 25 to 40 | One deep dive: conflict resolution by data class (default, 7.1 and 7.2), or failover and split brain (7.4), or data residency (7.6) | Options, choice, reason |
| 40 to 45 | Failures (region loss, partition between regions, bad global deploy), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer, "every region accepts
every write and replication is last-writer-wins", and show the lost update
it causes on a counter and on a unique username. Then split the data into
classes and give each the cheapest strategy that is still correct.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Serve reads and writes for each tenant from any healthy region of its home geography | Must |
| FR-02 | *Example:* Keep personal data of EU-homed tenants in EU regions, including replicas, backups and logs | Must |
| FR-03 | *Example:* Enforce global uniqueness of workspace names and exact billing credit balances | Must |
| FR-04 | *Example:* Evacuate a region (drain traffic and move write ownership) with one command | Must |
| FR-05 | *Example:* Show users a consistent view of their own writes (read your writes) | Must |
| FR-06 | *Example:* Move a tenant to another home geography on request, with an audit trail | Should |
| FR-07 | *Example:* Serve read-only reference data (templates, feature flags) from every region | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Availability | *Example:* 99.99% of requests succeed per month (4.32 min of budget in 30 days) | Synthetic and real traffic success ratio |
| NFR-02 | RTO for region loss | *Example:* Traffic moved within 2 min; writes resume within 2 min for affected tenants | Evacuation drill timing |
| NFR-03 | RPO for region loss | *Example:* At most 1 s of acknowledged writes for home-region data; zero for consensus data | Replication lag p99 |
| NFR-04 | Latency | *Example:* p95 read at or below 100 ms for users within their home geography | Real user monitoring |
| NFR-05 | Residency | *Example:* No EU personal data at rest or in processing outside EU regions | Automated data-flow checks, audit |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* recovery from data corruption or malicious deletion (backups and point-in-time restore, see the disaster recovery plan)]
- [*Example:* a third home geography (design keeps it possible)]
- [*Example:* multi-cloud deployment of the same service]
- [*Example:* real-time collaborative editing across geographies]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. Every number should end in a consequence. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Global peak requests per second | $R$ | *Example:* 120 000 | Current peak x growth |
| Write share | $w$ | *Example:* 10% (12 000 writes/s) | Access logs |
| Traffic by home geography | | *Example:* EU 55%, US 45% | Tenant directory |
| Regions per geography | $n$ | *Example:* 3 | Decision Q-01 |
| Target utilisation at peak after losing one region | $u$ | *Example:* 70% | Headroom policy |
| Replicated bytes per write | $b$ | *Example:* 1 KB | Change-stream sample |
| Replication lag p99 within a geography | $\lambda$ | *Example:* 1 s | Measured on pilot |
| Round-trip time between EU regions | | *Example:* 12 ms (eu-1 to eu-2), 25 ms (eu-1 to eu-3) | Network measurements |

### 3.2 Traffic per geography

- EU peak: $0.55 \times 120\,000 = 66\,000$ requests/s, of which 6 600 writes/s.
- US peak: $0.45 \times 120\,000 = 54\,000$ requests/s, of which 5 400 writes/s.
- Normal share per region with three regions: EU 22 000/s, US 18 000/s.

### 3.3 Capacity for the loss of one region

Each region must carry the geography's peak divided by the regions that
remain:

$$
C_{region} = \frac{R_{geo}}{(n - 1) \times u}, \qquad
C_{EU} = \frac{66\,000}{2 \times 0.7} \approx 47\,143 \ \text{requests/s per region}
$$

For the US the same formula gives about 38 571 requests/s per region. In
normal operation an EU region runs at $22\,000 / 47\,143 \approx 47\%$ of its
capacity. With only two regions per geography each region would need
$66\,000 / 0.7 \approx 94\,286$ requests/s and would run at 35%; total EU
capacity would be 188 571 instead of 141 429 requests/s. Consequence: three
regions per geography need 25% less total capacity than two, and they give a
majority for the consensus data in 7.2.

### 3.4 Replication bandwidth and data at risk

$$
B_{EU} = 6\,600 \times 1 \ \text{KB} = 6.6 \ \text{MB/s to each other EU region}
$$

At an average of 40% of peak this is about 6.8 TB a month per destination
region (about 5.6 TB for the US). With asynchronous replication, each EU
region accepts about $6\,600 / 3 = 2\,200$ writes per second, so a lag of 1 s
means about 2 200 acknowledged writes at risk if that region is lost
abruptly (1 800 for a US region). Consequence: the RPO for home-region data
is the replication lag, so lag is an alerting signal (9.3), and data that
cannot accept any loss goes to the consensus class.

### 3.5 Availability arithmetic

If a region is available 99.9% of the time and region failures were
independent, the chance that at least two of three regions are up is

$$
A_{2 \ of \ 3} = a^{3} + 3a^{2}(1-a) = 0.999^{3} + 3 \times 0.999^{2} \times 0.001 \approx 0.999997
$$

A single shared dependency (global DNS provider, identity service, a
deployment pipeline that pushes to all regions at once) with 99.99%
availability multiplies this down to about $0.999997 \times 0.9999 \approx 0.99990$.
Consequence: the shared dependencies, not the region count, decide whether
99.99% is met, so the design removes or regionalises them (7.5).

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Capacity per region | EU about 47 000/s, US about 38 600/s | Regions run below 50% normally; autoscaling is not on the failover path |
| Replication | 6.6 MB/s per EU destination | Cheap; the cost is inter-region transfer, not bandwidth |
| Data at risk | About 2 200 writes per EU region for 1 s of lag | Lag alerting; consensus class for zero-loss data |
| Availability | Shared dependencies cap the result near 99.99% | Regionalise identity, config and deploys |

## 4. API design

<!-- Guidance: Define what clients see of the multi-region design: routing, consistency hints, errors during evacuation, and residency-related behaviour. Use RFC 9457 problem details for errors. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Public API | Single global hostname `api.example.com` | Routed to the nearest healthy region in the tenant's home geography |
| Public API | `X-Consistency-Token` response header | Opaque token (region plus commit position) the client echoes to read its own writes in another region |
| Internal | `GET /home/{tenant_id}` on the home directory | Returns home geography, home region, lease epoch |
| Operations | `POST /regions/{region}/evacuate` | Drain traffic and transfer write ownership |
| Operations | `POST /tenants/{tenant_id}/move` | Move a tenant to another home region or geography |

### 4.2 Routing and consistency behaviour

- The edge reads the tenant from the credential or the URL, looks up the
  home geography in a cached copy of the directory and forwards the request
  to the nearest healthy region of that geography. Requests never leave the
  home geography.
- A write in a non-home region is forwarded to the home region over the
  private backbone (one extra round trip inside the geography).
- A read that carries a consistency token is served from the local replica
  only when the replica has applied that position; otherwise it waits up to
  *Example:* 200 ms, then forwards to the home region.

### 4.3 Error during a write ownership move

```http
HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2

{"type": "https://api.example.com/problems/home-region-moving", "title": "Tenant is moving to another region", "status": 503, "detail": "Writes for this workspace are paused for a few seconds while its home region changes. Reads continue. Retry the same request with the same Idempotency-Key.", "instance": "/v1/projects/prj_81c2/tasks"}
```

Clients send an `Idempotency-Key` on every write so a retry after a
failover never applies a change twice.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

Ownership is explicit: every tenant row points at one home region and the
lease that allows that region to write.

```mermaid
erDiagram
  GEOGRAPHY {
    string geo_id PK
    string residency_rule
  }
  REGION {
    string region_id PK
    string geo_id FK
    string state
  }
  TENANT_HOME {
    string tenant_id PK
    string geo_id FK
    string home_region FK
    bigint lease_epoch
    datetime lease_expires_at
  }
  RECORD {
    string tenant_id PK
    string record_id PK
    string data_class
    bigint commit_position
    string hlc_timestamp
  }
  UNIQUE_NAME {
    string geo_id PK
    string name PK
    string tenant_id FK
  }
  GEOGRAPHY ||--o{ REGION : contains
  GEOGRAPHY ||--o{ TENANT_HOME : hosts
  REGION ||--o{ TENANT_HOME : "is home for"
  TENANT_HOME ||--o{ RECORD : owns
  TENANT_HOME ||--o{ UNIQUE_NAME : reserves
```

### 5.2 Storage choices

| Data | Store | Key | Main queries | Why |
|---|---|---|---|---|
| Tenant home directory | *Example:* Consensus group across the three regions of each geography, cached at the edge | `tenant_id` | Route a request, check a write lease | Must never disagree about who owns a tenant |
| Home-region data (projects, tasks, comments) | Regional database per region, change stream replicated to the other two | `(tenant_id, record_id)` | All tenant reads and writes | Write partitioned, no conflicts |
| Shared counters and sets | CRDT fields inside records | Same | Increment, add, remove | Merge without coordination |
| Unique names, billing credits | Consensus-replicated store across the geography's three regions | `(geo_id, name)`, `(tenant_id, account)` | Reserve name, debit credit | Zero RPO, no conflicts |
| Reference data | Single writer region, read replicas everywhere | `template_id`, `flag_key` | Read everywhere | Rarely written |
| Derived data (search index, caches) | Rebuilt per region from the change stream | Varies | Search | No replication of derived state |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Route a request | Every request (120 000/s) | Edge cache of `tenant_id` to home geography and region | Directory is small and changes rarely; cache invalidated on move |
| Read tenant data | 108 000/s | Local regional replica by `(tenant_id, record_id)` | Partition by tenant keeps one tenant's data together for replication and moves |
| Write tenant data | 12 000/s | Home region primary, with lease epoch check | Single writer per tenant avoids conflicts |
| Reserve a unique name | About 5/s | Consensus store by `(geo_id, name)` | Majority of three regions; 12 ms quorum round trip in the EU |
| Move a tenant | A few per day | Directory entry plus bulk copy of `tenant_id` range | Tenant partitioning makes the move a range copy |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart TB
  user([User]) --> edge[Global edge and traffic manager]
  edge --> eu1
  edge --> us1
  subgraph EU["EU geography"]
    eu1[Region eu-1] <-->|async change stream| eu2[Region eu-2]
    eu2 <-->|async change stream| eu3[Region eu-3]
    eu1 <-->|async change stream| eu3
    eucons[(Consensus store and home directory, 3 regions)]
    eu1 --- eucons
    eu2 --- eucons
    eu3 --- eucons
  end
  subgraph US["US geography"]
    us1[Region us-1] <-->|async change stream| us2[Region us-2]
    us2 <-->|async change stream| us3[Region us-3]
    us1 <-->|async change stream| us3
    uscons[(Consensus store and home directory, 3 regions)]
    us1 --- uscons
    us2 --- uscons
    us3 --- uscons
  end
  ref[(Reference data writer)] -.->|read-only replicas, no personal data| eu1
  ref -.-> us1
```

| Component | Responsibility | Scales by |
|---|---|---|
| Global edge and traffic manager | Health-checked routing to the nearest healthy region of the home geography | Provider capacity |
| Region stamp | Identical deployment: API, workers, regional database, caches | Requests per region |
| Change stream replication | Ship committed changes of home-region data to peer regions in the geography | Write rate |
| Consensus store and home directory | Tenant ownership leases, unique names, billing credits | Small write rate |
| Reference data writer | Publish templates and flags to all regions | Rare writes |
| Evacuation controller | Drain a region, move leases, verify health | Operations |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Conflict strategy | Last writer wins everywhere, single global writer, per data class | Per data class | Correct where it matters, cheap elsewhere | 7.1, 7.2 |
| Write topology | Write global, write local, write partitioned | Write partitioned by tenant home region | No conflicts for most data, local latency | 7.3 |
| Failover and split brain | DNS switch only, manual promotion, lease-guarded ownership | Lease-guarded ownership with automatic traffic shift | An isolated region stops writing before another takes over | 7.4 |
| Residency | Global replication with access controls, geography-bounded data | Geography-bounded data, global metadata without personal data | Easier to prove and audit | 7.6 |

### 7.1 Data classes and conflict strategies

| Class | Examples | Strategy | RPO on region loss | Write latency |
|---|---|---|---|---|
| A. Tenant-owned | Projects, tasks, comments | Single writer: tenant's home region; async replication in the geography | Replication lag (*Example:* 1 s) | Local, or one hop to home region |
| B. Concurrently updated shared fields | View counts, reactions, tag sets | CRDT: PN-counter, observed-remove set, last-writer-wins register with hybrid logical clocks | Lag, but merges never lose concurrent updates | Local |
| C. Must never conflict | Unique workspace names, billing credits, invoice numbers | Consensus across three regions in the geography | Zero | Quorum round trip (*Example:* about 12 ms in the EU) |
| D. Reference data | Templates, feature flags, price lists | Single global writer, read-only replicas | Not applicable (re-publishable) | Writer region only |
| E. Derived | Search indexes, caches, aggregates | Rebuilt per region from the change stream | Rebuild time | Not applicable |

Rules of thumb: put data in class A unless several regions must write the
same record at the same time; use class B only for data types with a proven
merge (counters, sets, registers); keep class C small because every write
pays a cross-region round trip; never replicate class E.

### 7.2 Why plain last-writer-wins is not enough

*Example:* two regions read a counter at 10 and each add 1. With
last-writer-wins on the whole value, both write 11 and one increment is
lost; a PN-counter keeps one count per region and sums them, giving 12. For
a unique name, two regions may each accept "acme" for different tenants
within the replication lag; last-writer-wins silently gives the name to one
of them after both were told they own it. Class C exists for this case.
Where last-writer-wins is acceptable (a single user editing their own
profile field), timestamps come from hybrid logical clocks so that clock skew
between regions cannot make an older write win by more than the skew bound.

### 7.3 Write partitioning and the home directory

Each tenant has exactly one home region with a lease from the geography's
consensus group. The lease carries an epoch number and an expiry (*Example:*
10 s, renewed every 3 s). Every write in the home region checks that the
local lease is valid and stamps the epoch on the commit; replicas reject
changes with an older epoch than the one they already applied. Reads are
served locally from any region in the geography; read-your-writes uses the
consistency token in 4.2.

### 7.4 Region evacuation and split brain

```mermaid
sequenceDiagram
  participant H as Health checks
  participant E as Evacuation controller
  participant T as Traffic manager
  participant D as Home directory (consensus)
  participant R1 as eu-1 (failing)
  participant R2 as eu-2
  H->>E: eu-1 failed 3 checks in 30 s
  E->>T: set eu-1 weight to 0
  T-->>E: new requests go to eu-2 and eu-3
  Note over R1,D: eu-1 cannot renew leases (no quorum)
  R1->>R1: lease expires after 10 s, stop accepting writes
  E->>D: wait lease expiry plus 5 s safety margin
  E->>D: grant eu-1 tenants to eu-2 and eu-3 (epoch + 1)
  D-->>R2: leases granted
  R2->>R2: apply replicated changes up to last position, accept writes
  E-->>H: evacuation complete, alert on-call to approve failback later
```

An isolated eu-1 that is still alive cannot reach a majority of the
consensus group, so it cannot renew leases and stops writing within 10 s.
Other regions take over only after the lease has expired plus a safety
margin (15 s in total), so two regions never accept writes for the same
tenant. Timeline for NFR-02: 30 s detection, traffic shift without DNS
caching on an anycast edge (or up to the DNS TTL, *Example:* 60 s, with DNS
routing), 30 s to drain in-flight requests; writes resume after the 15 s
lease wait, which overlaps the drain. Failback is manual: data written in
eu-2 and eu-3 replicates back, and tenants move home only after lag is zero.

### 7.5 Removing shared dependencies

| Dependency | Risk | Change |
|---|---|---|
| Identity provider in one region | Login fails everywhere | Token signing keys and session validation in every region; issuer endpoints per geography |
| Global configuration push | A bad config breaks all regions at once | Staged rollout one region at a time with health gates |
| Deployment pipeline | Same bad build in all regions | Deploy waves: one region, bake time, then the rest of the geography, then the other geography |
| DNS provider | Routing fails globally | Anycast edge plus a secondary DNS provider; long-lived records for static endpoints |
| Control-plane APIs during failover | Provider control plane degraded in the same event | Pre-provisioned capacity (3.3); failover uses only data-plane actions such as health-check switches |

### 7.6 Data residency

<!-- Guidance: Laws are named here as dated examples only. Record which apply to you with your privacy team; this section is not legal advice. -->

Residency is enforced by architecture, not by access control alone: EU-homed
tenant data (classes A, B, C and E) lives only in EU regions, including
backups, logs, traces and support tooling. Global components (edge routing,
reference data, the cross-geography directory) carry no personal data,
only tenant IDs and geography codes.

| Jurisdiction (examples, checked October 2026) | Instrument | Questions for your privacy and legal team |
|---|---|---|
| EU | GDPR (Regulation (EU) 2016/679), Chapter V, Articles 44 to 50 on transfers to third countries | Is any remote access from outside the EU (support, on-call, vendors) a transfer? Which transfer tool applies (adequacy decision, standard contractual clauses)? Is a transfer impact assessment needed? |
| UK | UK GDPR restricted transfers (ICO guidance "A brief guide to international transfers", updated 15 January 2026) | Do UK data flows rely on UK adequacy regulations, the IDTA or the Addendum? Are UK tenants treated as their own geography? |
| US | Department of Justice Data Security Program, 28 CFR Part 202, in effect from 8 April 2025 (some due-diligence, audit and reporting duties from 5 October 2025) | Could any vendor, employee or investor arrangement give a country of concern access to bulk sensitive personal data or government-related data? |

Do not treat this table as complete: sector rules, contracts and other
countries may impose their own requirements. Check the current official
text and a qualified professional.

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). The AWS whitepaper
> "Disaster Recovery of Workloads on AWS" separates backup and restore,
> pilot light, warm standby and multi-site active/active. It notes that in
> active/active there is no failover in the usual sense, but that data
> corruption or deletion still needs point-in-time backups and so still has
> a non-zero recovery point. For writes it names three patterns: write
> global (one writer region, promote another on failure), write local (every
> region writes, with last-writer-wins reconciliation in DynamoDB global
> tables) and write partitioned (each record has an assigned region, which
> avoids conflicts). It also advises using data-plane operations rather than
> control-plane operations during failover. The DynamoDB global tables
> documentation describes two modes: eventual consistency across Regions,
> usually replicated within a second, with per-item last-writer-wins and
> transactions atomic only in the Region where they ran; and strong
> consistency across Regions, which must be deployed in exactly three
> Regions (three replicas, or two replicas and a witness), gives an RPO of
> zero, and rejects a write to an item that is being modified in another
> Region. The Azure Well-Architected disaster recovery guide ties
> active-active to its mission-critical tier (SLO above 99.99%, RTO in
> seconds). This design uses write partitioning for most data (7.3) and a
> three-region consensus group for the data that cannot conflict (7.1).
> Read the originals in 12.3.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Regional capacity | Each region sized for N+1 at 70% utilisation | *Example:* about 47 000 requests/s per EU region |
| Hot tenant | Tenant partitioning, cells inside a region | One tenant above a cell's capacity |
| Consensus writes | Kept to class C only | *Example:* a few thousand writes/s per consensus group |
| Replication lag | Parallel apply by tenant partition | Lag above 1 s at sustained peak writes |
| Tenant moves | Range copy plus lease handover | Large tenants need a throttled copy over hours |

Latency budget for a write from an EU user whose home region is not the
nearest (target p95 150 ms):

| Step | Budget |
|---|---|
| User to nearest EU region | 30 ms |
| Forward to home region inside the EU | 15 ms |
| Lease check and local commit | 10 ms |
| Response back through the nearest region | 45 ms |
| Headroom | 50 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Region | Full outage | Health checks from several vantage points | Tenants homed there lose up to 1 s of writes; all users rerouted | Evacuation (7.4), N+1 capacity | *Example:* 2 min |
| Inter-region network | Partition isolates one region | Lease renewal failures, lag | Isolated region stops writes after 10 s | Leases and epochs prevent split brain | *Example:* 2 min |
| Replication | Lag grows | Lag per tenant partition | RPO exceeds target | Alert at 1 s, shed background writes, pause tenant moves | *Example:* minutes |
| Consensus group | Loses majority (two regions down) | Quorum metrics | Class C writes in that geography stop | Accept: reads continue, class A writes continue where leases are held | *Example:* region recovery |
| Global deploy | Bad build reaches all regions | Error rate after each wave | All regions if waves are skipped | Wave deploys with bake time, automatic rollback | *Example:* 10 min |
| Data | Corruption or malicious deletion replicates everywhere | Integrity checks, audit | All replicas of the affected tenant | Point-in-time backups per geography, delayed-apply replica | *Example:* hours |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

| Threat | Control |
|---|---|
| Compromised credentials in one region used everywhere | Separate accounts or projects per region, scoped roles |
| Replication channel interception | Mutual TLS on the backbone, signed change batches |
| Residency bypass through support tools | Support tooling deployed per geography; access logged and reviewed |
| Failover abused to force data out of a geography | Evacuation controller cannot target regions outside the tenant's geography |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Replication lag | p99 lag per source and destination region | Above 1 s for 5 min |
| Lease health | Lease renewals failed per region | Any sustained failures for 30 s |
| Regional headroom | Peak utilisation per region | Above 50% in normal operation |
| Residency | Data-flow check results, cross-geography traffic with personal data | Any violation |
| Evacuation readiness | Last successful drill per region | Older than 90 days |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Regional stamps | [$/region-month] | *Example:* 6 regions sized for N+1 | [$] |
| Inter-region transfer | [$/GB] | *Example:* about 6.8 TB per EU destination, 5.6 TB per US destination, plus reverse directions | [$] |
| Consensus store | [$/month] | 2 groups of 3 regions | [$] |
| Global edge and traffic manager | [$/month] | 1 | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Topology | Active-passive warm standby; active-active with 2 regions; with 3 regions per geography | 3 regions per geography | 25% less capacity than 2 regions, majority for consensus | More regions to operate and deploy |
| Conflict handling | Last-writer-wins everywhere; single global writer; per data class | Per data class | Correct for money and names, fast for the rest | Developers must classify data |
| Write topology | Write global; write local; write partitioned | Write partitioned by tenant | No conflicts for most data | Writes from a non-home region pay one extra hop |
| Failover trigger | Manual; automatic traffic plus lease-guarded writes | Automatic, lease-guarded | Meets a 2 min RTO without split brain | False positives move traffic needlessly |
| Residency | Global data with access controls; geography-bounded data | Geography-bounded | Easier to prove | No cross-geography disaster recovery for EU data |
| What changes at 10x / 0.1x | 10x (1.2 million requests/s, 120 000 writes/s): cells inside each region with tenants assigned to cells, more geographies, and consensus groups split by key range. 0.1x (12 000 requests/s): active-active rarely pays; use one active region with a warm standby in the same geography and accept an RTO of tens of minutes | Review at each doubling of peak traffic or when a new residency requirement appears | Operational load grows faster than compute | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Data classification of every store; home directory and leases in one geography | Every table has a class and an owner | YYYY-MM-DD |
| M2 | Second and third EU regions active for read traffic | Read latency and lag targets met for 14 days | YYYY-MM-DD |
| M3 | Write partitioning, evacuation controller, first evacuation drill | Drill meets RTO and RPO with no duplicate writes | YYYY-MM-DD |
| M4 | US geography; residency checks automated | Privacy sign-off | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Regions join in read-only mode first. Tenants are moved to their
target home region in batches of 1% with lease handover, starting with
internal tenants. Rollback moves tenants back to the original region with the
same handover, since every region already holds a replica.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* A store is misclassified and conflicts silently lose data | Medium | High | Review of classes, conflict counters per table | [@name] |
| R-02 | *Example:* Regions drift in configuration | Medium | Medium | Same infrastructure code for every stamp, drift detection | [@name] |
| R-03 | *Example:* Residency rules change for a geography | Low | High | Geography is a parameter of the stamp; privacy review each quarter | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do customers outside the EU and US need their own geography? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which observability data contains personal data and must stay in the geography? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Home geography | The legal and routing area where a tenant's data lives |
| Home region | The one region that accepts writes for a tenant |
| Lease and epoch | Time-limited right to write for a tenant, numbered so older owners are rejected |
| CRDT | Conflict-free replicated data type: a data type whose replicas merge without coordination |
| Hybrid logical clock | Timestamp that combines physical time with a logical counter to order events across regions |
| Split brain | Two regions both believing they own the same data |
| RPO / RTO | Data loss allowed / time to recover |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| ["Disaster recovery options in the cloud"](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html), AWS whitepaper "Disaster Recovery of Workloads on AWS", checked October 2026 | 7.4, 7.5, 7.7 |
| ["How DynamoDB global tables work"](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/V2globaltables_HowItWorks.html), Amazon DynamoDB Developer Guide, checked October 2026 | 7.1, 7.7 |
| ["Develop a disaster recovery plan for multi-region deployments"](https://learn.microsoft.com/en-us/azure/well-architected/design-guides/disaster-recovery), Azure Well-Architected Framework, page updated 30 July 2026 | Criticality tiers, 7.7 |
| ["Well-Architected Framework: Reliability pillar"](https://docs.cloud.google.com/architecture/framework/reliability), Google Cloud, updated 27 October 2026 | Reliability principles |
| [Regulation (EU) 2016/679 (GDPR)](https://eur-lex.europa.eu/eli/reg/2016/679/oj), Chapter V, Articles 44 to 50 | 7.6 |
| ["A brief guide to international transfers"](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/international-transfers/international-transfers-a-guide/), ICO, updated 15 January 2026 | 7.6 |
| [Data Security Program](https://www.justice.gov/nsd/data-security), US Department of Justice National Security Division, 28 CFR Part 202 | 7.6 |
| Disaster Recovery Plan (`ops/disaster-recovery-plan`) | Backups and corruption recovery |
| Game Day Plan (`ops/game-day-plan`) | Evacuation drills |
| Distributed Lock and Coordination Service System Design (`system-design/design-distributed-lock-coordination`) | Leases and fencing |
| Distributed Key-Value Store System Design (`system-design/design-distributed-kv-store`) | Replication and consensus |
| Transfer Impact Assessment (`compliance/transfer-impact-assessment`) | Cross-border transfer review |
| Sovereign Cloud Architecture (`architecture/sovereign-cloud-architecture`) | Sovereignty requirements beyond residency |

## Standard and licence note

Based on current public cloud-provider guidance (AWS disaster recovery
whitepaper and DynamoDB documentation, Azure Well-Architected Framework,
Google Cloud Well-Architected Framework, all checked in October 2026) and on
common industry practice for multi-region systems. Provider documents are
paraphrased in 7.7 only; no text or figures were copied. Laws in 7.6 are
named as dated examples, not as a complete list, and this template is not
legal advice: check the current official text and a qualified professional.
All capacity numbers are examples computed from the stated assumptions.
Original text written for this collection. Template licence: MIT.