Multi-Region Active-Active System Design Template (Reference Design)

System Design: [Service name] multi-region active-active

Reference design for running one service actively in several cloud regions at once: regional topology and capacity for the loss of a region, data classes and how each resolves conflicting writes (home-region ownership, CRDTs, consensus across regions), traffic routing and evacuation, split brain, and data residency. It draws on current provider guidance (AWS "Disaster Recovery of Workloads on AWS" whitepaper and DynamoDB global tables documentation, Azure Well-Architected Framework disaster recovery guide updated 2026, Google Cloud Well-Architected Framework reliability pillar updated October 2026). It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks. Section 7.6 names data-protection laws as dated examples; it is a documentation format, not legal advice: check the current official text and a qualified professional.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: A project-management SaaS runs in one region. A four-hour regional outage last year stopped every customer, enterprise contracts now ask for a 99.99% monthly SLO, users in the EU and the US see 150 ms or more of extra round-trip time to the far region, and EU customers require that their personal data is stored and processed in EU regions.]

1.2 Proposed solution

[Example: Two home geographies, EU and US, each with three regions that all serve traffic. Every tenant has a home geography chosen at sign-up and a home region inside it. User-owned data is written only in the tenant's home region and replicated asynchronously to the other two regions of the same geography (write partitioned). Shared, concurrently edited fields use CRDTs. A small set of data that must never conflict (unique names, billing credits) uses a consensus group spread over the three regions of the geography. Global reference data has one writer and is replicated read-only everywhere. A global traffic layer routes users to the nearest healthy region in their home geography; a region that loses its write lease stops accepting writes before another region takes over its tenants.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer, "every region accepts every write and replication is last-writer-wins", and show the lost update it causes on a counter and on a unique username. Then split the data into classes and give each the cheapest strategy that is still correct.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: recovery from data corruption or malicious deletion (backups and point-in-time restore, see the disaster recovery plan)]
• [Example: a third home geography (design keeps it possible)]
• [Example: multi-cloud deployment of the same service]
• [Example: real-time collaborative editing across geographies]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Traffic per geography

• EU peak: $0.55 \times 120\,000 = 66\,000$ requests/s, of which 6 600 writes/s.
• US peak: $0.45 \times 120\,000 = 54\,000$ requests/s, of which 5 400 writes/s.
• Normal share per region with three regions: EU 22 000/s, US 18 000/s.

3.3 Capacity for the loss of one region

Each region must carry the geography's peak divided by the regions that remain:

For the US the same formula gives about 38 571 requests/s per region. In normal operation an EU region runs at $22\,000 / 47\,143 \approx 47\%$ of its capacity. With only two regions per geography each region would need $66\,000 / 0.7 \approx 94\,286$ requests/s and would run at 35%; total EU capacity would be 188 571 instead of 141 429 requests/s. Consequence: three regions per geography need 25% less total capacity than two, and they give a majority for the consensus data in 7.2.

3.4 Replication bandwidth and data at risk

At an average of 40% of peak this is about 6.8 TB a month per destination region (about 5.6 TB for the US). With asynchronous replication, each EU region accepts about $6\,600 / 3 = 2\,200$ writes per second, so a lag of 1 s means about 2 200 acknowledged writes at risk if that region is lost abruptly (1 800 for a US region). Consequence: the RPO for home-region data is the replication lag, so lag is an alerting signal (9.3), and data that cannot accept any loss goes to the consensus class.

3.5 Availability arithmetic

If a region is available 99.9% of the time and region failures were independent, the chance that at least two of three regions are up is

A single shared dependency (global DNS provider, identity service, a deployment pipeline that pushes to all regions at once) with 99.99% availability multiplies this down to about $0.999997 \times 0.9999 \approx 0.99990$. Consequence: the shared dependencies, not the region count, decide whether 99.99% is met, so the design removes or regionalises them (7.5).

3.6 Summary

4. API design

4.1 Interfaces

4.2 Routing and consistency behaviour

• The edge reads the tenant from the credential or the URL, looks up the home geography in a cached copy of the directory and forwards the request to the nearest healthy region of that geography. Requests never leave the home geography.
• A write in a non-home region is forwarded to the home region over the private backbone (one extra round trip inside the geography).
• A read that carries a consistency token is served from the local replica only when the replica has applied that position; otherwise it waits up to Example: 200 ms, then forwards to the home region.

4.3 Error during a write ownership move

HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2

{"type": "https://api.example.com/problems/home-region-moving", "title": "Tenant is moving to another region", "status": 503, "detail": "Writes for this workspace are paused for a few seconds while its home region changes. Reads continue. Retry the same request with the same Idempotency-Key.", "instance": "/v1/projects/prj_81c2/tasks"}

Clients send an Idempotency-Key on every write so a retry after a failover never applies a change twice.

5. Data model

5.1 Entities

Ownership is explicit: every tenant row points at one home region and the lease that allows that region to write.

5.2 Storage choices

5.3 Access patterns

6. High-level design

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Data classes and conflict strategies

Rules of thumb: put data in class A unless several regions must write the same record at the same time; use class B only for data types with a proven merge (counters, sets, registers); keep class C small because every write pays a cross-region round trip; never replicate class E.

7.2 Why plain last-writer-wins is not enough

Example: two regions read a counter at 10 and each add 1. With last-writer-wins on the whole value, both write 11 and one increment is lost; a PN-counter keeps one count per region and sums them, giving 12. For a unique name, two regions may each accept "acme" for different tenants within the replication lag; last-writer-wins silently gives the name to one of them after both were told they own it. Class C exists for this case. Where last-writer-wins is acceptable (a single user editing their own profile field), timestamps come from hybrid logical clocks so that clock skew between regions cannot make an older write win by more than the skew bound.

7.3 Write partitioning and the home directory

Each tenant has exactly one home region with a lease from the geography's consensus group. The lease carries an epoch number and an expiry (Example: 10 s, renewed every 3 s). Every write in the home region checks that the local lease is valid and stamps the epoch on the commit; replicas reject changes with an older epoch than the one they already applied. Reads are served locally from any region in the geography; read-your-writes uses the consistency token in 4.2.

7.4 Region evacuation and split brain

An isolated eu-1 that is still alive cannot reach a majority of the consensus group, so it cannot renew leases and stops writing within 10 s. Other regions take over only after the lease has expired plus a safety margin (15 s in total), so two regions never accept writes for the same tenant. Timeline for NFR-02: 30 s detection, traffic shift without DNS caching on an anycast edge (or up to the DNS TTL, Example: 60 s, with DNS routing), 30 s to drain in-flight requests; writes resume after the 15 s lease wait, which overlaps the drain. Failback is manual: data written in eu-2 and eu-3 replicates back, and tenants move home only after lag is zero.

7.5 Removing shared dependencies

7.6 Data residency

Residency is enforced by architecture, not by access control alone: EU-homed tenant data (classes A, B, C and E) lives only in EU regions, including backups, logs, traces and support tooling. Global components (edge routing, reference data, the cross-geography directory) carry no personal data, only tenant IDs and geography codes.

Do not treat this table as complete: sector rules, contracts and other countries may impose their own requirements. Check the current official text and a qualified professional.

7.7 How real systems do it

How real systems do it (paraphrased, not copied). The AWS whitepaper "Disaster Recovery of Workloads on AWS" separates backup and restore, pilot light, warm standby and multi-site active/active. It notes that in active/active there is no failover in the usual sense, but that data corruption or deletion still needs point-in-time backups and so still has a non-zero recovery point. For writes it names three patterns: write global (one writer region, promote another on failure), write local (every region writes, with last-writer-wins reconciliation in DynamoDB global tables) and write partitioned (each record has an assigned region, which avoids conflicts). It also advises using data-plane operations rather than control-plane operations during failover. The DynamoDB global tables documentation describes two modes: eventual consistency across Regions, usually replicated within a second, with per-item last-writer-wins and transactions atomic only in the Region where they ran; and strong consistency across Regions, which must be deployed in exactly three Regions (three replicas, or two replicas and a witness), gives an RPO of zero, and rejects a write to an item that is being modified in another Region. The Azure Well-Architected disaster recovery guide ties active-active to its mission-critical tier (SLO above 99.99%, RTO in seconds). This design uses write partitioning for most data (7.3) and a three-region consensus group for the data that cannot conflict (7.1). Read the originals in 12.3.

8. Scalability and performance

Latency budget for a write from an EU user whose home region is not the nearest (target p95 150 ms):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

9.3 Observability

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Regions join in read-only mode first. Tenants are moved to their target home region in batches of 1% with lease handover, starting with internal tenants. Rollback moves tenants back to the original region with the same handover, since every region already holds a replica.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on current public cloud-provider guidance (AWS disaster recovery whitepaper and DynamoDB documentation, Azure Well-Architected Framework, Google Cloud Well-Architected Framework, all checked in October 2026) and on common industry practice for multi-region systems. Provider documents are paraphrased in 7.7 only; no text or figures were copied. Laws in 7.6 are named as dated examples, not as a complete list, and this template is not legal advice: check the current official text and a qualified professional. All capacity numbers are examples computed from the stated assumptions. Original text written for this collection. Template licence: MIT.