# System Design: [API gateway name]

> Reference design for an edge API gateway: the shared front door that
> terminates client connections, authenticates callers, routes requests to
> backend services and applies cross-cutting policy. It covers routing,
> token validation, the rate-limit hook, request and response filters,
> retries with budgets, adaptive concurrency limits, connection scaling for
> long-lived and push connections, multi-region failover, observability and
> safe configuration rollout. It follows the structure of the System Design
> Document template, cites HTTP Semantics (RFC 9110), JSON Web Token best
> practice (RFC 8725) and Problem Details (RFC 9457), and is pre-filled with
> a worked design to adapt. Replace every `[bracketed]` value and every row
> marked *Example*; all numbers are illustrative assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Decide first what the gateway owns and what it must never own. A gateway
  that accumulates business logic becomes the slowest-moving, riskiest
  deploy in the company. Write the ownership boundary in 1.2 and 2.3.
- Size the fleet by the larger of two numbers: requests per second and open
  connections (section 3). Push and streaming clients often make connections
  the binding constraint.
- Treat configuration as the most dangerous input. Most gateway outages are
  caused by a bad route or filter change, not by load (7.6).
- Keep the 45-minute interview path in 1.4 even for a real review: it tells
  reviewers where the hard problems are.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-GW-001] |
| System / feature | [Edge API gateway] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Traffic in scope | [Public API, mobile apps, web, partner traffic] |
| Related | Rate Limiter System Design Template (`system-design/design-rate-limiter`), API Design Guidelines Template (`architecture/api-design-guidelines`), [Identity provider design] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, edge platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| Identity team | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced fixed retry counts with a per-origin retry budget |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added staged configuration rollout with automatic rollback |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Clients call 140 backend services through a mix of load
balancers and per-team proxies. Each team implements its own token checks,
so three services accepted expired tokens last quarter. Retries are
configured per client, and during the last origin incident retry storms
tripled traffic to the failing service and stretched a 4-minute degradation
into a 40-minute outage. Mobile push connections are held by a separate
legacy fleet that cannot be scaled without a release.]

### 1.2 Proposed solution

[*Example:* A single gateway fleet per region terminates TLS and HTTP/1.1,
HTTP/2 and long-lived push connections on a non-blocking event-loop
runtime. Each request passes through an ordered chain of inbound filters
(identity, token validation, rate-limit hook, request normalisation), an
endpoint stage that selects an origin cluster and proxies the request with
pooled connections, and outbound filters (headers, metrics, error mapping).
Retries are allowed only for idempotent requests and only within a retry
budget per origin. Concurrency limits adapt to observed origin latency.
Routes and filters are declared as versioned configuration, validated in
CI, and rolled out in stages with automatic rollback. The gateway owns
cross-cutting policy only; it never contains business logic.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Validate tokens locally or call the identity service per request? | Local signature validation with cached keys; introspection only for opaque partner tokens | 7.2 |
| Q-02 | *Example:* How do we bound retries? | Retry budget of 10% of origin requests plus per-request cap of 2 attempts | 7.3 |
| Q-03 | *Example:* Fixed or adaptive concurrency limits? | Adaptive per origin, with a static ceiling | 7.4 |
| Q-04 | *Example:* How do we ship route changes safely? | Config as code, canary to one cell, automatic rollback on error-rate delta | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order in an interview or a time-boxed review. Spend the most time on one deep dive rather than touching all of them. -->

| Minutes | Cover | Key point to land |
|---|---|---|
| 0 to 5 | Scope: what the gateway owns (auth, routing, policy) and what it does not (business logic) | Clarify connection types: request/response only, or also push and streaming |
| 5 to 10 | Core requirements and the two sizing numbers: peak requests per second and concurrent connections | Connections can dominate fleet size |
| 10 to 15 | API surface: route config, error format, propagated identity headers | The gateway contract is mostly configuration |
| 15 to 22 | High-level diagram: load balancer, gateway cells, filter chain, origin pools, config plane | Data plane and control plane are separate |
| 22 to 37 | One deep dive: retries with budgets and adaptive concurrency (7.3, 7.4) is the strongest choice; token validation (7.2) or config rollout (7.6) are good alternatives | Show how the gateway avoids amplifying an origin failure |
| 37 to 42 | Failure modes and multi-region failover | Blast radius per cell and per origin |
| 42 to 45 | Trade-offs and what changes at 10x | Where the design breaks first |

## 2. Requirements

<!-- Guidance: Separate core requirements (the design must meet them) from below-the-line items (explicitly out of scope). Non-functional targets need a number and a way to measure it. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Route requests by host, path prefix, method and header to an origin cluster | Must |
| FR-02 | *Example:* Validate access tokens and pass a verified identity to origins | Must |
| FR-03 | *Example:* Call the rate limiter for every request and return 429 on rejection | Must |
| FR-04 | *Example:* Apply ordered inbound and outbound filters per route | Must |
| FR-05 | *Example:* Retry idempotent requests on connection failure or 503, within a budget | Must |
| FR-06 | *Example:* Hold long-lived connections for server push to mobile and web clients | Must |
| FR-07 | *Example:* Change routes and filters without redeploying the gateway | Must |
| FR-08 | *Example:* Shift traffic between regions on regional failure | Should |
| FR-09 | *Example:* Shadow (mirror) a share of traffic to a new origin version | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Added latency | *Example:* p99 gateway overhead at or below 5 ms excluding origin time | Span timing in the gateway |
| NFR-02 | Availability | *Example:* 99.99% monthly for the gateway itself | Synthetic probes per region |
| NFR-03 | Throughput | *Example:* 1.2 million requests/s globally at peak | Load test |
| NFR-04 | Connections | *Example:* 20 million concurrent client connections globally | Connection gauge |
| NFR-05 | Isolation | *Example:* One failing origin does not raise latency for other origins | Fault injection test |
| NFR-06 | Config safety | *Example:* A bad config is rolled back automatically within 5 minutes | Game day |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* business logic, request aggregation or response composition for specific screens (owned by backend-for-frontend services)]
- [*Example:* layer 3 and 4 volumetric DDoS protection (edge provider)]
- [*Example:* user login, consent and token issuance (identity provider)]
- [*Example:* service-to-service east-west traffic (service mesh)]
- [*Example:* the rate limiter's own algorithms and storage (see Rate Limiter System Design Template, `system-design/design-rate-limiter`)]

## 3. Capacity estimation

<!-- Guidance: Every estimate must end in a design consequence. If a number changes nothing, drop it. Round generously and say which way you rounded. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Peak requests per second, global | *Example:* 1,200,000 | Edge metrics x 1.5 growth |
| Regions, active-active | *Example:* 3 | Deployment plan |
| Concurrent client connections, global | *Example:* 20 million (mostly idle push connections) | Mobile connection gauge |
| Requests per instance at 60% CPU | *Example:* 12,000/s (20,000/s at saturation) | Benchmark on target instance |
| Memory per idle connection incl. TLS state | *Example:* 20 KB | Heap profile |
| Average request plus response size | *Example:* 10 KB | Access logs |
| Access log record | *Example:* 1 KB | Log schema |

### 3.2 Fleet size

Per region the peak is 1,200,000 / 3 = 400,000 requests/s. At 12,000
requests/s per instance this needs 400,000 / 12,000 = 33.3, so 34 instances.

Connections per region are 20 million / 3 = 6.67 million. If one instance
holds 200,000 connections (4 GB of connection state at 20 KB each), this
also needs 6.67 million / 200,000 = 33.4, so 34 instances.

**Consequence:** both constraints land on about 34 instances, so plan 36 per
region (12 per zone across 3 zones) and size each instance for memory as
well as CPU. A region must also absorb a failed region's share: with 3
regions each survivor takes 50% more, so either run 54 per region or accept
pre-warmed scale-out within 10 minutes during failover (9.1).

### 3.3 Bandwidth

400,000 requests/s x 10 KB = 4 GB/s, about 32 Gbit/s per region through
the gateway.

**Consequence:** each instance carries about 1 Gbit/s at peak, well under a
typical NIC, but the regional load balancer tier must be provisioned for
32 Gbit/s plus failover headroom; enable response compression on the
gateway only for text bodies above a size threshold.

### 3.4 Token validation cost

If every request carries a signed token and a signature check costs
*Example:* 30 microseconds of CPU, then 400,000 x 30 microseconds = 12 CPU
seconds per second per region, about 12 cores spread across the fleet.

**Consequence:** local validation is affordable (one third of a core per
instance), so there is no need to call the identity service on the hot path.
A small cache of verified token digests for their remaining lifetime (capped
at 60 s) halves the cost for chatty clients.

### 3.5 Logs

400,000 requests/s x 1 KB = 400 MB/s per region, about 34.6 TB per day per
region.

**Consequence:** log 100% of errors and slow requests but sample successful
requests at 5%; rely on metrics and traces for aggregate views.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Requests per region | 400,000/s | 36 instances per region, 3 zones |
| Connections per region | 6.67 million | Memory-sized instances; connection draining matters for deploys |
| Bandwidth per region | about 32 Gbit/s | Provision load balancers with failover headroom |
| Token checks | about 12 cores per region | Local validation, no network call |
| Logs | 34.6 TB per day per region | Sample successes, keep all errors |

## 4. API design

<!-- Guidance: A gateway has three contracts: the client-facing behaviour (errors, headers), the origin-facing contract (identity and tracing headers), and the configuration API. Define all three. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Client-facing | Any route declared in config | Proxy to the selected origin |
| Client-facing | `GET /.well-known/gateway-health` | Load balancer health check (not exposed publicly) |
| Config API | `POST /v1/config/changes` | Submit a validated config bundle for rollout |
| Config API | `GET /v1/config/active?cell=` | Active config version per cell |
| Config API | `POST /v1/config/rollback` | Roll back a cell or all cells to a previous version |
| Admin API | `POST /v1/origins/{id}/drain` | Stop sending new requests to an origin pool |

### 4.2 Route definition

Routes are declared as code in a repository and compiled into a bundle:

```yaml
route: orders-v2
version: 41
match:
  host: api.example.com
  path_prefix: /v2/orders
  methods: [GET, POST]
auth:
  required: true
  audiences: ["orders-api"]
  scopes_by_method: { GET: ["orders:read"], POST: ["orders:write"] }
rate_limit:
  policy: per-key-default
origin:
  cluster: orders-service
  timeout_ms: 2000
  retry: { on: [connect_failure, http_503], max_attempts: 2, idempotent_only: true }
filters:
  inbound: [strip-internal-headers, request-id, auth, rate-limit, normalize-path]
  outbound: [security-headers, problem-details, metrics]
```

### 4.3 Origin-facing contract

| Header | Set by | Meaning |
|---|---|---|
| `X-Request-Id` | Gateway | Unique ID for the request, echoed to the client |
| `traceparent` | Gateway or client | W3C Trace Context propagation |
| `X-Verified-Subject` | Gateway | Verified subject; any client-supplied copy is stripped |
| `X-Verified-Scopes` | Gateway | Space-separated scopes from the validated token |
| `X-Attempt` | Gateway | Attempt number, so origins can spot retries |

Origins trust these headers only on connections from the gateway (mutual
TLS between gateway and origin).

### 4.4 Errors

Gateway-generated errors use Problem Details (RFC 9457) and the status
codes of RFC 9110:

```http
HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2

{"type": "https://api.example.com/problems/origin-overloaded", "title": "Service temporarily overloaded", "status": 503, "detail": "The orders service is shedding load. Retry after the indicated delay.", "instance": "/v2/orders", "request_id": "req_7f3a"}
```

| Situation | Status | Retry hint |
|---|---|---|
| No route matches | 404 | None |
| Missing or invalid token | 401 with `WWW-Authenticate` | None |
| Valid token, insufficient scope | 403 | None |
| Rate limited | 429 | `Retry-After` |
| Concurrency limit exceeded at gateway | 503 | `Retry-After` |
| Origin timeout | 504 | None by default |

## 5. Data model

<!-- Guidance: The gateway is mostly stateless on the request path. Show the state it does hold, where it lives, and the access pattern that each piece must serve. -->

### 5.1 Entities

All durable state belongs to the control plane; the data plane holds a
read-only compiled copy of it.

```mermaid
erDiagram
  CONFIG_BUNDLE {
    string bundle_id PK
    int version
    string checksum
    string author
    datetime created_at
    string status
  }
  ROUTE {
    string route_id PK
    int version
    string bundle_id FK
    string match_expr
    string origin_cluster FK
  }
  ORIGIN_CLUSTER {
    string cluster_id PK
    string discovery_name
    int timeout_ms
    float retry_budget_ratio
  }
  FILTER_BINDING {
    string route_id FK
    string filter_name
    int position
    string params_json
  }
  ROLLOUT {
    string rollout_id PK
    string bundle_id FK
    string stage
    string state
  }
  CONFIG_BUNDLE ||--o{ ROUTE : contains
  ROUTE }o--|| ORIGIN_CLUSTER : "proxies to"
  ROUTE ||--o{ FILTER_BINDING : applies
  CONFIG_BUNDLE ||--o{ ROLLOUT : "rolled out by"
```

### 5.2 Storage choices

| Data | Store | Why |
|---|---|---|
| Config bundles and history | Git repository plus an artifact store of compiled bundles | Review, diff and audit for free |
| Rollout state | Small relational database in the control plane | Transactions on stage changes |
| Active config on each instance | In memory, plus last-good bundle on local disk | Gateway starts even if the control plane is down |
| Signing keys (JWKS) | In memory, refreshed from the identity provider | Local token validation |
| Concurrency and retry counters | In memory per instance | Decisions must be sub-microsecond |

### 5.3 Access patterns

| Query | Served by | Key or index | Partition rationale |
|---|---|---|---|
| Find the route for a request | Compiled route trie in memory | Host, then path prefix, then method | No partitioning: every instance holds all routes (*Example:* 2,000 routes, 5 MB) |
| Find the signing key for a token | JWKS cache | `kid` header of the token | Small set, replicated to all instances |
| Pick an origin host | Origin pool in memory | Cluster ID, then load-balancing choice | Pools refreshed from service discovery |
| Which bundle is active in cell X | Rollout table | `cell_id`, `stage` | Single small table; read by control plane only |
| What changed between versions | Git history | Commit range | Not on the request path |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request path and where state lives. Keep the control plane visibly separate from the data plane. -->

```mermaid
flowchart LR
  client([Clients]) --> dns[Geo DNS / anycast]
  dns --> lb[Regional L4 load balancer]
  subgraph cell["Gateway cell (one of several per region)"]
    gw1[Gateway instance]
    gw2[Gateway instance]
  end
  lb --> gw1
  lb --> gw2
  gw1 --> rl[Rate limiter]
  gw1 --> pools[Origin connection pools]
  pools --> o1[Orders service]
  pools --> o2[Catalog service]
  pools --> o3[Push session service]
  idp[Identity provider JWKS] -.keys.-> gw1
  sd[Service discovery] -.hosts.-> pools
  subgraph cp["Control plane"]
    repo[Config repo] --> ci[Validate and compile]
    ci --> ro[Rollout controller]
  end
  ro -.bundles.-> gw1
  ro -.bundles.-> gw2
  gw1 --> tel[[Metrics, traces, sampled logs]]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Geo DNS / anycast | Send clients to the nearest healthy region | Provider |
| L4 load balancer | Spread connections across gateway instances | Provider capacity |
| Gateway instance | Terminate TLS, run filter chain, proxy | Instances per cell |
| Gateway cell | Group of instances that receives a config stage together | Cells per region |
| Origin connection pools | Reuse connections to origins, apply limits | With gateway |
| Rate limiter | Quota decisions (separate design) | Its own shards |
| Control plane | Validate, compile and roll out config | Single service per region, off the request path |

Cells bound the blast radius of a bad config or a poison request: a region
of 36 instances runs as *Example:* 4 cells of 9, and every rollout stage
targets whole cells.

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. For each deep dive list the options, the choice and the reason. -->

### 7.1 Request lifecycle and filter chain

```mermaid
sequenceDiagram
  participant C as Client
  participant G as Gateway (event loop)
  participant K as JWKS cache
  participant R as Rate limiter
  participant O as Origin pool
  C->>G: HTTPS request
  G->>G: route lookup, strip internal headers, assign request ID
  G->>K: key for token kid
  K-->>G: public key
  G->>G: verify signature, expiry, audience, scopes
  G->>R: check(subject, route)
  R-->>G: allow
  G->>O: acquire concurrency permit, proxy (timeout 2 s)
  alt origin ok
    O-->>G: 200
  else connect failure and idempotent and budget available
    G->>O: retry on a different host
    O-->>G: 200
  end
  G->>G: outbound filters (headers, error mapping, metrics)
  G-->>C: response
```

Rules for filters:

- Filters run on the event loop and must never block. Any filter that needs
  I/O (for example, calling the rate limiter) is written as an asynchronous
  filter with its own timeout.
- Filters are ordered explicitly per route; a shared default chain covers
  request ID, header stripping and security headers.
- A filter that throws converts to a 500 Problem Details response and a
  metric tagged with the filter name, so a faulty filter is identified in
  minutes.

### 7.2 Deep dive 1: authentication and token validation

| Option | How it works | Cost | Risk |
|---|---|---|---|
| A. Introspect every token | Call the identity provider per request | One network hop, identity provider sized for peak traffic | Identity outage takes down all APIs |
| B. Local signature validation | Verify signed tokens with cached public keys | CPU only (3.4) | Revocation is delayed until token expiry |
| C. Local validation plus short-lived tokens plus revocation list | B, with access tokens of 5 to 15 minutes and a pushed deny list for revoked sessions | CPU plus a small pushed list | More moving parts |

**Choice:** C. **Reason:** it removes the identity provider from the hot
path while bounding the revocation window to the token lifetime, and the
deny list closes the gap for high-risk revocations (compromised account).

Validation rules, following JSON Web Token best practice (RFC 8725):

- Accept only the algorithms configured per issuer; never accept `none`
  and never take the algorithm choice from the token alone.
- Check issuer, audience, expiry and not-before, with clock skew tolerance
  of *Example:* 60 s.
- Refresh the JWKS on a schedule and on an unknown `kid`, rate-limited so a
  stream of forged `kid` values cannot hammer the identity provider.
- Strip any client-supplied `X-Verified-*` header before validation.
- Opaque partner tokens are the exception: introspect them and cache the
  result for at most 60 s.

### 7.3 Deep dive 2: retries with budgets

Retries are the main way a gateway turns a small origin problem into a big
one. With 3 attempts per request at every layer of a 3-layer call chain,
one failed request can become 27 at the bottom.

| Option | Behaviour under origin failure | Verdict |
|---|---|---|
| A. No gateway retries | Clients see every transient error | Too strict for connection resets during deploys |
| B. Fixed retry count (for example 3) | Traffic to a failing origin can triple | Causes retry storms |
| C. Retry budget per origin plus per-request cap | Retries stop when they exceed a share of normal traffic | Chosen |

**Choice:** C. **Reason:** retries help when failures are rare and
independent (a host restarting) and hurt when failures are correlated (the
origin is overloaded). A budget distinguishes the two automatically.

Budget rule for each origin cluster, per gateway instance:

- Track requests and retries over a sliding 10 s window.
- Allow a retry only if retries are below *Example:* 10% of requests in the
  window, plus a small floor (*Example:* 5 retries/s) for low-traffic origins.
- Cap each request at 2 attempts; retry on a different host.
- Retry only idempotent methods (for example GET, HEAD, PUT, DELETE, OPTIONS as defined
  by RFC 9110) or requests that carry an idempotency key, and only on
  connection failure, 502 or 503; never retry on a timeout for non-idempotent
  requests.
- Honour `Retry-After` from origins by passing it through instead of
  retrying immediately.

At most the origin receives 1.1 times its normal load from retries, which
the origin's capacity plan must include.

### 7.4 Deep dive 3: concurrency limits and load shedding

Rate limits protect against callers; concurrency limits protect origins
from the gateway itself. Little's law gives the in-flight count: at
*Example:* 20,000 requests/s to an origin with 50 ms latency there are about
1,000 requests in flight. When latency rises to 2 s the same rate means
40,000 in flight, which exhausts origin threads and gateway memory.

| Option | Behaviour | Verdict |
|---|---|---|
| A. Static limit per origin | Simple; wrong after every capacity change | Keep only as a ceiling |
| B. Adaptive limit from latency (gradient style) | Limit shrinks when latency rises above the no-load baseline, grows when it recovers | Chosen as primary control |
| C. Queue requests at the gateway | Hides overload and adds latency | Use only a short queue (*Example:* 50 ms) |

**Choice:** B with a static ceiling. **Reason:** an adaptive limit tracks
the origin's real capacity without manual tuning, and excess requests are
rejected fast with 503 and `Retry-After` rather than piling up. Priority
classes (*Example:* interactive above batch above prefetch) shed the lowest
class first; see Resilience and Load Shedding Design Template
(`system-design/resilience-load-shedding-design`) for the shedding policy.

### 7.5 Connection scaling and push

- Use a non-blocking event-loop runtime: one thread per core serves many
  thousands of connections, so idle push connections cost memory, not
  threads.
- Origin connections are pooled per origin cluster and per event loop to
  avoid cross-thread locking; pool size follows the concurrency limit.
- Deploys drain connections gradually: stop accepting new connections, send
  HTTP/2 GOAWAY, and for push connections ask clients to reconnect with
  jittered delays spread over *Example:* 5 minutes. Closing 200,000
  connections at once causes a reconnect storm on the remaining instances.
- Load balancers balance connections, not requests; long-lived HTTP/2
  connections skew load, so cap connection age (*Example:* 30 minutes,
  jittered) to rebalance over time.

### 7.6 Deep dive 4: configuration rollout

| Stage | Target | Bake time | Promotion check |
|---|---|---|---|
| 0. CI | Schema validation, route conflict check, replay of recorded requests | None | All checks pass |
| 1. Canary cell | One cell in one region | *Example:* 10 minutes | Error rate and p99 latency within 0.5 percentage points and 10% of control |
| 2. Region | All cells in that region | 30 minutes | Same checks |
| 3. Global | Remaining regions one at a time | 30 minutes each | Same checks |

**Choice:** staged, cell-based rollout with automatic rollback. **Reason:**
gateway config is global by nature, so a bad change is the largest single
outage risk. Each instance keeps the last-good bundle on disk and refuses a
bundle that fails local validation. Emergency changes (blocking an abusive
client) use a fast path that skips bake times but still runs CI checks, and
they are logged for review.

> **How real systems do it** (paraphrased from public Netflix Zuul 2
> material; see 12.4). Netflix moved its edge gateway, Zuul, from a
> blocking thread-per-connection design to an asynchronous one built on
> Netty. The reported motivation was connection scaling: persistent
> connections for push and long-lived streams cost little on an event loop
> but a whole thread in the blocking model. The project documentation
> describes inbound filters (for example, authentication and routing), an
> endpoint stage that proxies to the origin or answers directly, and
> outbound filters (metrics, headers), and warns that filters must never
> block the event loop. The write-ups also note that the move was harder
> than expected because many libraries kept per-request context in thread
> locals, which breaks when a request hops threads. When Zuul 2 was open
> sourced, features such as adaptive retries and configurable concurrency
> limits were highlighted. Lesson for this design: plan for context
> propagation and non-blocking libraries before you choose the runtime.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Requests per second | Horizontal scaling per cell; stateless instances | *Example:* L4 load balancer capacity per region |
| Connections | Memory-sized instances, connection age cap | *Example:* about 250,000 connections per instance before GC pauses grow |
| Route table size | Compiled trie, bundle under 10 MB | *Example:* about 20,000 routes before compile and push time exceeds 60 s |
| Origin count | Pools created lazily, idle pools closed | *Example:* about 1,000 origin clusters per instance |
| Hot origin | Adaptive concurrency limit plus shedding | Origin capacity |

Latency budget for gateway overhead (target p99 5 ms):

| Step | Budget |
|---|---|
| TLS and HTTP parsing (resumed sessions) | 0.8 ms |
| Route lookup and header handling | 0.2 ms |
| Token validation (cache miss) | 0.1 ms |
| Rate-limit call | 2.0 ms |
| Pool acquire and proxy setup | 0.4 ms |
| Outbound filters and logging | 0.3 ms |
| Headroom | 1.2 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable in a game day. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Config bundle | Bad route or filter shipped | Error-rate delta in canary cell | One cell (25% of one region, about 8% of global traffic) | Automatic rollback to last-good bundle | *Example:* 5 min |
| Origin cluster | Overloaded or failing | Latency gradient, 5xx share | Routes to that origin only | Retry budget, adaptive limit, fast 503 | Origin's own RTO |
| Identity provider | JWKS endpoint down | Refresh failures | None while cached keys are valid | Keep last keys; alert before they rotate | *Example:* 0 (degraded) |
| Rate limiter | Timeout or unavailable | Error rate on hook | All routes in the region | Fail open with local coarse limits | *Example:* 0 (degraded) |
| Gateway instance | Crash or GC stall | Health check, connection gauge | Its connections reconnect | Load balancer removes it; jittered reconnect | *Example:* 30 s |
| Region | Full regional outage | Synthetic probes, DNS health | One region's users | DNS or anycast shift; pre-warmed headroom | *Example:* 10 min |
| Control plane | Unavailable | Rollout API errors | No new config | Instances keep running on last-good bundle | *Example:* hours acceptable |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security

| Threat | Control |
|---|---|
| Header spoofing of identity | Strip `X-Verified-*` and internal headers from client requests |
| Token forgery or algorithm confusion | Fixed algorithm allow-list per issuer (RFC 8725) |
| Request smuggling between gateway and origin | Normalise and reject ambiguous `Content-Length` and `Transfer-Encoding`; use HTTP/2 to origins |
| Path traversal and route bypass | Normalise path before route match; reject encoded separators |
| Oversized requests | Limits on header size, body size and header count per route |
| Config tampering | Signed bundles, two-person review, audit trail |
| Gateway-to-origin interception | Mutual TLS with workload identities |
| [Threat] | [Control] |

Run the Security Review Template (`security/security-review`) and a threat
model (`security/threat-model`) on the filter chain before launch.

### 9.3 Observability

| Signal | Source | Alert threshold |
|---|---|---|
| Gateway overhead p99 | Span timing excluding origin | *Example:* above 5 ms for 10 min |
| 5xx share by origin and by gateway | Metrics, tagged with source (gateway or origin) | *Example:* gateway-originated 5xx above 0.1% |
| Retry budget exhausted | Counter per origin | Any for more than 2 min |
| Concurrency limit rejections | Counter per origin and priority | *Example:* above 1% of requests |
| Connections per instance | Gauge | *Example:* above 220,000 |
| Config version skew | Active bundle per cell | Mixed versions for more than 1 hour outside a rollout |

Traces follow W3C Trace Context and the OpenTelemetry HTTP semantic
conventions so origin spans join the gateway span.

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Gateway instances | [USD per instance-month] | *Example:* 36 per region x 3 | [USD] |
| Load balancers and data transfer | [USD per GB] | *Example:* 32 Gbit/s peak per region | [USD] |
| Logs (5% sampled successes) | [USD per GB ingested] | [GB] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Include how the design changes at ten times and one tenth of the load. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Runtime | Thread per connection; event loop | Event loop | Connection scaling for push | Harder debugging; no blocking libraries |
| Token check | Introspect; local | Local plus short tokens plus deny list | No identity dependency on hot path | Revocation delayed up to token lifetime |
| Retries | None; fixed; budget | Budget | Avoids retry storms | Some transient errors reach clients |
| Concurrency | Static; adaptive | Adaptive with ceiling | Tracks origin capacity | Harder to reason about in incidents |
| Topology | One fleet per region; cells | Cells | Bounded blast radius | More capacity overhead |
| Product | Build; managed gateway; open-source proxy | [Decide] | [Reason] | [Cost] |
| What changes at 10x | 12 million requests/s, 200 million connections | Split fleets by traffic class (push vs request/response), anycast at the edge, regional route sharding | Connection memory and LB limits bind first | More fleets to operate |
| What changes at 0.1x | 120,000 requests/s | One cell per region, managed gateway may be cheaper than running your own | Team cost dominates infrastructure cost | Less control over filters |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Gateway in one region, shadow traffic only | Shadow responses match legacy path at 99.9% | YYYY-MM-DD |
| M2 | Public API routes, token validation, rate-limit hook | 14 days at NFR targets | YYYY-MM-DD |
| M3 | Push connections migrated | Reconnect storm test passed | YYYY-MM-DD |
| M4 | Multi-region failover and staged config rollout | Game day: region loss and bad config | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Move traffic per route group with weighted DNS from 1% to 100%
over two weeks. Rollback is a DNS weight change back to the legacy path,
effective within the DNS TTL of 60 s.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Blocking library call stalls the event loop | Medium | High | Event-loop lag metric; lint for blocking APIs | [@name] |
| R-02 | *Example:* Gateway becomes a home for business logic | High | Medium | Ownership rule in 2.3; architecture review for new filters | [@name] |
| R-03 | *Example:* Migration changes subtle header behaviour | Medium | Medium | Shadow comparison in M1 | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary, open questions and references. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Should GraphQL requests be costed by query complexity at the gateway? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Do partner integrations need mutual TLS at the edge? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Traceability

| Requirement | Design section | Verification |
|---|---|---|
| FR-02 | 7.2 | Token test suite with forged, expired and wrong-audience tokens |
| FR-05 | 7.3 | Fault injection: origin 503 at 50% |
| FR-06, NFR-04 | 7.5 | Connection soak test at 220,000 per instance |
| FR-07, NFR-06 | 7.6 | Game day with a deliberately bad bundle |
| NFR-05 | 7.4 | One origin slowed to 2 s; other origins unchanged |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Origin | A backend service cluster the gateway proxies to |
| Filter | A unit of request or response processing in the gateway chain |
| Cell | A group of gateway instances that receives config changes together |
| Retry budget | A cap on retries as a share of normal requests to an origin |
| Adaptive concurrency limit | An in-flight request cap adjusted from observed latency |
| JWKS | JSON Web Key Set: the public keys used to verify tokens |

### 12.4 References

| Reference | Use |
|---|---|
| RFC 9110 HTTP Semantics | Methods, idempotency, status codes, `Retry-After` |
| RFC 9457 Problem Details for HTTP APIs | Error bodies |
| RFC 7519 JSON Web Token and RFC 8725 JSON Web Token Best Current Practices | Token validation rules in 7.2 |
| W3C Trace Context; OpenTelemetry semantic conventions for HTTP spans | Tracing in 9.3 |
| Netflix Technology Blog, "Zuul 2: The Netflix Journey to Asynchronous, Non-Blocking Systems" (2016) and "Open Sourcing Zuul 2" (2018); Netflix/zuul project wiki, "How It Works 2.0" | Inspiration for the box in 7.6 (paraphrased) |
| Netflix/concurrency-limits project on GitHub | Background on adaptive concurrency limits in 7.4 |
| Rate Limiter System Design Template (`system-design/design-rate-limiter`) | Rate-limit hook |
| Resilience and Load Shedding Design Template (`system-design/resilience-load-shedding-design`) | Shedding priorities |
| API Design Guidelines Template (`architecture/api-design-guidelines`) | Client-facing API conventions |

## Standard and licence note

Based on the IETF specifications cited in 12.4 (RFC 9110, RFC 9457,
RFC 7519 and RFC 8725) and on common industry practice for edge gateways.
The Netflix material is used as inspiration only and is paraphrased, not
copied. All numbers are illustrative examples; measure on your own
runtime and traffic, and check the current version of each specification.
Original text written for this collection. Template licence: MIT.