System Design: [API gateway name]

Reference design for an edge API gateway: the shared front door that terminates client connections, authenticates callers, routes requests to backend services and applies cross-cutting policy. It covers routing, token validation, the rate-limit hook, request and response filters, retries with budgets, adaptive concurrency limits, connection scaling for long-lived and push connections, multi-region failover, observability and safe configuration rollout. It follows the structure of the System Design Document template, cites HTTP Semantics (RFC 9110), JSON Web Token best practice (RFC 8725) and Problem Details (RFC 9457), and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; all numbers are illustrative assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Clients call 140 backend services through a mix of load balancers and per-team proxies. Each team implements its own token checks, so three services accepted expired tokens last quarter. Retries are configured per client, and during the last origin incident retry storms tripled traffic to the failing service and stretched a 4-minute degradation into a 40-minute outage. Mobile push connections are held by a separate legacy fleet that cannot be scaled without a release.]

1.2 Proposed solution

[Example: A single gateway fleet per region terminates TLS and HTTP/1.1, HTTP/2 and long-lived push connections on a non-blocking event-loop runtime. Each request passes through an ordered chain of inbound filters (identity, token validation, rate-limit hook, request normalisation), an endpoint stage that selects an origin cluster and proxies the request with pooled connections, and outbound filters (headers, metrics, error mapping). Retries are allowed only for idempotent requests and only within a retry budget per origin. Concurrency limits adapt to observed origin latency. Routes and filters are declared as versioned configuration, validated in CI, and rolled out in stages with automatic rollback. The gateway owns cross-cutting policy only; it never contains business logic.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

• [Example: business logic, request aggregation or response composition for specific screens (owned by backend-for-frontend services)]
• [Example: layer 3 and 4 volumetric DDoS protection (edge provider)]
• [Example: user login, consent and token issuance (identity provider)]
• [Example: service-to-service east-west traffic (service mesh)]
• [Example: the rate limiter's own algorithms and storage (see Rate Limiter System Design Template, system-design/design-rate-limiter)]

3. Capacity estimation

3.1 Assumptions

3.2 Fleet size

Per region the peak is 1,200,000 / 3 = 400,000 requests/s. At 12,000 requests/s per instance this needs 400,000 / 12,000 = 33.3, so 34 instances.

Connections per region are 20 million / 3 = 6.67 million. If one instance holds 200,000 connections (4 GB of connection state at 20 KB each), this also needs 6.67 million / 200,000 = 33.4, so 34 instances.

Consequence: both constraints land on about 34 instances, so plan 36 per region (12 per zone across 3 zones) and size each instance for memory as well as CPU. A region must also absorb a failed region's share: with 3 regions each survivor takes 50% more, so either run 54 per region or accept pre-warmed scale-out within 10 minutes during failover (9.1).

3.3 Bandwidth

400,000 requests/s x 10 KB = 4 GB/s, about 32 Gbit/s per region through the gateway.

Consequence: each instance carries about 1 Gbit/s at peak, well under a typical NIC, but the regional load balancer tier must be provisioned for 32 Gbit/s plus failover headroom; enable response compression on the gateway only for text bodies above a size threshold.

3.4 Token validation cost

If every request carries a signed token and a signature check costs Example: 30 microseconds of CPU, then 400,000 x 30 microseconds = 12 CPU seconds per second per region, about 12 cores spread across the fleet.

Consequence: local validation is affordable (one third of a core per instance), so there is no need to call the identity service on the hot path. A small cache of verified token digests for their remaining lifetime (capped at 60 s) halves the cost for chatty clients.

3.5 Logs

400,000 requests/s x 1 KB = 400 MB/s per region, about 34.6 TB per day per region.

Consequence: log 100% of errors and slow requests but sample successful requests at 5%; rely on metrics and traces for aggregate views.

3.6 Summary

4. API design

4.1 Interfaces

4.2 Route definition

Routes are declared as code in a repository and compiled into a bundle:

route: orders-v2
version: 41
match:
  host: api.example.com
  path_prefix: /v2/orders
  methods: [GET, POST]
auth:
  required: true
  audiences: ["orders-api"]
  scopes_by_method: { GET: ["orders:read"], POST: ["orders:write"] }
rate_limit:
  policy: per-key-default
origin:
  cluster: orders-service
  timeout_ms: 2000
  retry: { on: [connect_failure, http_503], max_attempts: 2, idempotent_only: true }
filters:
  inbound: [strip-internal-headers, request-id, auth, rate-limit, normalize-path]
  outbound: [security-headers, problem-details, metrics]

4.3 Origin-facing contract

Origins trust these headers only on connections from the gateway (mutual TLS between gateway and origin).

4.4 Errors

Gateway-generated errors use Problem Details (RFC 9457) and the status codes of RFC 9110:

HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Retry-After: 2

{"type": "https://api.example.com/problems/origin-overloaded", "title": "Service temporarily overloaded", "status": 503, "detail": "The orders service is shedding load. Retry after the indicated delay.", "instance": "/v2/orders", "request_id": "req_7f3a"}

5. Data model

5.1 Entities

All durable state belongs to the control plane; the data plane holds a read-only compiled copy of it.

5.2 Storage choices

5.3 Access patterns

6. High-level design

Cells bound the blast radius of a bad config or a poison request: a region of 36 instances runs as Example: 4 cells of 9, and every rollout stage targets whole cells.

7. Detailed design

7.1 Request lifecycle and filter chain

Rules for filters:

• Filters run on the event loop and must never block. Any filter that needs I/O (for example, calling the rate limiter) is written as an asynchronous filter with its own timeout.
• Filters are ordered explicitly per route; a shared default chain covers request ID, header stripping and security headers.
• A filter that throws converts to a 500 Problem Details response and a metric tagged with the filter name, so a faulty filter is identified in minutes.

7.2 Deep dive 1: authentication and token validation

Choice: C. Reason: it removes the identity provider from the hot path while bounding the revocation window to the token lifetime, and the deny list closes the gap for high-risk revocations (compromised account).

Validation rules, following JSON Web Token best practice (RFC 8725):

• Accept only the algorithms configured per issuer; never accept none and never take the algorithm choice from the token alone.
• Check issuer, audience, expiry and not-before, with clock skew tolerance of Example: 60 s.
• Refresh the JWKS on a schedule and on an unknown kid, rate-limited so a stream of forged kid values cannot hammer the identity provider.
• Strip any client-supplied X-Verified-* header before validation.
• Opaque partner tokens are the exception: introspect them and cache the result for at most 60 s.

7.3 Deep dive 2: retries with budgets

Retries are the main way a gateway turns a small origin problem into a big one. With 3 attempts per request at every layer of a 3-layer call chain, one failed request can become 27 at the bottom.

Choice: C. Reason: retries help when failures are rare and independent (a host restarting) and hurt when failures are correlated (the origin is overloaded). A budget distinguishes the two automatically.

Budget rule for each origin cluster, per gateway instance:

• Track requests and retries over a sliding 10 s window.
• Allow a retry only if retries are below Example: 10% of requests in the window, plus a small floor (Example: 5 retries/s) for low-traffic origins.
• Cap each request at 2 attempts; retry on a different host.
• Retry only idempotent methods (for example GET, HEAD, PUT, DELETE, OPTIONS as defined by RFC 9110) or requests that carry an idempotency key, and only on connection failure, 502 or 503; never retry on a timeout for non-idempotent requests.
• Honour Retry-After from origins by passing it through instead of retrying immediately.

At most the origin receives 1.1 times its normal load from retries, which the origin's capacity plan must include.

7.4 Deep dive 3: concurrency limits and load shedding

Rate limits protect against callers; concurrency limits protect origins from the gateway itself. Little's law gives the in-flight count: at Example: 20,000 requests/s to an origin with 50 ms latency there are about 1,000 requests in flight. When latency rises to 2 s the same rate means 40,000 in flight, which exhausts origin threads and gateway memory.

Choice: B with a static ceiling. Reason: an adaptive limit tracks the origin's real capacity without manual tuning, and excess requests are rejected fast with 503 and Retry-After rather than piling up. Priority classes (Example: interactive above batch above prefetch) shed the lowest class first; see Resilience and Load Shedding Design Template (system-design/resilience-load-shedding-design) for the shedding policy.

7.5 Connection scaling and push

• Use a non-blocking event-loop runtime: one thread per core serves many thousands of connections, so idle push connections cost memory, not threads.
• Origin connections are pooled per origin cluster and per event loop to avoid cross-thread locking; pool size follows the concurrency limit.
• Deploys drain connections gradually: stop accepting new connections, send HTTP/2 GOAWAY, and for push connections ask clients to reconnect with jittered delays spread over Example: 5 minutes. Closing 200,000 connections at once causes a reconnect storm on the remaining instances.
• Load balancers balance connections, not requests; long-lived HTTP/2 connections skew load, so cap connection age (Example: 30 minutes, jittered) to rebalance over time.

7.6 Deep dive 4: configuration rollout

Choice: staged, cell-based rollout with automatic rollback. Reason: gateway config is global by nature, so a bad change is the largest single outage risk. Each instance keeps the last-good bundle on disk and refuses a bundle that fails local validation. Emergency changes (blocking an abusive client) use a fast path that skips bake times but still runs CI checks, and they are logged for review.

How real systems do it (paraphrased from public Netflix Zuul 2 material; see 12.4). Netflix moved its edge gateway, Zuul, from a blocking thread-per-connection design to an asynchronous one built on Netty. The reported motivation was connection scaling: persistent connections for push and long-lived streams cost little on an event loop but a whole thread in the blocking model. The project documentation describes inbound filters (for example, authentication and routing), an endpoint stage that proxies to the origin or answers directly, and outbound filters (metrics, headers), and warns that filters must never block the event loop. The write-ups also note that the move was harder than expected because many libraries kept per-request context in thread locals, which breaks when a request hops threads. When Zuul 2 was open sourced, features such as adaptive retries and configurable concurrency limits were highlighted. Lesson for this design: plan for context propagation and non-blocking libraries before you choose the runtime.

8. Scalability and performance

Latency budget for gateway overhead (target p99 5 ms):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security

Run the Security Review Template (security/security-review) and a threat model (security/threat-model) on the filter chain before launch.

9.3 Observability

Traces follow W3C Trace Context and the OpenTelemetry HTTP semantic conventions so origin spans join the gateway span.

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Move traffic per route group with weighted DNS from 1% to 100% over two weeks. Rollback is a DNS weight change back to the legacy path, effective within the DNS TTL of 60 s.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Traceability

12.3 Glossary

12.4 References

Standard and licence note

Based on the IETF specifications cited in 12.4 (RFC 9110, RFC 9457, RFC 7519 and RFC 8725) and on common industry practice for edge gateways. The Netflix material is used as inspiration only and is paraphrased, not copied. All numbers are illustrative examples; measure on your own runtime and traffic, and check the current version of each specification. Original text written for this collection. Template licence: MIT.