# Load Balancing and Service Discovery: [Service or platform]

> Building-block decision record for spreading traffic across instances of
> a service and for finding those instances. It covers layer 4 versus
> layer 7 balancing, balancing algorithms, active and passive health
> checks, service discovery (DNS, registry, platform endpoints), connection
> draining and capacity under failure. HTTP behaviour follows RFC 9110 HTTP
> Semantics (STD 97, June 2022), with RFC 9112 and RFC 9113 for connection
> handling in HTTP/1.1 and HTTP/2; DNS SRV records follow RFC 2782. Concepts
> also draw on the power of two choices (Mitzenmacher, IEEE TPDS, 2001), the
> Maglev load balancer (Eisenbud et al., NSDI 2016), the gRPC health
> checking protocol and Kubernetes probe semantics. Replace every
> `[bracketed]` value and every row marked *Example*. Worked examples use
> Quillmere, an invented book-review and reading-list platform shared by
> the other building-block templates in this category.

<!--
Guidance: How to use this template
- Start from capacity under failure (section 6): how many instances can
  fail while the rest stay below their safe utilisation? Health checks and
  draining only help if the survivors can absorb the load.
- Keep liveness and readiness separate. Liveness answers "restart me";
  readiness answers "send me traffic". Mixing them turns an overload into
  a restart storm.
- Plan for both deploys and failures: most dropped requests come from
  instances removed without draining, not from crashes.
- For edge concerns (TLS termination for the public internet,
  authentication, rate limits per client) see the API Gateway System
  Design template (`system-design/design-api-gateway`); this record covers
  the balancing and discovery layer itself.
-->

## Document control

<!-- Guidance: Include the platform team that owns the balancer and the service team that owns the health endpoint; both define the contract. -->

| Field | Value |
|---|---|
| Document ID | [LB-REC-001] |
| Service | [*Example:* `reviews-api`, HTTP/2 and gRPC, 12 instances] |
| Traffic sources | [*Example:* Public API gateway, internal services] |
| Balancer layers | [*Example:* Regional L4 network balancer, L7 proxy fleet, client-side balancing for gRPC] |
| Discovery mechanism | [*Example:* Platform endpoints API with DNS for legacy clients] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@platform-networking-lead] |
| Service owner | [@reviews-lead] |
| Reviewers | [@sre], [@security] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related | API Gateway System Design (`system-design/design-api-gateway`), Resilience and Load Shedding Design (`system-design/resilience-load-shedding-design`), CDN System Design (`system-design/design-cdn`) |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Platform networking | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Service owner | [@name] | [Decision] | YYYY-MM-DD |
| SRE | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added passive outlier detection after a 15 s error window during node failure |

## 1. Traffic path

<!-- Guidance: Draw every hop from client to instance. Each hop is a place where a connection can be held open to a dead instance, so each needs its own health and draining rule. -->

```mermaid
flowchart LR
  U["Clients"] --> L4["Regional L4 balancer"]
  L4 --> P1["L7 proxy 1"]
  L4 --> P2["L7 proxy 2"]
  P1 --> I["reviews-api instances x12"]
  P2 --> I
  S["Internal gRPC callers"] -->|"client-side balancing"| I
  D["Discovery: endpoints API"] -.-> P1
  D -.-> P2
  D -.-> S
```

| Hop | Layer | Balancing unit | Health source | Owner |
|---|---|---|---|---|
| *Example:* Regional L4 balancer | 4 (TCP, UDP) | Connection | TCP and HTTP check on proxies | Platform |
| *Example:* L7 proxy fleet | 7 (HTTP/1.1, HTTP/2) | Request | Active check plus passive outlier detection | Platform |
| *Example:* gRPC client library | 7 | Request (call) | Endpoints API plus gRPC health service | Service teams |
| [Hop] | [Layer] | [Unit] | [Source] | [Owner] |

## 2. Layer 4 versus layer 7

<!-- Guidance: L4 balancers see connections and addresses; L7 balancers see requests, methods, paths, headers and status codes. Long-lived HTTP/2 or gRPC connections balanced only at L4 pin all requests on a connection to one instance. -->

| Concern | L4 | L7 |
|---|---|---|
| Unit of balancing | Connection or flow | Request |
| Sees HTTP method, path, status code | No | Yes |
| Retries | Not safely (cannot see requests) | Can retry idempotent methods (RFC 9110 section 9.2.2) |
| Long-lived HTTP/2 or gRPC connections | All calls on a connection go to one instance | Calls spread across instances |
| TLS | Passthrough or termination | Usually terminates to inspect requests |
| Cost and latency | Lower | Higher per request |

Retry rule, *Example:* the L7 proxy retries only methods that RFC 9110
defines as idempotent (GET, HEAD, OPTIONS, TRACE, PUT, DELETE) and only on
connection failure or 503; POST is retried only when the request carries
an idempotency key the service honours. Retries are capped at one per
request and count against a retry budget (see
`system-design/resilience-load-shedding-design`).

## 3. Balancing algorithm

<!-- Guidance: Pick the algorithm for the request cost profile. Round robin is fine when requests cost about the same; least-outstanding-requests or power of two choices handles uneven costs; consistent hashing gives affinity (caches, sessions) at the cost of balance. -->

| Algorithm | How it picks | Good for | Watch out for |
|---|---|---|---|
| Round robin (weighted) | Next instance in turn, weighted by capacity | Uniform requests | Slow instances still get their share |
| Least outstanding requests | Instance with the fewest in-flight requests | Uneven request cost | Needs accurate counts per proxy |
| Power of two choices | Sample two instances at random, pick the less loaded | Many proxies with partial views; avoids herding onto one "best" instance | Needs a load signal |
| Consistent hashing (ring, Maglev table) | Hash of a key such as user ID or connection tuple | Affinity, connection persistence across balancer changes | Hot keys load one instance |
| Random | Uniform random | Very large fleets, simplicity | Higher variance on small fleets |

Mitzenmacher (2001) showed that choosing the less loaded of two random
choices reduces the maximum load dramatically compared with one random
choice. Maglev (NSDI 2016) uses consistent hashing plus connection
tracking so that connections survive changes in the balancer fleet.

Decision: [*Example:* Least outstanding requests at the L7 proxy, power
of two choices in the gRPC client library, slow start of 30 s for new
instances.]

## 4. Health checks

<!-- Guidance: Combine active checks (the balancer probes an endpoint) with passive checks (the balancer watches real responses). Active checks alone leave a detection window; passive checks alone never bring an instance back. Health endpoints should report readiness to serve, not the health of every dependency, or one shared dependency failure removes every instance at once. -->

| Setting | *Example* value | Reason |
|---|---|---|
| Active check | `GET /ready` every 5 s, timeout 2 s | Cheap, no dependency calls |
| Unhealthy threshold | 3 consecutive failures | Avoid flapping on one slow reply |
| Healthy threshold | 2 consecutive successes | Quick return after recovery |
| Passive outlier detection | Eject after 5 consecutive 5xx or connection errors, for 30 s, at most 2 of 12 instances ejected at once | Closes the active-check window |
| gRPC services | `grpc.health.v1` Health service, `Check` and `Watch`, status `SERVING` or `NOT_SERVING` | Standard protocol understood by clients |
| Kubernetes probes | Startup probe for slow boot; readiness removes the pod from service endpoints; liveness restarts only on deadlock | Matches the platform behaviour |
| Overload response | 503 with `Retry-After` (RFC 9110 sections 15.6.4 and 10.2.3) | Balancer and clients back off |

Detection window, *Example:* with a 5 s interval and 3 failures the
balancer needs up to 15 s (17 s including the last 2 s timeout) to mark a
crashed instance down. At 500 requests per second per instance that is
up to 7,500 to 8,500 failed requests without passive detection; outlier
detection after 5 errors cuts this to a few requests per proxy.

## 5. Service discovery

<!-- Guidance: Name the source of truth for "which instances exist and are ready", how clients learn about changes and how fast. DNS caching makes DNS slow to react; registries and platform endpoint APIs push changes but are a dependency to protect. -->

| Mechanism | How clients learn | Change propagation | Strengths | Weaknesses |
|---|---|---|---|---|
| DNS A/AAAA records | Resolve a name | Bounded by TTL and client caching | Universal | Clients and libraries may cache beyond TTL; no health detail |
| DNS SRV (RFC 2782) | Resolve service name to host, port, priority, weight | Bounded by TTL | Ports and weights in DNS | Limited client support |
| Service registry | Instances register and heartbeat; clients query or watch | Seconds | Rich metadata, health aware | Registry must be highly available |
| Platform endpoints API | Platform tracks ready pods; proxies and clients watch | Seconds | Tied to readiness probes | Platform specific |
| Static configuration | Deployed list | On redeploy | Simple | Manual, error-prone |

| Rule | *Example* decision |
|---|---|
| Source of truth | Platform endpoints, derived from readiness |
| DNS TTL for legacy clients | 30 s; legacy clients reconnect at least every 5 minutes |
| Discovery outage | Proxies and clients keep the last known list; never empty the list on a discovery error |
| Cross-zone | Prefer same-zone instances while they hold at least 70% of capacity, then spill over |

## 6. Capacity under failure

<!-- Guidance: Show the arithmetic. Size the fleet so that losing the largest likely unit (one instance, one zone) keeps survivors below the safe utilisation you measured in load tests. -->

| Input | *Example* value |
|---|---|
| Peak load | 6,000 requests per second |
| Capacity per instance | 800 requests per second at p99 target |
| Safe utilisation | 70% (560 requests per second per instance) |
| Instances | 12 |
| Capacity at safe utilisation | 12 x 560 = 6,720 requests per second |
| Load per instance, all healthy | 6,000 / 12 = 500 (62.5% of capacity) |
| Load per instance, 1 lost | 6,000 / 11 = 545 (68.2%) |
| Load per instance, 2 lost | 6,000 / 10 = 600 (75.0%), above safe utilisation |
| Minimum for N+1 | ceil(6,000 / 560) + 1 = 12 instances |

With 12 instances spread 4 per zone, losing a whole zone leaves 8
instances at 750 requests per second each (93.8%), well above safe
utilisation. [*Example* decision: accept degraded latency on zone loss
and shed non-logged-in traffic first, or grow to 18 instances (6 per
zone, 12 left after zone loss at 500 each, 62.5%, because at least
ceil(6,000 / 560) = 11 survivors are needed), record in LB-03.]

## 7. Connection draining and deploys

<!-- Guidance: Removing an instance must stop new requests first, let in-flight requests finish, then close connections. HTTP/1.1 uses `Connection: close` (RFC 9110 section 7.6.1, RFC 9112 section 9.6); HTTP/2 uses GOAWAY (RFC 9113 section 6.8) so clients open new streams elsewhere. -->

```mermaid
sequenceDiagram
  participant P as Platform
  participant I as Instance
  participant D as Discovery
  participant LB as L7 proxy
  P->>I: Termination signal
  I->>I: Readiness turns false
  D-->>LB: Instance removed from endpoints
  I->>LB: HTTP/2 GOAWAY on open connections
  Note over I: Finishes in-flight requests up to 30 s
  I->>P: Exit
```

| Setting | *Example* value | Reason |
|---|---|---|
| Pre-stop delay | 5 s before closing listeners | Discovery and proxies observe removal first |
| Drain timeout | 30 s | p99.9 request duration is 20 s |
| Long-lived streams | Server sends GOAWAY, client reconnects with backoff and jitter | Avoid reconnect storm |
| Slow start for new instances | 30 s ramp | Cold caches and JIT warm-up |
| Rolling deploy batch | 2 instances at a time | Keeps 10 of 12 serving; 600 per instance (75%) is accepted only during deploys outside peak |

## 8. Decisions and open items

<!-- Guidance: Record each decision with its reason; link larger ones to an ADR (`architecture/adr-full`). -->

| ID | Decision or open item | Reason | Owner | Status |
|---|---|---|---|---|
| LB-01 | *Example:* L7 balancing for all HTTP/2 and gRPC traffic | L4 pins long-lived connections | @platform-networking-lead | Decided |
| LB-02 | *Example:* Passive outlier detection on all proxies | 15 s active detection window | @sre | Decided |
| LB-03 | *Example:* Grow to 18 instances (6 per zone) for zone-loss tolerance | Zone loss puts survivors at 93.8% | @reviews-lead | Open |
| LB-04 | *Example:* Health endpoint excludes shared database check | Avoid removing every instance on one dependency failure | @reviews-lead | Decided |
| [LB-NN] | [Item] | [Reason] | [@owner] | [Status] |

## Glossary and references

<!-- Guidance: Keep terms consistent with the platform's own documentation. -->

| Term | Meaning |
|---|---|
| L4, L7 | Balancing on transport connections versus on application requests |
| Readiness | Whether an instance should receive traffic now |
| Liveness | Whether an instance should be restarted |
| Outlier detection | Passive removal of instances that return errors on real traffic |
| Connection draining | Stopping new work for an instance while in-flight work completes |
| Slow start | Ramping traffic to a new instance gradually |

References:

- RFC 9110, HTTP Semantics (STD 97, June 2022),
  https://www.rfc-editor.org/info/rfc9110 (sections 7.6.1, 9.2.2, 10.2.3,
  15.6.4).
- RFC 9112, HTTP/1.1 (June 2022), https://www.rfc-editor.org/info/rfc9112
  (section 9.6, tear-down).
- RFC 9113, HTTP/2 (June 2022), https://www.rfc-editor.org/info/rfc9113
  (section 6.8, GOAWAY).
- RFC 2782, A DNS RR for specifying the location of services (DNS SRV),
  February 2000, https://www.rfc-editor.org/info/rfc2782.
- Mitzenmacher M., "The power of two choices in randomized load
  balancing", IEEE Transactions on Parallel and Distributed Systems
  12(10), 2001, pages 1094 to 1104, https://doi.org/10.1109/71.963420.
- Eisenbud D. et al., "Maglev: A Fast and Reliable Software Network Load
  Balancer", 13th USENIX Symposium on Networked Systems Design and
  Implementation (NSDI '16), 2016, pages 523 to 535,
  https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud.
- gRPC, Health Checking guide, https://grpc.io/docs/guides/health-checking/
  (checked 2026-10-30).
- Kubernetes, Liveness, Readiness and Startup Probes,
  https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/
  (checked 2026-10-30).
- Related templates: API Gateway System Design
  (`system-design/design-api-gateway`), Resilience and Load Shedding Design
  (`system-design/resilience-load-shedding-design`), Rate Limiter System
  Design (`system-design/design-rate-limiter`), Partitioning and Sharding
  Plan (`system-design/partitioning-sharding-guide`), Back-of-Envelope
  Estimation (`system-design/back-of-envelope-estimation`).

## Standard and licence note

Based on RFC 9110, RFC 9112, RFC 9113 and RFC 2782 for protocol behaviour,
and on the publicly available papers and project documentation cited
above for balancing and health checking; no single standard governs load
balancer design. The text, tables, numbers and examples are original to
this collection, and Quillmere is an invented organisation. Check the
current RFC status and the documentation of the balancer, platform and
client libraries you use. Template licence: MIT.