Load Balancing and Service Discovery Template (L4/L7, Health Checks)

Traffic path, L4 vs L7, balancing algorithms, active and passive health checks, discovery, capacity under failure, draining

System DesignStandardRFC 9110 HTTP Semantics; RFC 9113; RFC 2782load balancing service discoveryUpdated Oct 29, 2026

Open in editorDownload .md

How to use: Use it when placing a service behind load balancers or changing how instances are found. Start with capacity under failure, then health checks and draining. The main pitfall is mixing liveness and readiness or removing instances without draining, which drops requests on every deploy.

Preview

Load Balancing and Service Discovery: [Service or platform]

Building-block decision record for spreading traffic across instances of a service and for finding those instances. It covers layer 4 versus layer 7 balancing, balancing algorithms, active and passive health checks, service discovery (DNS, registry, platform endpoints), connection draining and capacity under failure. HTTP behaviour follows RFC 9110 HTTP Semantics (STD 97, June 2022), with RFC 9112 and RFC 9113 for connection handling in HTTP/1.1 and HTTP/2; DNS SRV records follow RFC 2782. Concepts also draw on the power of two choices (Mitzenmacher, IEEE TPDS, 2001), the Maglev load balancer (Eisenbud et al., NSDI 2016), the gRPC health checking protocol and Kubernetes probe semantics. Replace every [bracketed] value and every row marked Example. Worked examples use Quillmere, an invented book-review and reading-list platform shared by the other building-block templates in this category.

Document control

Field	Value
Document ID	[LB-REC-001]
Service	[Example: `reviews-api`, HTTP/2 and gRPC, 12 instances]
Traffic sources	[Example: Public API gateway, internal services]
Balancer layers	[Example: Regional L4 network balancer, L7 proxy fleet, client-side balancing for gRPC]
Discovery mechanism	[Example: Platform endpoints API with DNS for legacy clients]
Version	[0.1]
Status	[Draft / In review / Approved / Implemented / Superseded]
Owner	[@platform-networking-lead]
Service owner	[@reviews-lead]
Reviewers	[@sre], [@security]
Last updated	YYYY-MM-DD
Classification	[Internal]
Related	API Gateway System Design (`system-design/design-api-gateway`), Resilience and Load Shedding Design (`system-design/resilience-load-shedding-design`), CDN System Design (`system-design/design-cdn`)

Approvals

Role	Name	Decision	Date
Platform networking	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Service owner	[@name]	[Decision]	YYYY-MM-DD
SRE	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Added passive outlier detection after a 15 s error window during node failure

1. Traffic path

Hop	Layer	Balancing unit	Health source	Owner
Example: Regional L4 balancer	4 (TCP, UDP)	Connection	TCP and HTTP check on proxies	Platform
Example: L7 proxy fleet	7 (HTTP/1.1, HTTP/2)	Request	Active check plus passive outlier detection	Platform
Example: gRPC client library	7	Request (call)	Endpoints API plus gRPC health service	Service teams
[Hop]	[Layer]	[Unit]	[Source]	[Owner]

2. Layer 4 versus layer 7

Concern	L4	L7
Unit of balancing	Connection or flow	Request
Sees HTTP method, path, status code	No	Yes
Retries	Not safely (cannot see requests)	Can retry idempotent methods (RFC 9110 section 9.2.2)
Long-lived HTTP/2 or gRPC connections	All calls on a connection go to one instance	Calls spread across instances
TLS	Passthrough or termination	Usually terminates to inspect requests
Cost and latency	Lower	Higher per request

Retry rule, Example: the L7 proxy retries only methods that RFC 9110 defines as idempotent (GET, HEAD, OPTIONS, TRACE, PUT, DELETE) and only on connection failure or 503; POST is retried only when the request carries an idempotency key the service honours. Retries are capped at one per request and count against a retry budget (see system-design/resilience-load-shedding-design).

3. Balancing algorithm

Algorithm	How it picks	Good for	Watch out for
Round robin (weighted)	Next instance in turn, weighted by capacity	Uniform requests	Slow instances still get their share
Least outstanding requests	Instance with the fewest in-flight requests	Uneven request cost	Needs accurate counts per proxy
Power of two choices	Sample two instances at random, pick the less loaded	Many proxies with partial views; avoids herding onto one "best" instance	Needs a load signal
Consistent hashing (ring, Maglev table)	Hash of a key such as user ID or connection tuple	Affinity, connection persistence across balancer changes	Hot keys load one instance
Random	Uniform random	Very large fleets, simplicity	Higher variance on small fleets

Mitzenmacher (2001) showed that choosing the less loaded of two random choices reduces the maximum load dramatically compared with one random choice. Maglev (NSDI 2016) uses consistent hashing plus connection tracking so that connections survive changes in the balancer fleet.

Decision: [Example: Least outstanding requests at the L7 proxy, power of two choices in the gRPC client library, slow start of 30 s for new instances.]

4. Health checks

Setting	Example value	Reason
Active check	`GET /ready` every 5 s, timeout 2 s	Cheap, no dependency calls
Unhealthy threshold	3 consecutive failures	Avoid flapping on one slow reply
Healthy threshold	2 consecutive successes	Quick return after recovery
Passive outlier detection	Eject after 5 consecutive 5xx or connection errors, for 30 s, at most 2 of 12 instances ejected at once	Closes the active-check window
gRPC services	`grpc.health.v1` Health service, `Check` and `Watch`, status `SERVING` or `NOT_SERVING`	Standard protocol understood by clients
Kubernetes probes	Startup probe for slow boot; readiness removes the pod from service endpoints; liveness restarts only on deadlock	Matches the platform behaviour
Overload response	503 with `Retry-After` (RFC 9110 sections 15.6.4 and 10.2.3)	Balancer and clients back off

Detection window, Example: with a 5 s interval and 3 failures the balancer needs up to 15 s (17 s including the last 2 s timeout) to mark a crashed instance down. At 500 requests per second per instance that is up to 7,500 to 8,500 failed requests without passive detection; outlier detection after 5 errors cuts this to a few requests per proxy.

5. Service discovery

Mechanism	How clients learn	Change propagation	Strengths	Weaknesses
DNS A/AAAA records	Resolve a name	Bounded by TTL and client caching	Universal	Clients and libraries may cache beyond TTL; no health detail
DNS SRV (RFC 2782)	Resolve service name to host, port, priority, weight	Bounded by TTL	Ports and weights in DNS	Limited client support
Service registry	Instances register and heartbeat; clients query or watch	Seconds	Rich metadata, health aware	Registry must be highly available
Platform endpoints API	Platform tracks ready pods; proxies and clients watch	Seconds	Tied to readiness probes	Platform specific
Static configuration	Deployed list	On redeploy	Simple	Manual, error-prone

Rule	Example decision
Source of truth	Platform endpoints, derived from readiness
DNS TTL for legacy clients	30 s; legacy clients reconnect at least every 5 minutes
Discovery outage	Proxies and clients keep the last known list; never empty the list on a discovery error
Cross-zone	Prefer same-zone instances while they hold at least 70% of capacity, then spill over

6. Capacity under failure

Input	Example value
Peak load	6,000 requests per second
Capacity per instance	800 requests per second at p99 target
Safe utilisation	70% (560 requests per second per instance)
Instances	12
Capacity at safe utilisation	12 x 560 = 6,720 requests per second
Load per instance, all healthy	6,000 / 12 = 500 (62.5% of capacity)
Load per instance, 1 lost	6,000 / 11 = 545 (68.2%)
Load per instance, 2 lost	6,000 / 10 = 600 (75.0%), above safe utilisation
Minimum for N+1	ceil(6,000 / 560) + 1 = 12 instances

With 12 instances spread 4 per zone, losing a whole zone leaves 8 instances at 750 requests per second each (93.8%), well above safe utilisation. [Example decision: accept degraded latency on zone loss and shed non-logged-in traffic first, or grow to 18 instances (6 per zone, 12 left after zone loss at 500 each, 62.5%, because at least ceil(6,000 / 560) = 11 survivors are needed), record in LB-03.]

7. Connection draining and deploys

Setting	Example value	Reason
Pre-stop delay	5 s before closing listeners	Discovery and proxies observe removal first
Drain timeout	30 s	p99.9 request duration is 20 s
Long-lived streams	Server sends GOAWAY, client reconnects with backoff and jitter	Avoid reconnect storm
Slow start for new instances	30 s ramp	Cold caches and JIT warm-up
Rolling deploy batch	2 instances at a time	Keeps 10 of 12 serving; 600 per instance (75%) is accepted only during deploys outside peak

8. Decisions and open items

ID	Decision or open item	Reason	Owner	Status
LB-01	Example: L7 balancing for all HTTP/2 and gRPC traffic	L4 pins long-lived connections	@platform-networking-lead	Decided
LB-02	Example: Passive outlier detection on all proxies	15 s active detection window	@sre	Decided
LB-03	Example: Grow to 18 instances (6 per zone) for zone-loss tolerance	Zone loss puts survivors at 93.8%	@reviews-lead	Open
LB-04	Example: Health endpoint excludes shared database check	Avoid removing every instance on one dependency failure	@reviews-lead	Decided
[LB-NN]	[Item]	[Reason]	[@owner]	[Status]

Glossary and references

Term	Meaning
L4, L7	Balancing on transport connections versus on application requests
Readiness	Whether an instance should receive traffic now
Liveness	Whether an instance should be restarted
Outlier detection	Passive removal of instances that return errors on real traffic
Connection draining	Stopping new work for an instance while in-flight work completes
Slow start	Ramping traffic to a new instance gradually

References:

RFC 9110, HTTP Semantics (STD 97, June 2022), https://www.rfc-editor.org/info/rfc9110 (sections 7.6.1, 9.2.2, 10.2.3, 15.6.4).
RFC 9112, HTTP/1.1 (June 2022), https://www.rfc-editor.org/info/rfc9112 (section 9.6, tear-down).
RFC 9113, HTTP/2 (June 2022), https://www.rfc-editor.org/info/rfc9113 (section 6.8, GOAWAY).
RFC 2782, A DNS RR for specifying the location of services (DNS SRV), February 2000, https://www.rfc-editor.org/info/rfc2782.
Mitzenmacher M., "The power of two choices in randomized load balancing", IEEE Transactions on Parallel and Distributed Systems 12(10), 2001, pages 1094 to 1104, https://doi.org/10.1109/71.963420.
Eisenbud D. et al., "Maglev: A Fast and Reliable Software Network Load Balancer", 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI '16), 2016, pages 523 to 535, https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud.
gRPC, Health Checking guide, https://grpc.io/docs/guides/health-checking/ (checked 2026-10-30).
Kubernetes, Liveness, Readiness and Startup Probes, https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/ (checked 2026-10-30).
Related templates: API Gateway System Design (system-design/design-api-gateway), Resilience and Load Shedding Design (system-design/resilience-load-shedding-design), Rate Limiter System Design (system-design/design-rate-limiter), Partitioning and Sharding Plan (system-design/partitioning-sharding-guide), Back-of-Envelope Estimation (system-design/back-of-envelope-estimation).

Standard and licence note

Based on RFC 9110, RFC 9112, RFC 9113 and RFC 2782 for protocol behaviour, and on the publicly available papers and project documentation cited above for balancing and health checking; no single standard governs load balancer design. The text, tables, numbers and examples are original to this collection, and Quillmere is an invented organisation. Check the current RFC status and the documentation of the balancer, platform and client libraries you use. Template licence: MIT.

Related templates

System Design