Resilience and Load Shedding Design Template (Retries, Circuit Breakers)

Failure model, deadlines, retries with jitter and budgets, circuit breakers, bulkheads, backpressure, criticality-based shedding, degradation, fault tests

Системний дизайнПовнийGoogle SRE book ch. 21-22; Amazon Builders' Libraryresilience load shedding design mathОновлено 21 жовт. 2026 р.

Відкрити в редакторіЗавантажити .md

Як використовувати: Use it when a service must survive slow or failing dependencies and its own overload without cascading. Fill the dependency inventory and failure model first, then the per-dependency matrix (13) with real numbers. The main pitfall is retrying at every layer without a budget, which multiplies load on a dependency exactly when it is failing.

Перегляд

Resilience Design: [Service name]

Design record for how one service survives slow, failing and overloaded dependencies and its own overload: failure model, timeouts and deadline propagation, retries with exponential backoff and jitter, retry budgets, circuit breakers, bulkheads, backpressure, load shedding and admission control by criticality, graceful degradation, idempotency, health checks, testing with fault injection, metrics and alerts, and a per-dependency resilience matrix. Concepts follow the Google Site Reliability Engineering book (chapters 21 "Handling Overload" and 22 "Addressing Cascading Failures") and the Amazon Builders' Library articles "Timeouts, retries, and backoff with jitter" and "Using load shedding to avoid overload". Replace every [bracketed] value and every row marked Example; numbers are illustrative assumptions.

Document control

Field	Value
Document ID	[RES-ORD-001]
Service	[Order API]
Version	[0.1]
Status	[Draft / In review / Approved / Superseded]
Owner	[@service-owner]
Reviewers	[@sre], [@staff-engineer]
Last updated	YYYY-MM-DD
Classification	[Internal]
Service tier	[Tier 1 / 2 / 3]
SLO reference	[Link to SLO document]
Related	[System design], [FMEA], [Chaos experiments], [Runbook]

Approvals

Role	Name	Decision	Date
Service owner	[@name]	[Approved / Changes requested]	YYYY-MM-DD
SRE lead	[@name]	[Decision]	YYYY-MM-DD
Staff / principal engineer	[@name]	[Decision]	YYYY-MM-DD
Owners of critical dependencies	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Added retry budget after retry storm in incident INC-[nnn]
0.3	YYYY-MM-DD	[@name]	Example: Criticality classes and shedding thresholds from load test
1.0	YYYY-MM-DD	[@name]	Example: Approved

Introduction and scope
Service context and dependencies
Failure model
Timeouts and deadline propagation
Retries, backoff and retry budgets
Circuit breakers
Bulkheads and concurrency limits
Backpressure
Load shedding and admission control
Graceful degradation
Idempotency
Health checks
Per-dependency resilience matrix
Testing and fault injection
Metrics and alerts
Decisions, risks and traceability
Glossary and references

1. Introduction and scope

1.1 Purpose

[Example: The Order API accepts and tracks customer orders. It depends on inventory, pricing, payment, a recommendation service and a relational database. In the last two quarters it suffered two cascading failures: a slow recommendation service exhausted the request thread pool, and a retry storm against payment tripled payment's load during its own incident. This document defines how the Order API contains those failures.]

1.2 Resilience objectives

ID	Objective	Target
RO-01	Example: Checkout stays available when any non-critical dependency fails	Checkout success at or above 99.9% with recommendations down
RO-02	Example: Overload degrades gracefully instead of collapsing	Goodput at 2 times capacity stays at or above 90% of capacity
RO-03	Example: The service never amplifies a dependency incident	Retry traffic at or below 10% of request traffic at all times
RO-04	Example: Recovery is automatic after a dependency recovers	Back to normal within 2 minutes without manual action
[RO-nn]	[Objective]	[Target]

1.3 Out of scope

[Example: loss of a full region (see IT Disaster Recovery Plan Template (ops/disaster-recovery-plan))]
[Example: data corruption and backup restore]
[Example: volumetric network attacks (edge provider)]

2. Service context and dependencies

2.1 Call graph

Solid arrows are synchronous calls on the checkout path; dotted arrows are optional or asynchronous.

2.2 Dependency inventory

ID	Dependency	Type	Criticality for checkout	Normal p50 / p99 latency	Owner
DEP-01	Example: Orders database	Sync, stateful	Hard (no checkout without it)	3 ms / 25 ms	[@team]
DEP-02	Example: Inventory service	Sync	Hard	15 ms / 120 ms	[@team]
DEP-03	Example: Pricing service	Sync	Soft (cached prices acceptable for 10 minutes)	10 ms / 80 ms	[@team]
DEP-04	Example: Payment service	Sync	Hard	250 ms / 1 500 ms	[@team]
DEP-05	Example: Recommendation service	Sync	None (page renders without it)	40 ms / 300 ms	[@team]
DEP-06	Example: Event bus	Async	Soft (outbox buffers events)	5 ms / 50 ms	[@team]
[DEP-nn]	[Dependency]	[Sync / Async]	[Hard / Soft / None]	[Latency]	[@team]

3. Failure model

ID	Failure mode	Typical cause	Effect without protection
FMD-01	Dependency slow (latency 10 to 100 times normal)	Garbage collection, lock contention, overloaded callee	Threads and connections held; our own queue grows; cascade
FMD-02	Dependency returns errors	Bug, bad deploy, exhausted resource	Errors propagate; retries add load
FMD-03	Dependency unreachable	Network partition, DNS, certificate expiry	Connection timeouts; slow failure if timeouts are long
FMD-04	Own overload	Traffic spike, retry storm from clients, reduced capacity after a failure	Latency rises for everyone; work completes after clients have given up
FMD-05	Partial instance failure	One bad host or zone	Fraction of requests fail; health checks may not see it
FMD-06	Resource exhaustion	Memory leak, file descriptors, connection pool	Gradual degradation then crash
FMD-07	Correlated restart	Crash loop, simultaneous deploy	Capacity drop; thundering herd on dependencies at start
[FMD-nn]	[Mode]	[Cause]	[Effect]

A cascading failure, as described in the SRE book chapter "Addressing Cascading Failures", is a failure that grows through positive feedback: overload on some instances shifts traffic to the rest, which then overload in turn. The mechanisms in sections 4 to 10 each break one of those feedback loops. Score each mode for severity and likelihood in Software FMEA Template (IEC 60812) (ops/fmea-software).

4. Timeouts and deadline propagation

4.1 Setting timeouts

Set each dependency timeout from its latency distribution at an acceptable false-timeout rate. Example: if a 0.1% false-timeout rate is acceptable, use the p99.9 latency plus a margin.
Separate connection timeout (short, Example: 100 ms in the same region) from request timeout.
Never leave a library default (often infinite or 30 to 60 s) on a request path.

4.2 Deadline propagation

The edge sets an overall deadline for the request (Example: 2 000 ms for checkout). Each hop receives the remaining time and subtracts its own expected work before calling further:

T_{call} = \min\left(T_{dep},\ T_{deadline} - t_{elapsed} - t_{reserve}\right)

where $T_{dep}$ is the dependency's own timeout and $t_{reserve}$ is the time needed to build the response after the call returns. If $T_{call}$ is below the callee's minimum useful time, fail fast instead of calling. Propagate the deadline in a request header or the RPC framework's deadline field, and make callees drop queued work whose deadline has passed.

4.3 Timeout chain check

Hop	Timeout	Must be less than
Client to edge	Example: 3 000 ms (mobile app)	User patience
Edge to Order API	2 000 ms deadline	Client timeout
Order API to payment	1 600 ms or remaining deadline, whichever is lower	Edge deadline minus reserve
Payment to provider	1 400 ms	Order API's payment timeout
Order API to recommendations	150 ms	Its contribution to page latency budget

Rule: an inner timeout must be shorter than the outer one, otherwise the caller gives up while the callee keeps working, and a retry starts a second copy of the same work.

5. Retries, backoff and retry budgets

5.1 Retry policy

Rule	Value
Retryable conditions	Example: connection failure, timeout on idempotent calls, 503 with `Retry-After`, 429 after `Retry-After`
Never retried	Validation errors (4xx other than 408 and 429), non-idempotent calls without an idempotency key
Attempts per request	Example: at most 3 (1 initial and 2 retries)
Where retries happen	At one layer only, closest to the failing dependency

5.2 Exponential backoff with jitter

With base delay $b$ , cap $c$ and retry number $n$ (starting at 0), full jitter draws the wait uniformly:

d_n \sim \mathcal{U}\left(0,\ \min\left(c,\ b \cdot 2^{n}\right)\right)

Example: $b = 100$ ms, $c = 2$ s. The first retry waits up to 100 ms, the second up to 200 ms, the fifth up to 1 600 ms and later retries are capped at 2 s. Jitter spreads clients that failed at the same moment so their retries do not arrive together. Honour a server's Retry-After when it is larger than the computed delay.

5.3 Retry amplification

If each of $d$ layers makes up to $a$ attempts, the deepest dependency can receive

A = a^{d}

times the original load. Example: 3 attempts at each of 3 layers gives $3^3 = 27$ times the load, exactly when the dependency is already failing. This is why retries happen at one layer and are bounded by a budget.

5.4 Retry budget

A retry budget caps retries as a share of successful or total requests over a sliding window. The SRE book describes a per-client budget of retries below 10% of requests, and the Builders' Library article describes limiting retries locally with a token bucket. With budget ratio $\rho$ the worst-case load on a failing dependency is

L_{max} = (1 + \rho) \times L, \qquad \rho = 0.1 \Rightarrow L_{max} = 1.1 \, L

Parameter	Value
Budget ratio	Example: 10% of requests in a 10 s window, per dependency, per instance
Implementation	Example: token bucket: each request adds 0.1 token, each retry spends 1, bucket size 10
When the budget is empty	Fail the request without retrying and count `retry_budget_exhausted`

6. Circuit breakers

Parameter	Value
Scope	Example: one breaker per dependency and per endpoint group, per instance
Failure definition	Timeouts, connection errors and 5xx; not 4xx
Trip threshold	Example: 50% failures with at least 20 requests in a 10 s window
Open interval	Example: 5 s, doubling up to 60 s while probes keep failing
Half-open probes	Example: 3 requests; all must succeed to close
Fallback when open	Per dependency, see section 13

Breakers and retries must agree: a request rejected by an open breaker is not retried against the same dependency.

7. Bulkheads and concurrency limits

By Little's law the average number of requests in flight equals arrival rate times time in system:

N = \lambda \times W

Example: each Order API instance handles $\lambda = 200$ requests/s to inventory with $W = 0.06$ s mean latency, so $N = 12$ calls in flight on average. A bulkhead of 30 concurrent calls (2.5 times the mean) absorbs normal variance; when inventory slows to 0.5 s, demand would be 100 in flight and the bulkhead rejects the excess instead of letting inventory calls take every worker.

Bulkhead	Resource	Limit per instance	On full
BH-01 (Example)	Inventory client pool	30 concurrent	Fail fast with 503 for checkout
BH-02 (Example)	Payment client pool	60 concurrent	Queue up to 20 with deadline, then fail
BH-03 (Example)	Recommendation client pool	10 concurrent	Skip recommendations
BH-04 (Example)	Request workers for batch API vs interactive API	20% / 80% of workers	Reject batch first
[BH-nn]	[Resource]	[Limit]	[Behaviour]

Consider adaptive concurrency limits that adjust the limit from observed latency instead of a fixed number, once fixed limits are proven in tests.

8. Backpressure

Bound every queue (in-process, thread pool, message consumer prefetch). Example: request queue of 50 per instance; at 50, reject new work.
Prefer queues ordered so fresh work wins under overload: drop requests whose deadline has passed before processing, and consider last-in first-out for interactive traffic when the queue is long.
For asynchronous consumers, limit prefetch and in-flight messages, and let the broker hold the backlog. Scale consumers on backlog age, not count.
Signal backpressure to callers explicitly: 429 or 503 with Retry-After over HTTP, flow control in streaming protocols.
Queue time is latency. Example: at 200 requests/s a queue of 50 adds up to 250 ms; with a 2 s deadline this is acceptable, with 200 ms it is not.

9. Load shedding and admission control

9.1 Criticality classes

The SRE book chapter "Handling Overload" describes four criticality levels propagated with each request. Map your traffic to classes like these:

Class	Example traffic	Shed order
CRITICAL_PLUS	Example: payment confirmation, order status for in-flight checkout	Last; shed only to survive
CRITICAL	Example: add to cart, checkout start, user-facing reads	After sheddable classes
SHEDDABLE_PLUS	Example: order history page, partial failure acceptable	Second
SHEDDABLE	Example: batch exports, prefetch, analytics callbacks	First

Criticality is set at the edge from the route and propagated to every downstream call; a dependency sheds by the class of the original request.

9.2 Admission control

Measure capacity per instance as the throughput at which latency still meets the SLO in a load test. Example: 250 requests/s per instance.
Start shedding before saturation. Example: at 90% of measured capacity (225 requests/s) or when the request queue exceeds 30 or CPU exceeds 85%, reject SHEDDABLE; at 100% also reject SHEDDABLE_PLUS; protect CRITICAL classes until the queue deadline check fails.
Reject at the cheapest point: before parsing the body or touching the database, with 503 and Retry-After.
Account for the cost of rejecting: rejection is cheap but not free, so the load balancer and clients must back off too.

9.3 Client-side throttling

As described in the SRE book, a client can track over a window the number of requests it attempted and accepts the backend took, and reject locally with probability

p_{reject} = \max\left(0,\ \frac{requests - K \times accepts}{requests + 1}\right)

with $K = 2$ as the book's usual multiplier. This stops overloaded backends from spending capacity on rejecting requests.

9.4 Goodput under overload

The aim is to keep goodput (work finished within its deadline) close to capacity as offered load rises, instead of letting it collapse. Each request passes this decision before any expensive work starts:

10. Graceful degradation

ID	Trigger	Degraded behaviour	User impact	Exit condition
DG-01 (Example)	Recommendation breaker open	Render page without recommendations	Minor	Breaker closed
DG-02 (Example)	Pricing unavailable	Use cached prices up to 10 minutes old; re-price at payment	Small risk of price change at payment	Pricing healthy for 2 minutes
DG-03 (Example)	Load above 100% of capacity	Disable order history search; serve last 10 orders only	Reduced feature	Load below 80% for 5 minutes
DG-04 (Example)	Event bus unavailable	Buffer events in outbox table	None; notifications delayed	Bus healthy; outbox drained
[DG-nn]	[Trigger]	[Behaviour]	[Impact]	[Exit]

Degraded modes must be visible: emit a metric and a log for every request served in a degraded mode, and show the mode on the service dashboard.

11. Idempotency

Operation	Naturally idempotent	Mechanism
`GET /orders/{id}`	Yes	None needed
`PUT /orders/{id}/address`	Yes (full replacement)	None needed
`POST /orders`	No	`Idempotency-Key` header; key, request hash and response stored 24 h
`POST /orders/{id}/pay`	No	Idempotency key passed to payment and to the provider
Event consumers	No	Deduplicate by event ID in a processed-events table

A repeated key with a different request body is an error (422). Store the idempotency record in the same transaction as the effect, so a crash cannot leave one without the other.

12. Health checks

Check	Purpose	What it tests	Must not test
Liveness	Restart a stuck process	Process responds; event loop not blocked	Any dependency
Readiness	Remove an instance from rotation	Local resources: warmed caches, connection pool open, not draining	Shared dependencies whose failure affects every instance
Startup	Delay liveness until initialised	Configuration loaded, migrations checked
Deep health (dashboard only)	Diagnose	Dependencies reachable	Never used by the load balancer to remove all instances

Rules: fail open if every instance reports unhealthy (keep sending traffic rather than none); stagger restarts; warm up new instances with a slow start on the load balancer so a restarted fleet is not hit at full rate.

13. Per-dependency resilience matrix

Dependency	Timeout	Retries	Budget	Breaker	Bulkhead	Fallback	Degraded mode
DEP-01 Orders database (Example)	300 ms query, 100 ms connect	1 on connection error only	10%	No (pool limits instead)	Pool of 40	None; fail request	None
DEP-02 Inventory (Example)	250 ms	2, idempotent reads	10%	50% over 10 s	BH-01	Reserve later at payment for low-risk items	None
DEP-03 Pricing (Example)	150 ms	1	10%	50% over 10 s	20 concurrent	Cached price	DG-02
DEP-04 Payment (Example)	1 600 ms or deadline	1 with idempotency key	5%	50% over 30 s	BH-02	Queue order as "payment pending"	None
DEP-05 Recommendations (Example)	150 ms	0	none	30% over 10 s	BH-03	Empty list	DG-01
DEP-06 Event bus (Example)	200 ms	Relay retries with backoff	n/a	No	Relay workers 4	Outbox table	DG-04
[DEP-nn]	[Timeout]	[Retries]	[Budget]	[Breaker]	[Bulkhead]	[Fallback]	[Mode]

14. Testing and fault injection

Test ID	Fault injected	Mechanism verified	Expected result	Environment
FT-01 (Example)	Add 2 s latency to recommendations	Timeout, BH-03, DG-01	Checkout p99 unchanged; degraded metric rises	Staging, then production canary
FT-02 (Example)	Payment returns 503 for 50% of calls	Retry budget, breaker	Retry traffic at or below 5%; breaker opens; orders marked pending	Staging
FT-03 (Example)	Load at 2 times capacity	Shedding by class	CRITICAL classes meet SLO; SHEDDABLE rejected with 503	Load test environment
FT-04 (Example)	Kill 30% of instances	Health checks, slow start	No crash loop; recovery within 2 minutes	Game day
FT-05 (Example)	Database connection exhaustion	Pool limits, fail fast	Errors bounded; no thread starvation	Staging
[FT-nn]	[Fault]	[Mechanism]	[Result]	[Environment]

Plan individual experiments with Chaos Experiment Plan Template (ops/chaos-experiment) and team exercises with Game Day Plan Template (ops/game-day-plan).

15. Metrics and alerts

Signal	Metric	Alert
Goodput	Successful requests/s by criticality class	CRITICAL success below SLO for 5 min
Shed rate	Rejected requests/s by class and reason	Any CRITICAL_PLUS shed
Retry ratio	Retries / requests per dependency	Above budget for 5 min
Breaker state	Open breakers per dependency	Open for more than 2 min
Bulkhead saturation	In-flight / limit per bulkhead	Above 90% for 5 min
Queue time	p99 time in request queue	Above 25% of deadline
Deadline drops	Requests dropped as expired	Rising trend vs. baseline
Degraded mode	Requests served per degraded mode	Any mode active for more than 30 min

Tie alert thresholds to the error budget in SLO Document Template (dev/slo-doc) and signal definitions to Observability Specification Template (OpenTelemetry) (ops/observability-spec).

16. Decisions, risks and traceability

16.1 Decisions

ID	Decision	Alternatives	Reason
DEC-01 (Example)	Retries only in the client library closest to each dependency	Retries at every layer	Avoids amplification (5.3)
DEC-02 (Example)	Readiness excludes shared dependencies	Deep readiness	Prevents fleet-wide removal
DEC-03 (Example)	Fixed bulkheads first, adaptive limits later	Adaptive from day one	Simpler to test and reason about
[DEC-nn]	[Decision]	[Alternatives]	[Reason]

16.2 Risks

ID	Risk	Likelihood	Impact	Mitigation	Owner
R-01	Example: Shedding thresholds drift as code changes capacity	Medium	High	Quarterly load test; thresholds in config	[@name]
R-02	Example: Clients ignore `Retry-After`	High	Medium	Client SDK enforces backoff; edge rate limits	[@name]
R-03	Example: Degraded pricing causes customer complaints	Low	Medium	Re-price at payment; notify on change	[@name]
[R-nn]	[Risk]	[L / M / H]	[L / M / H]	[Mitigation]	[@name]

16.3 Traceability

Objective	Failure modes	Mechanisms	Tests
RO-01	FMD-01, FMD-02	Timeouts (4), BH-03, DG-01, matrix row DEP-05	FT-01
RO-02	FMD-04	Shedding (9), backpressure (8)	FT-03
RO-03	FMD-02, FMD-04	Retry budget (5.4), breakers (6)	FT-02
RO-04	FMD-05, FMD-07	Health checks (12), breaker half-open	FT-04
[RO-nn]	[Modes]	[Mechanisms]	[Tests]

17. Glossary and references

17.1 Glossary

Term	Meaning
Deadline	Absolute time by which the caller needs an answer, propagated down the call chain
Goodput	Useful work completed within its deadline, as opposed to raw throughput
Retry budget	Upper bound on retries as a share of requests
Bulkhead	Resource partition that limits how much of a service one dependency or workload can consume
Load shedding	Deliberately rejecting some requests to keep serving the rest within target
Backpressure	Signal from a consumer that makes a producer slow down
Criticality	Importance class carried with a request that decides shed order

17.2 References

Reference	Use
Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering, O'Reilly, 2016, chapter 21 "Handling Overload" (sre.google/sre-book)	Criticality, client throttling, retry budget (9.1, 9.3, 5.4)
Site Reliability Engineering, chapter 22 "Addressing Cascading Failures"	Failure model, deadlines, health checks (3, 4, 12)
Amazon Builders' Library, "Timeouts, retries, and backoff with jitter"	Timeout selection, jitter, local retry limiting (4, 5)
Amazon Builders' Library, "Using load shedding to avoid overload"	Admission control and goodput (9)
Software FMEA Template (IEC 60812) (`ops/fmea-software`)	Scoring failure modes
Capacity Plan Template (`ops/capacity-plan`)	Capacity figures used in 9.2

Standard and licence note

Based on publicly documented industry practice, in particular the Google Site Reliability Engineering book chapters and the Amazon Builders' Library articles listed in 17.2; criticality level names and the client throttling formula are cited with attribution. No text from those works is reproduced. Check the current version of each source. Numbers are illustrative assumptions. Original text written for this collection. Template licence: MIT.

Схожі шаблони

Системний дизайн