# Resilience Design: [Service name]

> Design record for how one service survives slow, failing and overloaded
> dependencies and its own overload: failure model, timeouts and deadline
> propagation, retries with exponential backoff and jitter, retry budgets,
> circuit breakers, bulkheads, backpressure, load shedding and admission
> control by criticality, graceful degradation, idempotency, health checks,
> testing with fault injection, metrics and alerts, and a per-dependency
> resilience matrix. Concepts follow the Google *Site Reliability
> Engineering* book (chapters 21 "Handling Overload" and 22 "Addressing
> Cascading Failures") and the Amazon Builders' Library articles "Timeouts,
> retries, and backoff with jitter" and "Using load shedding to avoid
> overload". Replace every `[bracketed]` value and every row marked
> *Example*; numbers are illustrative assumptions.

<!--
Guidance: How to use this template
- Fill the dependency inventory (2.2) and the failure model (3) before
  choosing mechanisms. Every timeout, retry and breaker in this document must
  name the dependency and failure it handles.
- Write numbers, not adjectives: "timeout 200 ms, 2 retries, budget 10%" can
  be reviewed and tested; "reasonable timeouts" cannot.
- Remember that resilience mechanisms interact. Retries at every layer
  multiply load; a timeout shorter than the callee's own timeout wastes work.
  Check the whole call chain in section 4.3 and 5.3.
- Prove each mechanism with a fault injection test (section 14). An untested
  circuit breaker is a guess.
-->

## Document control

<!-- Guidance: Identify the service, its owners and the reviewers who accept the residual risk. -->

| Field | Value |
|---|---|
| Document ID | [RES-ORD-001] |
| Service | [Order API] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@service-owner] |
| Reviewers | [@sre], [@staff-engineer] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Service tier | [Tier 1 / 2 / 3] |
| SLO reference | [Link to SLO document] |
| Related | [System design], [FMEA], [Chaos experiments], [Runbook] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Service owner | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| SRE lead | [@name] | [Decision] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Owners of critical dependencies | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record changes to limits, budgets and degradation behaviour, and the incident or test that prompted them. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added retry budget after retry storm in incident INC-[nnn] |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Criticality classes and shedding thresholds from load test |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Introduction and scope](#1-introduction-and-scope)
2. [Service context and dependencies](#2-service-context-and-dependencies)
3. [Failure model](#3-failure-model)
4. [Timeouts and deadline propagation](#4-timeouts-and-deadline-propagation)
5. [Retries, backoff and retry budgets](#5-retries-backoff-and-retry-budgets)
6. [Circuit breakers](#6-circuit-breakers)
7. [Bulkheads and concurrency limits](#7-bulkheads-and-concurrency-limits)
8. [Backpressure](#8-backpressure)
9. [Load shedding and admission control](#9-load-shedding-and-admission-control)
10. [Graceful degradation](#10-graceful-degradation)
11. [Idempotency](#11-idempotency)
12. [Health checks](#12-health-checks)
13. [Per-dependency resilience matrix](#13-per-dependency-resilience-matrix)
14. [Testing and fault injection](#14-testing-and-fault-injection)
15. [Metrics and alerts](#15-metrics-and-alerts)
16. [Decisions, risks and traceability](#16-decisions-risks-and-traceability)
17. [Glossary and references](#17-glossary-and-references)

## 1. Introduction and scope

<!-- Guidance: State what the service must keep doing when things go wrong, and which failures are explicitly not handled here (for example region loss, covered by the disaster recovery plan). -->

### 1.1 Purpose

[*Example:* The Order API accepts and tracks customer orders. It depends on
inventory, pricing, payment, a recommendation service and a relational
database. In the last two quarters it suffered two cascading failures: a
slow recommendation service exhausted the request thread pool, and a retry
storm against payment tripled payment's load during its own incident. This
document defines how the Order API contains those failures.]

### 1.2 Resilience objectives

| ID | Objective | Target |
|---|---|---|
| RO-01 | *Example:* Checkout stays available when any non-critical dependency fails | Checkout success at or above 99.9% with recommendations down |
| RO-02 | *Example:* Overload degrades gracefully instead of collapsing | Goodput at 2 times capacity stays at or above 90% of capacity |
| RO-03 | *Example:* The service never amplifies a dependency incident | Retry traffic at or below 10% of request traffic at all times |
| RO-04 | *Example:* Recovery is automatic after a dependency recovers | Back to normal within 2 minutes without manual action |
| [RO-nn] | [Objective] | [Target] |

### 1.3 Out of scope

- [*Example:* loss of a full region (see IT Disaster Recovery Plan Template (`ops/disaster-recovery-plan`))]
- [*Example:* data corruption and backup restore]
- [*Example:* volumetric network attacks (edge provider)]

## 2. Service context and dependencies

<!-- Guidance: Draw the call graph with every synchronous and asynchronous dependency. Mark which ones are on the critical path of the most important user journey. -->

### 2.1 Call graph

Solid arrows are synchronous calls on the checkout path; dotted arrows are
optional or asynchronous.

```mermaid
flowchart LR
  client([Clients and mobile apps]) --> lb[Load balancer]
  lb --> api[Order API]
  api --> db[(Orders database)]
  api --> inv[Inventory service]
  api --> price[Pricing service]
  api --> pay[Payment service]
  api -.->|optional| reco[Recommendation service]
  api -.->|async| bus[[Event bus]]
  pay --> psp[[External payment provider]]
```

### 2.2 Dependency inventory

| ID | Dependency | Type | Criticality for checkout | Normal p50 / p99 latency | Owner |
|---|---|---|---|---|---|
| DEP-01 | *Example:* Orders database | Sync, stateful | Hard (no checkout without it) | 3 ms / 25 ms | [@team] |
| DEP-02 | *Example:* Inventory service | Sync | Hard | 15 ms / 120 ms | [@team] |
| DEP-03 | *Example:* Pricing service | Sync | Soft (cached prices acceptable for 10 minutes) | 10 ms / 80 ms | [@team] |
| DEP-04 | *Example:* Payment service | Sync | Hard | 250 ms / 1 500 ms | [@team] |
| DEP-05 | *Example:* Recommendation service | Sync | None (page renders without it) | 40 ms / 300 ms | [@team] |
| DEP-06 | *Example:* Event bus | Async | Soft (outbox buffers events) | 5 ms / 50 ms | [@team] |
| [DEP-nn] | [Dependency] | [Sync / Async] | [Hard / Soft / None] | [Latency] | [@team] |

## 3. Failure model

<!-- Guidance: List the ways each dependency and the service itself can fail. Slow is worse than down: most cascades start with latency, not errors. -->

| ID | Failure mode | Typical cause | Effect without protection |
|---|---|---|---|
| FMD-01 | Dependency slow (latency 10 to 100 times normal) | Garbage collection, lock contention, overloaded callee | Threads and connections held; our own queue grows; cascade |
| FMD-02 | Dependency returns errors | Bug, bad deploy, exhausted resource | Errors propagate; retries add load |
| FMD-03 | Dependency unreachable | Network partition, DNS, certificate expiry | Connection timeouts; slow failure if timeouts are long |
| FMD-04 | Own overload | Traffic spike, retry storm from clients, reduced capacity after a failure | Latency rises for everyone; work completes after clients have given up |
| FMD-05 | Partial instance failure | One bad host or zone | Fraction of requests fail; health checks may not see it |
| FMD-06 | Resource exhaustion | Memory leak, file descriptors, connection pool | Gradual degradation then crash |
| FMD-07 | Correlated restart | Crash loop, simultaneous deploy | Capacity drop; thundering herd on dependencies at start |
| [FMD-nn] | [Mode] | [Cause] | [Effect] |

A cascading failure, as described in the SRE book chapter "Addressing
Cascading Failures", is a failure that grows through positive feedback:
overload on some instances shifts traffic to the rest, which then overload in
turn. The mechanisms in sections 4 to 10 each break one of those feedback
loops. Score each mode for severity and likelihood in Software FMEA Template
(IEC 60812) (`ops/fmea-software`).

## 4. Timeouts and deadline propagation

<!-- Guidance: Every network call needs a timeout. Set it from measured latency, not from a round number, and make sure the caller's deadline is passed down so callees stop work nobody is waiting for. -->

### 4.1 Setting timeouts

- Set each dependency timeout from its latency distribution at an
  acceptable false-timeout rate. *Example:* if a 0.1% false-timeout rate is
  acceptable, use the p99.9 latency plus a margin.
- Separate connection timeout (short, *Example:* 100 ms in the same region)
  from request timeout.
- Never leave a library default (often infinite or 30 to 60 s) on a request
  path.

### 4.2 Deadline propagation

The edge sets an overall deadline for the request (*Example:* 2 000 ms for
checkout). Each hop receives the remaining time and subtracts its own
expected work before calling further:

$$
T_{call} = \min\left(T_{dep},\ T_{deadline} - t_{elapsed} - t_{reserve}\right)
$$

where $T_{dep}$ is the dependency's own timeout and $t_{reserve}$ is the time
needed to build the response after the call returns. If $T_{call}$ is below
the callee's minimum useful time, fail fast instead of calling. Propagate the
deadline in a request header or the RPC framework's deadline field, and make
callees drop queued work whose deadline has passed.

### 4.3 Timeout chain check

| Hop | Timeout | Must be less than |
|---|---|---|
| Client to edge | *Example:* 3 000 ms (mobile app) | User patience |
| Edge to Order API | 2 000 ms deadline | Client timeout |
| Order API to payment | 1 600 ms or remaining deadline, whichever is lower | Edge deadline minus reserve |
| Payment to provider | 1 400 ms | Order API's payment timeout |
| Order API to recommendations | 150 ms | Its contribution to page latency budget |

Rule: an inner timeout must be shorter than the outer one, otherwise the
caller gives up while the callee keeps working, and a retry starts a second
copy of the same work.

## 5. Retries, backoff and retry budgets

<!-- Guidance: Retry only what is safe and likely to succeed on a second try. Bound retries per request and per client, and add jitter so clients do not retry in lockstep. -->

### 5.1 Retry policy

| Rule | Value |
|---|---|
| Retryable conditions | *Example:* connection failure, timeout on idempotent calls, 503 with `Retry-After`, 429 after `Retry-After` |
| Never retried | Validation errors (4xx other than 408 and 429), non-idempotent calls without an idempotency key |
| Attempts per request | *Example:* at most 3 (1 initial and 2 retries) |
| Where retries happen | At one layer only, closest to the failing dependency |

### 5.2 Exponential backoff with jitter

With base delay $b$, cap $c$ and retry number $n$ (starting at 0), full
jitter draws the wait uniformly:

$$
d_n \sim \mathcal{U}\left(0,\ \min\left(c,\ b \cdot 2^{n}\right)\right)
$$

*Example:* $b = 100$ ms, $c = 2$ s. The first retry waits up to 100 ms, the
second up to 200 ms, the fifth up to 1 600 ms and later retries are capped at
2 s. Jitter spreads clients that failed at the same moment so their retries
do not arrive together. Honour a server's `Retry-After` when it is larger than
the computed delay.

### 5.3 Retry amplification

If each of $d$ layers makes up to $a$ attempts, the deepest dependency can
receive

$$
A = a^{d}
$$

times the original load. *Example:* 3 attempts at each of 3 layers gives
$3^3 = 27$ times the load, exactly when the dependency is already failing.
This is why retries happen at one layer and are bounded by a budget.

### 5.4 Retry budget

A retry budget caps retries as a share of successful or total requests over
a sliding window. The SRE book describes a per-client budget of retries
below 10% of requests, and the Builders' Library article describes limiting
retries locally with a token bucket. With budget ratio $\rho$ the worst-case
load on a failing dependency is

$$
L_{max} = (1 + \rho) \times L, \qquad \rho = 0.1 \Rightarrow L_{max} = 1.1 \, L
$$

| Parameter | Value |
|---|---|
| Budget ratio | *Example:* 10% of requests in a 10 s window, per dependency, per instance |
| Implementation | *Example:* token bucket: each request adds 0.1 token, each retry spends 1, bucket size 10 |
| When the budget is empty | Fail the request without retrying and count `retry_budget_exhausted` |

## 6. Circuit breakers

<!-- Guidance: A breaker stops calling a dependency that is clearly failing, so requests fail fast and the dependency gets room to recover. Define thresholds, the open-state fallback and how it probes recovery. -->

```mermaid
stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failure rate over threshold in window
  Open --> HalfOpen: after open interval
  HalfOpen --> Closed: probe requests succeed
  HalfOpen --> Open: any probe fails
```

| Parameter | Value |
|---|---|
| Scope | *Example:* one breaker per dependency and per endpoint group, per instance |
| Failure definition | Timeouts, connection errors and 5xx; not 4xx |
| Trip threshold | *Example:* 50% failures with at least 20 requests in a 10 s window |
| Open interval | *Example:* 5 s, doubling up to 60 s while probes keep failing |
| Half-open probes | *Example:* 3 requests; all must succeed to close |
| Fallback when open | Per dependency, see section 13 |

Breakers and retries must agree: a request rejected by an open breaker is
not retried against the same dependency.

## 7. Bulkheads and concurrency limits

<!-- Guidance: Isolate resources so one slow dependency or one tenant cannot consume everything. Size pools from measured throughput and latency with Little's law. -->

By Little's law the average number of requests in flight equals arrival rate
times time in system:

$$
N = \lambda \times W
$$

*Example:* each Order API instance handles $\lambda = 200$ requests/s to
inventory with $W = 0.06$ s mean latency, so $N = 12$ calls in flight on
average. A bulkhead of 30 concurrent calls (2.5 times the mean) absorbs
normal variance; when inventory slows to 0.5 s, demand would be 100 in
flight and the bulkhead rejects the excess instead of letting inventory
calls take every worker.

| Bulkhead | Resource | Limit per instance | On full |
|---|---|---|---|
| BH-01 *(Example)* | Inventory client pool | 30 concurrent | Fail fast with 503 for checkout |
| BH-02 *(Example)* | Payment client pool | 60 concurrent | Queue up to 20 with deadline, then fail |
| BH-03 *(Example)* | Recommendation client pool | 10 concurrent | Skip recommendations |
| BH-04 *(Example)* | Request workers for batch API vs interactive API | 20% / 80% of workers | Reject batch first |
| [BH-nn] | [Resource] | [Limit] | [Behaviour] |

Consider adaptive concurrency limits that adjust the limit from observed
latency instead of a fixed number, once fixed limits are proven in tests.

## 8. Backpressure

<!-- Guidance: When a consumer cannot keep up, the producer must slow down or stop. Unbounded queues hide overload until memory runs out and latency is already unrecoverable. -->

- Bound every queue (in-process, thread pool, message consumer prefetch).
  *Example:* request queue of 50 per instance; at 50, reject new work.
- Prefer queues ordered so fresh work wins under overload: drop requests
  whose deadline has passed before processing, and consider last-in
  first-out for interactive traffic when the queue is long.
- For asynchronous consumers, limit prefetch and in-flight messages, and let
  the broker hold the backlog. Scale consumers on backlog age, not count.
- Signal backpressure to callers explicitly: 429 or 503 with `Retry-After`
  over HTTP, flow control in streaming protocols.
- Queue time is latency. *Example:* at 200 requests/s a queue of 50 adds up
  to 250 ms; with a 2 s deadline this is acceptable, with 200 ms it is not.

## 9. Load shedding and admission control

<!-- Guidance: Decide in advance which work to drop when demand exceeds capacity, and drop it early and cheaply. Shedding the least important work keeps the most important work inside its SLO. -->

### 9.1 Criticality classes

The SRE book chapter "Handling Overload" describes four criticality levels
propagated with each request. Map your traffic to classes like these:

| Class | Example traffic | Shed order |
|---|---|---|
| CRITICAL_PLUS | *Example:* payment confirmation, order status for in-flight checkout | Last; shed only to survive |
| CRITICAL | *Example:* add to cart, checkout start, user-facing reads | After sheddable classes |
| SHEDDABLE_PLUS | *Example:* order history page, partial failure acceptable | Second |
| SHEDDABLE | *Example:* batch exports, prefetch, analytics callbacks | First |

Criticality is set at the edge from the route and propagated to every
downstream call; a dependency sheds by the class of the original request.

### 9.2 Admission control

- Measure capacity per instance as the throughput at which latency still
  meets the SLO in a load test. *Example:* 250 requests/s per instance.
- Start shedding before saturation. *Example:* at 90% of measured capacity
  (225 requests/s) or when the request queue exceeds 30 or CPU exceeds 85%,
  reject SHEDDABLE; at 100% also reject SHEDDABLE_PLUS; protect CRITICAL
  classes until the queue deadline check fails.
- Reject at the cheapest point: before parsing the body or touching the
  database, with 503 and `Retry-After`.
- Account for the cost of rejecting: rejection is cheap but not free, so the
  load balancer and clients must back off too.

### 9.3 Client-side throttling

As described in the SRE book, a client can track over a window the number of
`requests` it attempted and `accepts` the backend took, and reject locally
with probability

$$
p_{reject} = \max\left(0,\ \frac{requests - K \times accepts}{requests + 1}\right)
$$

with $K = 2$ as the book's usual multiplier. This stops overloaded backends
from spending capacity on rejecting requests.

### 9.4 Goodput under overload

The aim is to keep goodput (work finished within its deadline) close to
capacity as offered load rises, instead of letting it collapse. Each request
passes this decision before any expensive work starts:

```mermaid
flowchart TD
  req[Incoming request] --> dl{Deadline already passed?}
  dl -->|yes| drop1[Drop, no work done]
  dl -->|no| load{Load above shed threshold?}
  load -->|no| serve[Serve]
  load -->|yes| crit{Class sheddable at this level?}
  crit -->|yes| rej[Reject 503 with Retry-After]
  crit -->|no| serve
```

## 10. Graceful degradation

<!-- Guidance: List what the service does with less: cached data, reduced features, static fallbacks. Each degraded mode needs a trigger, an owner and a way back. -->

| ID | Trigger | Degraded behaviour | User impact | Exit condition |
|---|---|---|---|---|
| DG-01 *(Example)* | Recommendation breaker open | Render page without recommendations | Minor | Breaker closed |
| DG-02 *(Example)* | Pricing unavailable | Use cached prices up to 10 minutes old; re-price at payment | Small risk of price change at payment | Pricing healthy for 2 minutes |
| DG-03 *(Example)* | Load above 100% of capacity | Disable order history search; serve last 10 orders only | Reduced feature | Load below 80% for 5 minutes |
| DG-04 *(Example)* | Event bus unavailable | Buffer events in outbox table | None; notifications delayed | Bus healthy; outbox drained |
| [DG-nn] | [Trigger] | [Behaviour] | [Impact] | [Exit] |

Degraded modes must be visible: emit a metric and a log for every request
served in a degraded mode, and show the mode on the service dashboard.

## 11. Idempotency

<!-- Guidance: Retries are only safe when repeating a request has the same effect as doing it once. Name the operations that are naturally idempotent and how the others are made so. -->

| Operation | Naturally idempotent | Mechanism |
|---|---|---|
| `GET /orders/{id}` | Yes | None needed |
| `PUT /orders/{id}/address` | Yes (full replacement) | None needed |
| `POST /orders` | No | `Idempotency-Key` header; key, request hash and response stored 24 h |
| `POST /orders/{id}/pay` | No | Idempotency key passed to payment and to the provider |
| Event consumers | No | Deduplicate by event ID in a processed-events table |

A repeated key with a different request body is an error (422). Store the
idempotency record in the same transaction as the effect, so a crash cannot
leave one without the other.

## 12. Health checks

<!-- Guidance: Health checks decide whether an instance gets traffic or gets restarted. A check that depends on shared dependencies can take down the whole fleet when one dependency fails. -->

| Check | Purpose | What it tests | Must not test |
|---|---|---|---|
| Liveness | Restart a stuck process | Process responds; event loop not blocked | Any dependency |
| Readiness | Remove an instance from rotation | Local resources: warmed caches, connection pool open, not draining | Shared dependencies whose failure affects every instance |
| Startup | Delay liveness until initialised | Configuration loaded, migrations checked | |
| Deep health (dashboard only) | Diagnose | Dependencies reachable | Never used by the load balancer to remove all instances |

Rules: fail open if every instance reports unhealthy (keep sending traffic
rather than none); stagger restarts; warm up new instances with a slow start
on the load balancer so a restarted fleet is not hit at full rate.

## 13. Per-dependency resilience matrix

<!-- Guidance: One row per dependency so reviewers can check that every dependency has every mechanism set, and that the values agree with sections 4 to 10. -->

| Dependency | Timeout | Retries | Budget | Breaker | Bulkhead | Fallback | Degraded mode |
|---|---|---|---|---|---|---|---|
| DEP-01 Orders database *(Example)* | 300 ms query, 100 ms connect | 1 on connection error only | 10% | No (pool limits instead) | Pool of 40 | None; fail request | None |
| DEP-02 Inventory *(Example)* | 250 ms | 2, idempotent reads | 10% | 50% over 10 s | BH-01 | Reserve later at payment for low-risk items | None |
| DEP-03 Pricing *(Example)* | 150 ms | 1 | 10% | 50% over 10 s | 20 concurrent | Cached price | DG-02 |
| DEP-04 Payment *(Example)* | 1 600 ms or deadline | 1 with idempotency key | 5% | 50% over 30 s | BH-02 | Queue order as "payment pending" | None |
| DEP-05 Recommendations *(Example)* | 150 ms | 0 | none | 30% over 10 s | BH-03 | Empty list | DG-01 |
| DEP-06 Event bus *(Example)* | 200 ms | Relay retries with backoff | n/a | No | Relay workers 4 | Outbox table | DG-04 |
| [DEP-nn] | [Timeout] | [Retries] | [Budget] | [Breaker] | [Bulkhead] | [Fallback] | [Mode] |

## 14. Testing and fault injection

<!-- Guidance: Each mechanism above needs a test that injects the failure it handles and checks the expected behaviour and metrics. Run them before launch and after significant changes. -->

| Test ID | Fault injected | Mechanism verified | Expected result | Environment |
|---|---|---|---|---|
| FT-01 *(Example)* | Add 2 s latency to recommendations | Timeout, BH-03, DG-01 | Checkout p99 unchanged; degraded metric rises | Staging, then production canary |
| FT-02 *(Example)* | Payment returns 503 for 50% of calls | Retry budget, breaker | Retry traffic at or below 5%; breaker opens; orders marked pending | Staging |
| FT-03 *(Example)* | Load at 2 times capacity | Shedding by class | CRITICAL classes meet SLO; SHEDDABLE rejected with 503 | Load test environment |
| FT-04 *(Example)* | Kill 30% of instances | Health checks, slow start | No crash loop; recovery within 2 minutes | Game day |
| FT-05 *(Example)* | Database connection exhaustion | Pool limits, fail fast | Errors bounded; no thread starvation | Staging |
| [FT-nn] | [Fault] | [Mechanism] | [Result] | [Environment] |

Plan individual experiments with Chaos Experiment Plan Template
(`ops/chaos-experiment`) and team exercises with Game Day Plan Template
(`ops/game-day-plan`).

## 15. Metrics and alerts

<!-- Guidance: Measure the mechanisms, not only the service: breaker state, shed rate and retry budget use tell you how close you are to a cascade. -->

| Signal | Metric | Alert |
|---|---|---|
| Goodput | Successful requests/s by criticality class | CRITICAL success below SLO for 5 min |
| Shed rate | Rejected requests/s by class and reason | Any CRITICAL_PLUS shed |
| Retry ratio | Retries / requests per dependency | Above budget for 5 min |
| Breaker state | Open breakers per dependency | Open for more than 2 min |
| Bulkhead saturation | In-flight / limit per bulkhead | Above 90% for 5 min |
| Queue time | p99 time in request queue | Above 25% of deadline |
| Deadline drops | Requests dropped as expired | Rising trend vs. baseline |
| Degraded mode | Requests served per degraded mode | Any mode active for more than 30 min |

Tie alert thresholds to the error budget in SLO Document Template
(`dev/slo-doc`) and signal definitions to Observability Specification
Template (OpenTelemetry) (`ops/observability-spec`).

## 16. Decisions, risks and traceability

<!-- Guidance: Record the contested choices, the residual risks accepted, and trace each objective to mechanisms and tests. -->

### 16.1 Decisions

| ID | Decision | Alternatives | Reason |
|---|---|---|---|
| DEC-01 *(Example)* | Retries only in the client library closest to each dependency | Retries at every layer | Avoids amplification (5.3) |
| DEC-02 *(Example)* | Readiness excludes shared dependencies | Deep readiness | Prevents fleet-wide removal |
| DEC-03 *(Example)* | Fixed bulkheads first, adaptive limits later | Adaptive from day one | Simpler to test and reason about |
| [DEC-nn] | [Decision] | [Alternatives] | [Reason] |

### 16.2 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Shedding thresholds drift as code changes capacity | Medium | High | Quarterly load test; thresholds in config | [@name] |
| R-02 | *Example:* Clients ignore `Retry-After` | High | Medium | Client SDK enforces backoff; edge rate limits | [@name] |
| R-03 | *Example:* Degraded pricing causes customer complaints | Low | Medium | Re-price at payment; notify on change | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

### 16.3 Traceability

| Objective | Failure modes | Mechanisms | Tests |
|---|---|---|---|
| RO-01 | FMD-01, FMD-02 | Timeouts (4), BH-03, DG-01, matrix row DEP-05 | FT-01 |
| RO-02 | FMD-04 | Shedding (9), backpressure (8) | FT-03 |
| RO-03 | FMD-02, FMD-04 | Retry budget (5.4), breakers (6) | FT-02 |
| RO-04 | FMD-05, FMD-07 | Health checks (12), breaker half-open | FT-04 |
| [RO-nn] | [Modes] | [Mechanisms] | [Tests] |

## 17. Glossary and references

<!-- Guidance: Define terms the team uses differently, and cite the sources the design relies on. -->

### 17.1 Glossary

| Term | Meaning |
|---|---|
| Deadline | Absolute time by which the caller needs an answer, propagated down the call chain |
| Goodput | Useful work completed within its deadline, as opposed to raw throughput |
| Retry budget | Upper bound on retries as a share of requests |
| Bulkhead | Resource partition that limits how much of a service one dependency or workload can consume |
| Load shedding | Deliberately rejecting some requests to keep serving the rest within target |
| Backpressure | Signal from a consumer that makes a producer slow down |
| Criticality | Importance class carried with a request that decides shed order |

### 17.2 References

| Reference | Use |
|---|---|
| Beyer, Jones, Petoff and Murphy (eds.), *Site Reliability Engineering*, O'Reilly, 2016, chapter 21 "Handling Overload" (sre.google/sre-book) | Criticality, client throttling, retry budget (9.1, 9.3, 5.4) |
| *Site Reliability Engineering*, chapter 22 "Addressing Cascading Failures" | Failure model, deadlines, health checks (3, 4, 12) |
| Amazon Builders' Library, "Timeouts, retries, and backoff with jitter" | Timeout selection, jitter, local retry limiting (4, 5) |
| Amazon Builders' Library, "Using load shedding to avoid overload" | Admission control and goodput (9) |
| Software FMEA Template (IEC 60812) (`ops/fmea-software`) | Scoring failure modes |
| Capacity Plan Template (`ops/capacity-plan`) | Capacity figures used in 9.2 |

## Standard and licence note

Based on publicly documented industry practice, in particular the Google
*Site Reliability Engineering* book chapters and the Amazon Builders' Library
articles listed in 17.2; criticality level names and the client throttling
formula are cited with attribution. No text from those works is reproduced.
Check the current version of each source. Numbers are illustrative
assumptions. Original text written for this collection. Template licence:
MIT.