Markdown Editor

Error Budget Policy Template

Error budget policy: SLOs covered, budget and burn-rate maths, thresholds, feature freeze, escalation, exceptions, sign-off

使い方: Use it when a service has SLOs and product and engineering need to agree what happens when reliability slips. Fill the SLO list and budget calculation first, then graduated thresholds with concrete consequences. The main pitfall is vague consequences or a policy signed by only one side.

プレビュー

Error Budget Policy: [Service or product name]

Policy that turns service level objectives (SLOs) into decisions: how the error budget is calculated, what happens when it is being spent too fast or is exhausted, who decides, which exceptions apply and how the policy is reviewed. Based on the error budget concept from site reliability engineering practice. Replace every [bracketed] value and every row marked Example.

FieldValue
Policy ID[EBP-SERVICE-001]
Service / product[Name]
Version[0.1]
Status[Draft / Approved / Superseded]
Policy owner[@sre-lead]
Product owner[@product-manager]
Effective fromYYYY-MM-DD
Next reviewYYYY-MM-DD
Related[SLO document], [On-call handbook], [Incident response plan], [Release process]

1. Purpose and principles

This policy balances the speed of change against the reliability users expect. The SLO defines how unreliable the service is allowed to be; the difference between perfect and the SLO is the error budget. While budget remains, teams may spend it on releases, experiments and planned risk. When it is spent, reliability work takes priority until the service is back within objective.

Principles:

  • 100% is the wrong target. Users cannot tell the difference above the SLO, and chasing it slows delivery.
  • The budget belongs to the product, not to one team. Spending and saving it are joint decisions.
  • Consequences are triggered by data, not by opinion or by the loudest voice in the room.
  • The policy applies to all changes, including those from platform and dependency owners, unless an exception in section 6 applies.

2. SLOs covered

SLO IDSLITargetWindowSource
SLO-01Example: Availability: share of HTTP requests that return non-5xx within 1 s99.9%28 days rolling[SLO doc link]
SLO-02Example: Latency: share of checkout requests under 400 ms99%28 days rolling[SLO doc link]
SLO-03Example: Freshness: share of minutes where order status lag is under 60 s99.5%28 days rolling[SLO doc link]
[SLO-NN][SLI][Target][Window][Source]

3. Budget calculation

For an SLO target TT over a window, the error budget is the allowed share of bad events:

Error budget=1−T\text{Error budget} = 1 - T

For request-based SLIs, the remaining budget at any time is:

Budget remaining=1−bad events(1−T)×total events\text{Budget remaining} = 1 - \frac{\text{bad events}}{(1 - T) \times \text{total events}}

Example: SLO-01 has T=99.9%T = 99.9\% and 28 days of traffic with 50,000,00050{,}000{,}000 requests. The budget is 0.001×50,000,000=50,0000.001 \times 50{,}000{,}000 = 50{,}000 bad requests. With 18,50018{,}500 bad requests so far, the remaining budget is 1−18,500/50,000=63%1 - 18{,}500 / 50{,}000 = 63\%.

For time-based reasoning, a 99.9% availability target over 28 days allows 0.001×28×24×60≈400.001 \times 28 \times 24 \times 60 \approx 40 minutes of full downtime.

The burn rate shows how fast the budget is being spent relative to the rate that would use exactly all of it by the end of the window:

Burn rate=observed error rate1−T\text{Burn rate} = \frac{\text{observed error rate}}{1 - T}

A burn rate of 1 spends the budget exactly over the window; a burn rate of 14.4 sustained for one hour spends 2% of a 30-day budget.

Live values are shown on [dashboard link] and reported weekly in [channel or meeting].

4. Policy thresholds and consequences

LevelTriggerConsequencesDecided by
GreenBudget remaining above 50% and no fast burn alertNormal release cadence; planned risky changes and experiments allowedTeam
YellowBudget remaining 25% to 50%, or burn rate above [2] over 6 hRelease only with canary and automated rollback; postpone planned chaos experiments in production; review top budget consumers at next stand-upEngineering lead
OrangeBudget remaining 0% to 25%Feature releases need product and engineering lead approval; at least [50%] of team capacity moves to reliability backlogProduct and engineering leads
RedBudget exhausted (below 0%) over the windowFeature freeze: only fixes for reliability, security and legal issues ship; postmortem for each major consumer; reliability work until budget is positive for [7] consecutive daysProduct and engineering leads, informed: director

4.1 Single-incident rule

Any single incident that consumes more than [20%] of the window's budget requires a postmortem with at least one priority action to prevent recurrence, whatever the overall level.

4.2 Reliability work

Reliability work under Orange or Red is chosen from the reliability backlog ranked by budget impact. Example: fix the retry storm found in INC-2310, add load shedding to the search API, raise test coverage on the checkout flow from 61% to 80%.

5. Escalation

SituationEscalate toTimeframe
Product and engineering disagree on applying a consequence[Director of engineering] and [head of product] jointlyDecision within 2 working days
Budget consumed mainly by a dependency owned by another teamOwner of the dependency and its SLO; raise in [reliability review]Next weekly review
Repeated Red level in two consecutive windows[VP engineering]; consider revising SLO, architecture or staffingWithin one window
Customer contract (SLA) at risk[Account owner] and [legal or commercial lead]Same day

6. Exceptions

The following do not count against the budget, or allow releases during a freeze, when approved by [role] and recorded in the exceptions log:

  • Outages caused by a dependency with its own SLO that is covered by its own policy, when this service degraded gracefully as designed.
  • Planned maintenance announced at least [5 working days] in advance and within the agreed maintenance window.
  • Security fixes and legal or regulatory changes, which may always ship.
  • Load from abuse or attack that is blocked within [15 minutes] of detection.
  • Fixes that reduce budget burn, which are always allowed.
Exception IDDateDescriptionBudget impact excludedApproved by
EX-01Example: YYYY-MM-DDCloud provider regional outage, 34 min85% of window budget[@director]
[EX-NN]YYYY-MM-DD[Description][Impact][@name]

7. Reporting and review

ActivityFrequencyOwnerOutput
Budget status reportWeekly[@sre-lead]Level, remaining budget, top consumers
Reliability reviewMonthly[@engineering-lead]Trends, actions, SLO fitness
Policy reviewAfter each Red level, and yearlyPolicy owner and product ownerUpdated policy version
SLO target reviewQuarterlyProduct and engineeringConfirm targets still match user needs

Signs that the SLO or policy needs revision: the budget is never used (target may be too loose or the team too cautious), the budget is exhausted every window (target too tight or the system needs investment), or users complain while the SLO is met (the SLI measures the wrong thing).

8. Sign-off

By signing, the parties agree to apply this policy as written until it is revised.

RoleNameSignature / approvalDate
Product owner[@name][Approved]YYYY-MM-DD
Engineering lead[@name][Approved]YYYY-MM-DD
SRE lead[@name][Approved]YYYY-MM-DD
Director (escalation owner)[@name][Approved]YYYY-MM-DD

Standard and licence note

Based on common site reliability engineering practice for service level objectives, error budgets and burn-rate alerting. This template describes the concepts in original text and reproduces no published policy. Template licence: MIT.

関連テンプレート