# Alerting Policy: [Organisation name]

> Organisation-wide policy for production alerting: symptom-based
> paging, the choice between page, ticket and log, SLO burn-rate
> alerting as the default for paging (multiple windows and multiple
> burn rates, with worked numbers), required alert metadata, naming,
> routing, the noise budget, alert review and retirement, and alerts
> managed as code. Attribute names follow the OpenTelemetry semantic
> conventions 1.44.0. The multi-window, multi-burn-rate approach comes
> from Google's The Site Reliability Workbook, chapter "Alerting on
> SLOs", cited by name as its source. The default parameters match the
> values commonly published for this method, including in the Workbook;
> here they are derived step by step from the error-budget formula, and
> the explanation and worked examples were written and recomputed for
> this template. Replace every `[bracketed]` value and every row marked
> *Example*. Worked examples use Corvid Commerce, a fictional SaaS
> commerce platform.

<!--
Guidance: How to use this template
- This policy decides what may interrupt a human. The Observability
  Specification (`ops/observability-spec`) decides what telemetry
  exists; the SLO Document (`dev/slo-doc`) decides the targets; the
  Incident Severity Matrix (`ops/incident-severity-matrix`) decides
  how serious an incident is once someone responds.
- Start with sections 3 and 4. Agree that pages are for urgent,
  user-visible symptoms before arguing about thresholds.
- Fill the burn-rate table in section 5 from your own SLOs and
  recompute every number; the worked values assume a 99.9% SLO over
  30 days.
- The main pitfall is keeping every alert someone once found useful.
  An alert that pages and needs no action costs sleep and teaches
  people to ignore pages.
-->

## Document control

<!-- Guidance: The owner is usually the SRE or observability lead. Approval by engineering leadership matters because the policy constrains every team. -->

| Field | Value |
|---|---|
| Document ID | [ALR-POL-001] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@observability-lead] |
| Approvers/reviewers | [@head-of-reliability], [@vp-engineering], [@ciso], [@on-call-representatives] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Applies to | [All alerts that notify a human about production services] |
| Alert rules repository | [Repository URL] |
| Paging tool | [Paging or incident tool name] |
| Effective date | YYYY-MM-DD |
| Next review | YYYY-MM-DD (at least annually) |
| Related documents | [Severity standard], [SLO documents], [Observability spec], [On-call handbook] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Head of reliability | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| VP engineering | [@name] | [Decision] | YYYY-MM-DD |
| Chief information security officer (security alert scope) | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved; burn-rate paging mandatory for tier 1 services |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Noise budget lowered after quarterly review |

## Contents

<!-- Guidance: Keep this list aligned with the headings. -->

1. [Purpose and scope](#1-purpose-and-scope)
2. [Roles and responsibilities](#2-roles-and-responsibilities)
3. [Alerting principles](#3-alerting-principles)
4. [Page, ticket or log](#4-page-ticket-or-log)
5. [SLO burn-rate alerting](#5-slo-burn-rate-alerting)
6. [Required alert metadata and naming](#6-required-alert-metadata-and-naming)
7. [Routing and escalation](#7-routing-and-escalation)
8. [Noise budget and alert quality metrics](#8-noise-budget-and-alert-quality-metrics)
9. [Alert review and retirement](#9-alert-review-and-retirement)
10. [Alerts as code](#10-alerts-as-code)
11. [Policy statements](#11-policy-statements)
12. [Exceptions, compliance measurement and review](#12-exceptions-compliance-measurement-and-review)
13. [Glossary and references](#13-glossary-and-references)
14. [Standard and licence note](#standard-and-licence-note)

## 1. Purpose and scope

<!-- Guidance: State what problem the policy solves for this organisation, usually alert fatigue and missed incidents. Questions: which alert sources are in scope (metrics, logs, synthetics, security tools)? Pitfall: leaving vendor-default alerts outside the policy. -->

### 1.1 Purpose

Alerts are the main way production asks a human for help. This policy
makes sure that every page is urgent, actionable and tied to user
experience, that everything else reaches the right team without
waking anyone, and that the alert set stays small and trustworthy as
systems change.

### 1.2 Scope

- **In scope:** every rule that notifies a person or creates work for
  a team about a production service, whatever the source: metrics,
  logs, traces, synthetic checks, cloud provider health events,
  third-party status feeds, and default rules shipped by monitoring
  vendors.
- **Security detections:** security alerts follow the routing,
  metadata and review rules here; their logic and triage follow the
  Detection Rule Specification (`security/detection-rule-spec`) and
  the SOC Playbook (`security/soc-playbook`).
- **Out of scope:** dashboards without notifications, ad hoc queries,
  and non-production environments (unless a team chooses to apply
  the policy there).

### 1.3 Service tiers

Requirements in this policy depend on the service tier recorded in the
service catalogue.

| Tier | Meaning | Paging requirement |
|---|---|---|
| Tier 1 | Core customer journeys (*Example:* checkout-api, payments-gateway, storefront-web, identity-service) | 24-hour paging; burn-rate alerts mandatory |
| Tier 2 | Important but not core (*Example:* catalogue-search, webhook-delivery, merchant-dashboard) | 24-hour or business-hours paging per the team's handbook; burn-rate alerts mandatory |
| Tier 3 | Internal or batch (*Example:* analytics-export) | Tickets only unless an exception is approved |

## 2. Roles and responsibilities

<!-- Guidance: Every alert needs one owning team. Pitfall: alerts owned by "platform" for services platform does not run. -->

| Role | Responsibilities |
|---|---|
| Policy owner ([@observability-lead]) | Maintains this policy, the shared alert library and the quarterly quality report; approves exceptions |
| Service owning team | Owns the alerts for its services, their runbooks and their thresholds; keeps metadata current |
| On-call engineer | Responds to pages; records for each page whether it was actionable; raises tickets for bad alerts |
| Team lead | Runs the weekly page review; keeps the team within the noise budget |
| SRE or platform team | Provides the alerting platform, burn-rate rule templates and alert linting in CI |
| Security operations | Owns security detections; applies this policy's routing and metadata rules |
| Reviewers of alert changes | Check every new or changed alert against section 10.2 before merge |

## 3. Alerting principles

<!-- Guidance: These principles are the test every alert must pass. Keep them few and quotable. -->

1. **Page on symptoms, not causes.** Page when users experience, or
   are about to experience, failure: errors, latency, unavailability,
   incorrect results, data freshness. A high CPU reading, a full queue
   or a restarted pod is a cause; it pages only when it reliably
   predicts user impact soon and nothing better exists.
2. **Every page needs a human now.** If the right response can wait
   until business hours, it is a ticket. If no response is needed, it
   is not an alert.
3. **Every page is actionable.** The responder can do something useful,
   and the alert links to a runbook that says what.
4. **Alert on the SLO.** For services with SLOs, the default paging
   alert is an error-budget burn-rate alert (section 5), because it
   measures what users feel and scales urgency to the rate of damage.
5. **Keep the cause alerts for diagnosis.** Cause-based signals belong
   on dashboards and in tickets, where they help the responder explain
   a symptom page.
6. **Fewer, better alerts.** Each alert has a cost in attention. New
   alerts must justify themselves; old alerts must keep justifying
   themselves (section 9).
7. **Alerts are code.** Alert rules are versioned, reviewed and tested
   like any other production change (section 10).

## 4. Page, ticket or log

<!-- Guidance: Give responders and authors one decision rule. Questions: what is urgent enough to interrupt sleep? Pitfall: "warning" pages that are really tickets with a pager attached. -->

### 4.1 Notification classes

| Class | Meaning | Delivery | Response expectation *(Example)* |
|---|---|---|---|
| Page | Urgent and user-affecting now or within hours | Paging tool, high urgency, any hour for 24-hour rotations | Acknowledge within 5 min at tier 1, 15 min at tier 2 |
| Ticket | Needs action but can wait for business hours | Team queue with due date | Triage next business day; fix within the ticket's priority SLA |
| Log or dashboard | Useful context; no action required by itself | Dashboard panel, event stream, chat channel with no mention | None |

A page is not automatically an incident. The responder classifies an
incident using the Incident Severity Matrix
(`ops/incident-severity-matrix`) when impact is confirmed. As a default
*(Example)*, a fast-burn page on a tier 1 service is assessed as at
least SEV2 until shown otherwise.

### 4.2 Decision flow

Authors use this flow when proposing an alert; reviewers use it to
challenge the class chosen.

```mermaid
flowchart TD
  A["Proposed alert"] --> B{"Does it indicate user impact now or soon?"}
  B -- "No" --> C{"Does someone need to act eventually?"}
  C -- "Yes" --> T["Ticket"]
  C -- "No" --> L["Log or dashboard only"]
  B -- "Yes" --> D{"Will waiting until business hours cause meaningful harm?"}
  D -- "No" --> T
  D -- "Yes" --> E{"Can the responder act on it, with a runbook?"}
  E -- "No" --> F["Fix the signal or write the runbook first"]
  E -- "Yes" --> P["Page"]
```

### 4.3 Examples

| ID | Signal | Class | Reason |
|---|---|---|---|
| ALR-EX-001 *(Example)* | checkout-api availability SLO burning budget 14.4 times faster than sustainable | Page | User-visible and fast; budget for a month gone in about 2 days |
| ALR-EX-002 *(Example)* | checkout-api availability SLO burning budget at 1 times the sustainable rate over 3 days | Ticket | Real but slow; can wait for business hours |
| ALR-EX-003 *(Example)* | Disk on a database node 80% full, growing 1% per day | Ticket | 20 days of runway |
| ALR-EX-004 *(Example)* | Disk on a database node will be full within 4 hours at current growth | Page | Predicts imminent write failure for users |
| ALR-EX-005 *(Example)* | One pod restarted | Log | Self-healing; restart counts on the dashboard |
| ALR-EX-006 *(Example)* | TLS certificate for the storefront expires in 14 days | Ticket | Action needed, not urgent; page only at 48 hours if still not renewed |
| ALR-EX-007 *(Example)* | Synthetic checkout journey failing from 3 of 5 probe locations for 5 min | Page | Direct symptom; covers low-traffic hours where SLIs are thin |

## 5. SLO burn-rate alerting

<!-- Guidance: This is the default way to page for any service with an SLO. Recompute every number for your SLO target and window. Pitfall: alerting when the error rate crosses the SLO threshold, which either pages for every brief blip or detects slow damage far too late. -->

### 5.1 Concepts

- **Error budget:** the amount of unreliability the SLO allows over its
  window. For an SLO of 99.9% of good requests over 30 days, the
  budget is 0.1% of the requests in that window. Measured as time, a
  complete outage could last 0.1% of 30 days: 43,200 minutes times
  0.001 = 43.2 minutes.
- **Burn rate:** how fast the budget is being spent compared with the
  rate that would spend exactly all of it by the end of the window.
  Burn rate = observed error ratio / (1 - SLO target). A burn rate of
  1 spends the whole budget in exactly 30 days; a burn rate of 10
  spends it in 3 days.
- **Time to exhaustion:** window length / burn rate. At burn rate 14.4,
  a 30-day (720-hour) budget lasts 720 / 14.4 = 50 hours.
- **Budget consumed in a lookback:** burn rate times lookback length /
  window length. At burn rate 14.4 over 1 hour, 14.4 x 1 / 720 = 2% of
  the monthly budget is consumed.

### 5.2 Why one window and one threshold are not enough

A single alert such as "error ratio above 0.1% for 5 minutes" fails
in two directions. It pages on short spikes that cost almost no budget,
and it says nothing about a slow leak that stays just under the
threshold all month. Choosing instead a long window catches slow leaks
but keeps firing for a long time after the problem is fixed, because
the bad minutes stay inside the window.

The approach below solves both problems:

1. **Several burn rates.** A high burn rate over a short-to-medium
   window catches fast damage quickly and pages. A low burn rate over a
   long window catches slow damage and opens a ticket.
2. **Two windows per rule.** Each rule requires the threshold to be
   exceeded over a long window (proof that significant budget was
   spent) and over a short window, one twelfth of the long one (proof
   that the problem is still happening). When the problem stops, the
   short window drops below the threshold within minutes and the alert
   resolves, even though the long window still contains the bad period.

### 5.3 Default rule set

The thresholds are not chosen directly. They follow from one decision
per rule, how much of the budget may be spent before someone is told,
and from the error-budget formula in section 5.1. With:

- $W$ the SLO window (720 hours for 30 days),
- $L$ the long lookback window of the rule,
- $c$ the share of the budget the team accepts to lose before being
  told,
- $b$ the burn rate,

the budget consumed over the lookback is $c = b \cdot L / W$, so the
burn-rate threshold is

$$
b = \frac{c \cdot W}{L}
$$

and the error-ratio threshold the rule compares against is
$b \cdot (1 - \text{SLO})$. The short window is set to one twelfth of
the long window ($L / 12$) so the alert resets quickly once the
problem stops. The page rules use a budget share that would empty the
budget within days; the ticket rule uses a share that empties it no
faster than the window itself.

All values are *Example* defaults for a 30-day window and a 99.9% SLO;
recompute for yours.

| Rule ID | Budget consumed before alerting | Long window | Short window | Burn rate threshold | Error ratio threshold at 99.9% SLO | Class |
|---|---|---|---|---|---|---|
| BRN-FAST *(Example)* | 2% | 1 h | 5 min | 0.02 x 720 / 1 = 14.4 | 14.4 x 0.1% = 1.44% | Page |
| BRN-MED *(Example)* | 5% | 6 h | 30 min | 0.05 x 720 / 6 = 6 | 6 x 0.1% = 0.6% | Page |
| BRN-SLOW *(Example)* | 10% | 3 days (72 h) | 6 h | 0.10 x 720 / 72 = 1 | 1 x 0.1% = 0.1% | Ticket |

Time to exhaust the whole budget at each threshold: 720 / 14.4 = 50
hours; 720 / 6 = 120 hours (5 days); 720 / 1 = 720 hours (30 days).

The rule fires when both conditions are true:

```text
error_ratio(long window)  > burn_rate_threshold * (1 - SLO)
AND
error_ratio(short window) > burn_rate_threshold * (1 - SLO)
```

### 5.4 Worked detection examples

The following *Example* cases use the BRN-FAST rule on checkout-api
(SLO 99.9% over 30 days, threshold 1.44% error ratio).

Derivation: if the error ratio jumps from zero to a constant $e$, the
average over the long window after $t$ minutes is $e \cdot t / L$.
Setting it equal to the threshold gives the time to detect,
$t = (1.44\% / e) \cdot 60$ minutes for BRN-FAST. The budget consumed
at that moment is $(e / 0.1\%) \cdot t / W$ (with $t$ and $W$ in the
same unit), which simplifies to
$b \cdot L / W = c$: every failure is detected when about 2% of the
budget is gone, whatever its size. The 5-minute short window crosses
1.44% earlier in every case below, so the long window decides.

| Case | Actual error ratio | Burn rate | Time until the 1-hour average reaches 1.44% | Budget consumed at that moment | Result |
|---|---|---|---|---|---|
| Total outage | 100% | 1,000 | 1.44 / 100 x 60 min = 0.864 min, about 52 s | 1,000 x (0.864 / 60) / 720 = 2.0% | Page in under a minute |
| Serious partial failure | 18% | 180 | 1.44 / 18 x 60 min = 4.8 min | 180 x (4.8 / 60) / 720 = 2.0% | Page in about 5 min (the checkout incident in the Postmortem template, `dev/postmortem`) |
| Serious partial failure | 10% | 100 | 1.44 / 10 x 60 min = 8.64 min | 100 x (8.64 / 60) / 720 = 2.0% | Page in about 9 min |
| Moderate failure | 2% | 20 | 1.44 / 2 x 60 min = 43.2 min | 20 x (43.2 / 60) / 720 = 2.0% | Page in about 43 min; BRN-MED (0.6%) would fire after 6 h x 0.6 / 2 = 1.8 h, so BRN-FAST is first |
| Slow leak | 0.3% | 3 | Never (0.3% stays under 1.44%) | Not applicable | BRN-FAST and BRN-MED stay quiet; BRN-SLOW (0.1%) fires after 72 h x 0.1 / 0.3 = 24 h and opens a ticket, by which time 3 x 24 / 720 = 10% of budget is spent |

Each detection time assumes the error starts abruptly from zero and the
window was clean before. The pattern to notice: the alert fires when
about the chosen share of budget has been spent, whatever the severity
of the failure, and faster failures are detected faster.

Reset time: once checkout-api recovers, the 5-minute short window falls
below 1.44% within about 5 minutes, so BRN-FAST resolves even though
the 1-hour window still contains the failure.

### 5.5 Special cases

- **Low traffic.** With few requests, a handful of errors creates a
  huge error ratio. Require a minimum number of events in the long
  window (*Example:* at least [100] requests), add synthetic traffic,
  or combine low-traffic services into one SLO.
- **Latency SLOs.** Treat each request slower than the SLO threshold as
  a bad event and apply the same burn-rate rules to the ratio of slow
  requests.
- **Freshness and correctness SLOs.** Express them as good and bad
  events (*Example:* a minute in which data is older than 10 minutes is
  bad) and reuse the rule set.
- **Dependencies.** Alert on your own SLO. A dependency's outage pages
  its owners; your page fires only if your users are affected.
- **Release safety.** Burn-rate pages during a rollout should stop the
  rollout automatically where the delivery pipeline supports it.
- **Budget policy.** What the team does when the budget runs low
  (release freeze, reliability work) belongs in the Error Budget Policy
  (`ops/error-budget-policy`), not in alert rules.

## 6. Required alert metadata and naming

<!-- Guidance: Metadata makes an alert usable at 03:00. Questions: can the responder tell from the notification alone what is broken, who owns it and where to start? Pitfall: a runbook link that points to a page that no longer exists. -->

### 6.1 Required fields

Every alert, page or ticket, MUST carry the fields below. Where the
field describes a resource, use the OpenTelemetry semantic convention
attribute name so alerts, traces and logs can be joined.

| Field | Required for | Source or format | Example |
|---|---|---|---|
| Alert name | All | Naming rule in 6.2 | `CheckoutApiAvailabilityFastBurn` |
| Service | All | OpenTelemetry `service.name` | `checkout-api` |
| Environment | All | OpenTelemetry `deployment.environment.name` | `production` |
| Owning team | All | Service catalogue team identifier | `team-checkout` |
| Class | All | `page`, `ticket` or `log` | `page` |
| Service tier | All | Tier from section 1.3 | `tier-1` |
| Summary | All | One line, impact first, with the measured value | "Checkout availability burning error budget at 19x in eu-west (1.9% errors, page threshold 14.4x)" |
| Runbook URL | Page and ticket | Link to the Runbook (`dev/runbook`) section for this alert | `[runbook URL]#fast-burn` |
| Dashboard URL | Page and ticket | Link pre-filtered to service and time | `[dashboard URL]` |
| SLO reference | Burn-rate alerts | Link to the SLO Document (`dev/slo-doc`) | `[SLO document URL]` |
| Region or scope | When applicable | Cloud region, cell or tenant group | `eu-west` |
| Rule source | All | Path in the alert rules repository | `alerts/checkout/availability.yaml` |

### 6.2 Naming

- Format: `<Service><Signal><Condition>` in UpperCamelCase *(Example
  convention)*, where Signal is the SLI or symptom (Availability,
  Latency, Freshness, Correctness, Saturation) and Condition is the
  trigger (FastBurn, MediumBurn, SlowBurn, SyntheticFailure,
  WillFillIn4h).
- Names describe the symptom, not the suspected cause: prefer
  `CheckoutApiLatencyFastBurn` to `CheckoutDbSlowQueries`.
- Never reuse a retired alert's name for a different condition.

### 6.3 Notification text

The notification body states impact first, then scope, then the
measured value against the threshold, then the links. The responder
must be able to decide whether to get out of bed from the first line.

```text
[PAGE] CheckoutApiAvailabilityFastBurn | checkout-api | production | eu-west
Impact: checkout requests failing, 1.9% errors (threshold 1.44%), 2.6% of monthly budget spent in 1 h
Runbook: [runbook URL]#fast-burn   Dashboard: [dashboard URL]   SLO: [SLO URL]
Owner: team-checkout   Tier: tier-1
```

## 7. Routing and escalation

<!-- Guidance: Route by owning team, never to a shared catch-all. Questions: what happens if nobody acknowledges? Pitfall: one page delivered to five people, each assuming another will act. -->

| Rule | Requirement |
|---|---|
| Page routing | To the owning team's primary on-call via its escalation policy in the paging tool |
| Escalation | Unacknowledged pages escalate to secondary on-call after [5] min at tier 1 or [15] min at tier 2, then to the team lead after a further [15] min *(Example)* |
| Ticket routing | To the owning team's queue with the service label; never to a personal inbox |
| Log routing | Dashboard or event stream; chat channels only without mentions |
| Deduplication | Alerts with the same name, service and scope group into one notification |
| Dependencies | When a shared dependency is down, suppress downstream symptom pages only if the dependency page has been acknowledged, and record the suppression |
| Maintenance | Silences during planned maintenance are created through the maintenance process, time-boxed, and announced per the Maintenance Notice (`ops/maintenance-notice`) |
| Security detections | To security operations on-call; availability impact also notifies the owning team |
| Incident escalation | The responder declares an incident per the Incident Severity Matrix; the IC process follows the Incident Commander Handbook (`ops/incident-commander-handbook`) |

## 8. Noise budget and alert quality metrics

<!-- Guidance: A noise budget is a team agreement on how much interruption is tolerable. Set it from on-call health, not from current volume. Pitfall: measuring only pages per week and ignoring whether they were actionable. -->

### 8.1 Noise budget

Each team agrees a noise budget in its on-call handbook. The *Example*
default for Corvid Commerce:

- no more than [2] pages per on-call shift on average across a week;
- no more than [1] out-of-hours page per night on average across a
  week;
- at least [80%] of pages actionable.

When a team exceeds its noise budget for [2] consecutive weeks, the
team lead MUST schedule alert clean-up work ahead of feature work
until the team is back within budget, and record the actions in the
weekly page review. Workload from pages is also tracked in the Toil
Review (`ops/toil-review`).

### 8.2 Metrics

| Metric | Formula | Target *(Example)* |
|---|---|---|
| Pages per shift | pages / on-call shifts in the period | At most 2 |
| Out-of-hours pages | pages between [22:00 and 07:00] local time / nights in the period | At most 1 |
| Actionable rate | pages marked actionable / all pages | At least 80% |
| Duplicate rate | pages grouped into another page or incident / all pages | Under 10% |
| Flapping rate | alerts that fired and resolved more than [3] times in 1 h / all alerts that fired | Under 5% |
| Runbook coverage | page-class alerts with a working runbook link / all page-class alerts | 100% |
| Missed detection | incidents found by customers or support before any alert / all SEV1 and SEV2 incidents | Under 10% |
| Mean time to acknowledge | mean of (ack time minus page time) | Tier 1 under 5 min |

### 8.3 Worked example

*Example* data for team-checkout over one week of 14 on-call shifts of
12 hours each:

| Measure | Value |
|---|---|
| Pages | 24 |
| Pages marked actionable | 17 |
| Out-of-hours pages | 9 over 7 nights |
| Duplicates | 3 |

Pages per shift = 24 / 14 = 1.71, within the budget of 2. Actionable
rate = 17 / 24 = 70.8%, below the 80% target. Out-of-hours pages =
9 / 7 = 1.29 per night, above the budget of 1. Duplicate rate =
3 / 24 = 12.5%, above 10%. The team is outside its targets on three
measures (actionable rate, out-of-hours pages and duplicate rate); the weekly review identifies the 7 non-actionable pages and
the 3 duplicates as the first clean-up candidates.

## 9. Alert review and retirement

<!-- Guidance: Alerts rot as systems change. Review them on a fixed rhythm and make retirement normal. Pitfall: never deleting alerts because "it might catch something one day". -->

### 9.1 Review rhythm

| Review | Cadence | Participants | Inputs | Outputs |
|---|---|---|---|---|
| Page review | Weekly | Outgoing and incoming on-call, team lead | Every page of the week with its actionable flag | Tickets to fix, tune or retire alerts |
| Alert inventory review | Quarterly | Team, SRE representative | Full list of the team's alerts with fire counts and metrics from 8.2 | Retire, merge, reclassify or keep decisions |
| Policy quality report | Quarterly | Policy owner, team leads | Organisation-wide metrics | Teams over budget, policy changes |
| Post-incident check | After every SEV1 and SEV2 | Postmortem owner | Postmortem (`dev/postmortem`) | Missing or late alerts added as actions |

### 9.2 Retirement criteria

Retire, merge or reclassify an alert when any of the following holds:

- it paged and was marked non-actionable in [3] or more of its last
  [5] firings;
- another alert always fires with it and covers the same symptom;
- its owner, service or runbook no longer exists;
- it is cause-based and a burn-rate alert now covers the same impact;
- the team cannot say what a responder should do when it fires.

An alert that has not fired for a long time is **not** retired for that
reason alone; check instead that it still evaluates correctly (section
10.3) and still matches a real failure mode.

### 9.3 Alert lifecycle

Each alert moves through Proposed, Testing, Active, Tuning and Retired.
A new page-class alert enters Testing when merged with test cases and
runs as a ticket for [1 to 2] weeks *(Example)*, so the team can see how
often it would have paged, before it becomes Active. An Active alert that
fails the quality criteria moves to Tuning; one that cannot be fixed, or
meets the retirement criteria, moves to Retired.

## 10. Alerts as code

<!-- Guidance: Treat alert rules like application code: versioned, reviewed, tested and deployed through a pipeline. Pitfall: rules edited by hand in a web console, so nobody knows why a threshold changed. -->

### 10.1 Repository and pipeline

- All alert rules, routing rules and silences longer than [24] hours
  live in the alert rules repository and are deployed by pipeline.
- Changes made in a console during an incident MUST be committed to the
  repository within [1] business day or reverted.
- The pipeline validates syntax, required metadata (section 6.1),
  naming (section 6.2) and runbook link reachability before deploy.

### 10.2 Review checklist for a new or changed alert

| Check | Question for the reviewer |
|---|---|
| Symptom | Does it detect user impact, or is the cause-based exception justified? |
| Class | Is page, ticket or log correct under section 4? |
| SLO | For an SLO-backed service, is it a burn-rate rule from section 5, with recomputed thresholds? |
| Actionability | Does the runbook say what to do in the first 10 minutes? |
| Metadata | Are all required fields in section 6.1 present and valid? |
| Noise | What would its fire count have been over the last [30] days, replayed against history? |
| Tests | Are there test cases for firing and not firing? |
| Ownership | Is the owning team the one that will be paged? |

### 10.3 Testing

- Write unit tests for rule logic where the tooling supports it: given a
  synthetic time series, the alert fires (or does not) at the expected
  time. For burn-rate rules, test the total outage case and the slow
  leak case from section 5.4.
- Replay new rules against at least [30] days of historical data before
  enabling paging.
- Exercise paging end to end at least quarterly, for example during a
  Game Day (`ops/game-day-plan`), to prove that routing and escalation
  work.

## 11. Policy statements

<!-- Guidance: Number statements so exceptions and audits can reference them. MUST is mandatory; SHOULD needs a reason to deviate. -->

| ID | Statement |
|---|---|
| ALR-POL-001 | Pages MUST indicate current or imminent user impact and require action by a human before business hours. |
| ALR-POL-002 | Tier 1 and tier 2 services with an SLO MUST page on error-budget burn rate using at least a fast and a medium rule with long and short windows, and SHOULD open tickets on a slow-burn rule. |
| ALR-POL-003 | Cause-based pages SHOULD NOT exist; any that remain MUST record the justification in the rule and be reviewed quarterly. |
| ALR-POL-004 | Every alert MUST carry the required metadata in section 6.1; page-class alerts MUST link to a working runbook. |
| ALR-POL-005 | Pages MUST route to the owning team's on-call through an escalation policy; catch-all routes MUST NOT receive pages. |
| ALR-POL-006 | Alert rules and routing MUST be managed as code and reviewed before deployment. |
| ALR-POL-007 | Each team MUST record whether each page was actionable and review pages weekly. |
| ALR-POL-008 | A team over its noise budget for 2 consecutive weeks MUST prioritise alert clean-up. |
| ALR-POL-009 | New page-class alerts SHOULD run as tickets for a trial period before paging. |
| ALR-POL-010 | Silences MUST be time-boxed and linked to a maintenance or incident record. |

## 12. Exceptions, compliance measurement and review

<!-- Guidance: Exceptions are fine when they are visible, owned and temporary. Pitfall: permanent exceptions for the noisiest team. -->

### 12.1 Exceptions

Request an exception from the policy owner with the statement ID, the
reason, the compensating control, the risk owner and an expiry date of
at most [6] months.

| ID | Statement | Service | Reason | Compensating control | Risk owner | Expiry |
|---|---|---|---|---|---|---|
| ALR-EXC-001 *(Example)* | ALR-POL-002 | webhook-delivery | SLO not yet defined | Synthetic delivery check pages; SLO due next quarter | [@team-lead] | YYYY-MM-DD |

### 12.2 Compliance measurement

| Measure | Formula | Target *(Example)* |
|---|---|---|
| Burn-rate coverage | tier 1 and 2 SLOs with fast and medium burn-rate pages / all tier 1 and 2 SLOs | 100% |
| Metadata compliance | alerts passing metadata lint / all alerts | 100% |
| Alerts as code | alerts defined in the repository / all alerts found in the paging and monitoring tools | 100% |
| Teams within noise budget | teams within budget for the quarter / teams with on-call | At least 80% |
| Open expired exceptions | count | 0 |

### 12.3 Review

Review this policy at least annually, after any SEV1 where alerting was
a contributing factor, and when the SLO approach or tooling changes.
Changes that affect paging update the On-call Handbook
(`ops/on-call-handbook`) and the Service Readiness review
(`dev/service-readiness`) in the same change.

## 13. Glossary and references

<!-- Guidance: Keep definitions consistent with the SLO document and the observability specification. -->

### 13.1 Glossary

| Term | Meaning |
|---|---|
| Page | A high-urgency notification that requires a human response now |
| Ticket | A work item created by an alert that can wait for business hours |
| Symptom | A user-visible effect such as errors, latency or stale data |
| SLI | Service level indicator: the measured ratio of good events to total events |
| SLO | Service level objective: the target for an SLI over a window |
| Error budget | 1 minus the SLO target, applied to the window |
| Burn rate | Observed error ratio divided by the error budget ratio |
| Noise budget | The agreed maximum interruption load for an on-call rotation |
| Actionable page | A page after which the responder took a meaningful action |

### 13.2 Related templates

- SLO Document (`dev/slo-doc`) and Error Budget Policy (`ops/error-budget-policy`)
- Observability Specification (`ops/observability-spec`)
- Incident Severity Matrix (`ops/incident-severity-matrix`)
- Incident Commander Handbook (`ops/incident-commander-handbook`)
- On-call Handbook (`ops/on-call-handbook`) and Runbook (`dev/runbook`)
- Service Readiness (`dev/service-readiness`)
- Detection Rule Specification (`security/detection-rule-spec`)
- Toil Review (`ops/toil-review`)

### 13.3 References

| Reference | Version or date checked | Use |
|---|---|---|
| Google, The Site Reliability Workbook (O'Reilly, 2018), chapter "Alerting on SLOs", sre.google/workbook/alerting-on-slos | Online edition, checked 2026-10-24 | Source of the multi-window, multi-burn-rate approach; the default parameters match the values commonly published for this method, including in the Workbook; the explanation and worked examples are original and were recomputed for this template |
| Google, Site Reliability Engineering (O'Reilly), chapter "Monitoring Distributed Systems", sre.google | 1st edition 2016 (online); 2nd edition announced October 2026 | Further reading on symptom-based alerting; no text reproduced |
| OpenTelemetry semantic conventions, opentelemetry.io/docs/specs/semconv | 1.44.0, checked 2026-10-24 | Attribute names `service.name`, `deployment.environment.name` |

## Standard and licence note

Based on the OpenTelemetry semantic conventions 1.44.0
(opentelemetry.io, specification under Apache-2.0, documentation under
CC BY 4.0) for resource attribute names, attributed to the
OpenTelemetry Authors. The multi-window, multi-burn-rate method is
described in Google's The Site Reliability Workbook, chapter "Alerting
on SLOs", which is cited by name as the source of the approach; no text
or table from it is reproduced. The default parameters match the values
commonly published for this method, including in the Workbook; the
derivation, explanation and worked examples in this template are
original and were recomputed for it. OpenTelemetry is a trademark of its owner;
this template is not affiliated with or endorsed by it. Corvid Commerce
is fictional. This template is original text written for this
collection. Check the current official versions of the sources before
use. Template licence: MIT.