# On-call Handbook: [Team or service name]

> Handbook for engineers on an on-call rotation: how the rotation and
> handover work, what is expected of the on-call engineer, how to
> classify and escalate incidents, what to do in the first 15 minutes,
> how to communicate, and how the team looks after the people on call.
> Based on common site reliability engineering practice. Replace every
> `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Write for a new team member on their first shift. Link to tools and
  runbooks rather than assuming knowledge.
- Keep severity definitions identical to the incident response plan; if
  they differ, people will argue about severity during an incident.
- Fill in the wellbeing and compensation section with HR before the
  rotation starts. Unclear rules are a leading cause of on-call burnout.
-->

| Field | Value |
|---|---|
| Document ID | [OCH-TEAM-001] |
| Team / services | [Team name], [services covered] |
| Version | [0.1] |
| Status | [Draft / Approved] |
| Owner | [@on-call-lead] |
| Last reviewed | YYYY-MM-DD |
| Related | [Incident response plan], [Runbooks], [Observability spec], [Escalation policy in paging tool] |

## 1. Rotation and handover

<!-- Guidance: Describe the schedule so anyone can work out who is on call and when their next shift is. Include how to swap shifts. -->

### 1.1 Rotation

| Item | Value |
|---|---|
| Rotation type | *Example:* Weekly, primary and secondary |
| Shift change | *Example:* Monday 10:00 [time zone] |
| Coverage | *Example:* 24x7 for Tier 1 services; business hours for Tier 2 |
| Minimum team size | *Example:* 6 engineers per rotation, so each person is primary at most one week in six |
| Follow-the-sun handoffs | *Example:* 08:00 and 20:00 UTC between [Region A] and [Region B] |
| Schedule | [Link to paging tool schedule] |

Eligibility: engineers join the rotation after completing the onboarding
in section 10 and shadowing [two] shifts.

### 1.2 Shift swaps and cover

- Arrange swaps directly with a colleague and update the paging tool before the shift starts.
- If you are unwell or unavailable at short notice, tell the secondary and the on-call lead; the secondary becomes primary until cover is found.
- Never leave a shift uncovered. The on-call lead is the final fallback.

### 1.3 Handover

At each shift change the outgoing engineer completes a handover note in
[channel or document] and talks it through with the incoming engineer
for [15 minutes].

| Handover item | Content |
|---|---|
| Open incidents | ID, status, next action, owner |
| Ongoing risks | *Example:* Database migration running until Wednesday |
| Noisy or silenced alerts | Alert ID, reason, silence expiry |
| Planned changes this week | Deploys, maintenance windows, vendor work |
| Follow-ups | Tickets created during the shift |

## 2. Responsibilities

<!-- Guidance: Be explicit about what on-call does and does not include. "Fix everything" is not a job description. -->

The primary on-call engineer:

- acknowledges pages within the targets in section 4;
- triages, mitigates and, if needed, escalates incidents;
- acts as incident commander for SEV3 and SEV4 incidents and until a dedicated commander takes over for SEV1 and SEV2;
- keeps the incident channel and status updated;
- records follow-up work and completes the handover.

The secondary on-call engineer backs up the primary if a page is not
acknowledged, joins major incidents and takes over when the primary needs
rest.

On-call is **not** responsible for: feature work during the shift
([agreed reduced load, for example 50%]), answering general questions
outside the support channel, or fixing root causes alone; those become
tickets for the team.

## 3. Severity matrix

<!-- Guidance: Use the same levels as the organisation's incident response plan. Give concrete examples for this service so people can classify quickly. -->

| Severity | Definition | Examples for this service | Incident commander | Status page |
|---|---|---|---|---|
| SEV1 | Critical: core service down or data loss for many users | *Example:* Checkout failing for all users; data corruption | Dedicated IC from IC rotation | Yes, within 15 min |
| SEV2 | Major: significant degradation or a key feature unavailable | *Example:* p99 latency over 5 s; one region down | Dedicated IC | Yes, within 30 min |
| SEV3 | Minor: limited impact, workaround exists | *Example:* Export feature failing for some users | Primary on-call | Optional |
| SEV4 | Low: no current user impact | *Example:* One replica unhealthy; disk at 80% | Primary on-call | No |

When unsure, choose the higher severity. Downgrading later is cheap;
under-reacting is not.

## 4. Response expectations

<!-- Guidance: Set targets that are achievable for the rotation's size and coverage. Measure them and review monthly. -->

| Severity | Acknowledge within | First update within | Mitigate target | Update cadence |
|---|---|---|---|---|
| SEV1 | 5 min | 15 min | 1 h | Every 30 min |
| SEV2 | 15 min | 30 min | 4 h | Every 60 min |
| SEV3 | 1 h (working hours) | 4 h | 2 working days | Daily |
| SEV4 | Next working day | Next working day | Planned work | Weekly |

While on call you must be able to reach a laptop with network access
within [10 minutes] of a page, be fit to work, and keep your phone
charged with notifications enabled. Mitigation, not root cause, is the
first goal: roll back, fail over or shed load before debugging.

## 5. Escalation paths

<!-- Guidance: Escalating early is a skill to encourage, not a failure. List exactly who to call for what. -->

```mermaid
flowchart TD
  P["Page fires"] --> PR["Primary on-call"]
  PR -- "No ack in 5 min" --> SE["Secondary on-call"]
  SE -- "No ack in 10 min" --> L["On-call lead / manager"]
  PR -- "SEV1 or SEV2" --> IC["Incident commander rotation"]
  PR -- "Dependency failing" --> DT["Owning team on-call"]
  IC -- "Customer or legal impact" --> EX["Executive on duty"]
```

| Need | Contact | How |
|---|---|---|
| Incident commander | IC rotation | [Paging tool service name] |
| Database expertise | DBA on-call | [Paging tool service] |
| Security concern | Security on-call | [Paging tool service]; do not discuss in public channels |
| Cloud provider support | [Provider] | [Support portal, account ID] |
| Customer communications | Support lead | [Channel] |
| [Need] | [Contact] | [How] |

Escalate when: you are not making progress after [20 minutes], the
severity increases, more than one service is involved, or you need a
decision you are not authorised to make.

## 6. Tools and access

<!-- Guidance: Check that every new on-call engineer can log in to each tool before their first shift. -->

| Tool | Purpose | Link | Access check |
|---|---|---|---|
| [Paging tool] | Pages, schedules, escalation | [Link] | Test page received |
| [Chat] | Incident channels, status updates | [Link] | Can create channels |
| [Dashboards] | Service health | [Link] | Can view production |
| [Logs and traces] | Investigation | [Link] | Can query production |
| [Deploy tool] | Rollback | [Link] | Has rollback permission |
| [Status page] | Customer updates | [Link] | Can post incident |
| [Runbook index] | Procedures | [Link] | Bookmarked |

## 7. The first 15 minutes

<!-- Guidance: A short checklist reduces panic. Keep it in the order people should act. -->

1. **Acknowledge** the page so escalation stops and others know someone is on it.
2. **Assess impact**: check the service overview dashboard. Who is affected, how many, since when?
3. **Declare** an incident if impact is real: open a channel named `inc-[date]-[short-name]` and set the severity.
4. **Assign roles**: for SEV1 and SEV2 page the incident commander; you become operations lead.
5. **Check recent changes**: deploys, config changes, feature flags and provider status in the last [2 hours].
6. **Mitigate**: if a recent change correlates, roll it back. Otherwise follow the alert's runbook.
7. **Communicate**: post the first update using template 8.1 within the target in section 4.
8. **Record**: note times and actions in the incident channel as you go; they become the timeline.

## 8. Communication templates

<!-- Guidance: Plain language, facts only, next update time always included. -->

### 8.1 Internal incident update

> **[SEV level] [Short title]** | Status: [Investigating / Identified /
> Mitigated / Resolved]
> Impact: [who and what is affected]
> Current action: [what we are doing now]
> Next update: [hh:mm timezone]
> IC: @[name] | Ops: @[name]

### 8.2 Customer status page update

> We are investigating [plain-language symptom, for example "failed
> checkouts"] affecting [scope]. Our team is working on a fix. Next update
> by [hh:mm timezone].

### 8.3 Resolution notice

> [Service] has been operating normally since [hh:mm timezone]. The issue
> was caused by [short cause]. We will publish a summary within [n days].

## 9. Wellbeing and compensation

<!-- Guidance: Agree these rules with management and HR, and write them down. Placeholders below must be replaced with your organisation's policy. -->

| Topic | Policy |
|---|---|
| On-call allowance | [Amount or policy reference per shift or week] |
| Time off after night pages | [Example policy: start late the next day after any page between 00:00 and 06:00] |
| Overtime or time in lieu | [Policy reference] |
| Maximum consecutive shifts | [Example: no more than one primary week in four] |
| Hand-off when tired | Any engineer may hand an incident to the secondary after [n hours] of continuous work |
| Holidays and leave | [How public holidays are covered and paid] |
| Health and personal circumstances | [How to opt out temporarily, confidentially] |

Signals that the rotation is unhealthy, reviewed monthly by the on-call
lead: more than [2] pages per night on average, repeated pages for the
same cause, or shifts with unacknowledged pages.

## 10. Onboarding to the rotation

<!-- Guidance: List the steps that make someone ready for their first shift. -->

- [ ] Read this handbook and the incident response plan.
- [ ] Confirm access to every tool in section 6.
- [ ] Shadow [two] shifts with an experienced primary.
- [ ] Complete a reverse shadow (you are primary, mentor watches).
- [ ] Run one game day or tabletop exercise.
- [ ] Added to the schedule by the on-call lead.

## 11. Post-incident duties

<!-- Guidance: The shift does not end when the graph recovers. Make follow-up expectations explicit. -->

| Duty | When | Owner |
|---|---|---|
| Mark the incident resolved and post the resolution notice | At resolution | Incident commander |
| Write the incident timeline from the channel log | Within 1 working day | Operations lead |
| Schedule a blameless postmortem for SEV1 and SEV2 | Within 5 working days | Incident commander |
| Create tickets for follow-up actions with owners | During postmortem | Team |
| Update or create runbooks used during the incident | Within 5 working days | On-call engineer |
| Review alert quality: was it actionable, timely, correctly routed? | Weekly on-call review | On-call lead |

## 12. Review of this handbook

<!-- Guidance: Treat the handbook like code. Changes go through review. -->

This handbook is reviewed every [quarter] at the on-call retrospective
and after any incident that exposed a gap. Propose changes by [process,
for example "pull request to the team handbook repository"].

Based on common site reliability engineering practice for on-call
rotations and incident response. Original text written for this
collection. Template licence: MIT.