On-call Handbook: [Team or service name]

Handbook for engineers on an on-call rotation: how the rotation and handover work, what is expected of the on-call engineer, how to classify and escalate incidents, what to do in the first 15 minutes, how to communicate, and how the team looks after the people on call. Based on common site reliability engineering practice. Replace every [bracketed] value and every row marked Example.

1. Rotation and handover

1.1 Rotation

Eligibility: engineers join the rotation after completing the onboarding in section 10 and shadowing [two] shifts.

1.2 Shift swaps and cover

• Arrange swaps directly with a colleague and update the paging tool before the shift starts.
• If you are unwell or unavailable at short notice, tell the secondary and the on-call lead; the secondary becomes primary until cover is found.
• Never leave a shift uncovered. The on-call lead is the final fallback.

1.3 Handover

At each shift change the outgoing engineer completes a handover note in [channel or document] and talks it through with the incoming engineer for [15 minutes].

2. Responsibilities

The primary on-call engineer:

• acknowledges pages within the targets in section 4;
• triages, mitigates and, if needed, escalates incidents;
• acts as incident commander for SEV3 and SEV4 incidents and until a dedicated commander takes over for SEV1 and SEV2;
• keeps the incident channel and status updated;
• records follow-up work and completes the handover.

The secondary on-call engineer backs up the primary if a page is not acknowledged, joins major incidents and takes over when the primary needs rest.

On-call is not responsible for: feature work during the shift ([agreed reduced load, for example 50%]), answering general questions outside the support channel, or fixing root causes alone; those become tickets for the team.

3. Severity matrix

When unsure, choose the higher severity. Downgrading later is cheap; under-reacting is not.

4. Response expectations

While on call you must be able to reach a laptop with network access within [10 minutes] of a page, be fit to work, and keep your phone charged with notifications enabled. Mitigation, not root cause, is the first goal: roll back, fail over or shed load before debugging.

5. Escalation paths

Escalate when: you are not making progress after [20 minutes], the severity increases, more than one service is involved, or you need a decision you are not authorised to make.

6. Tools and access

7. The first 15 minutes

1. Acknowledge the page so escalation stops and others know someone is on it.
2. Assess impact: check the service overview dashboard. Who is affected, how many, since when?
3. Declare an incident if impact is real: open a channel named inc-[date]-[short-name] and set the severity.
4. Assign roles: for SEV1 and SEV2 page the incident commander; you become operations lead.
5. Check recent changes: deploys, config changes, feature flags and provider status in the last [2 hours].
6. Mitigate: if a recent change correlates, roll it back. Otherwise follow the alert's runbook.
7. Communicate: post the first update using template 8.1 within the target in section 4.
8. Record: note times and actions in the incident channel as you go; they become the timeline.

8. Communication templates

8.1 Internal incident update

[SEV level] [Short title] | Status: [Investigating / Identified / Mitigated / Resolved] Impact: [who and what is affected] Current action: [what we are doing now] Next update: [hh:mm timezone] IC: @[name] | Ops: @[name]

8.2 Customer status page update

We are investigating [plain-language symptom, for example "failed checkouts"] affecting [scope]. Our team is working on a fix. Next update by [hh:mm timezone].

8.3 Resolution notice

[Service] has been operating normally since [hh:mm timezone]. The issue was caused by [short cause]. We will publish a summary within [n days].

9. Wellbeing and compensation

Signals that the rotation is unhealthy, reviewed monthly by the on-call lead: more than [2] pages per night on average, repeated pages for the same cause, or shifts with unacknowledged pages.

10. Onboarding to the rotation

• Read this handbook and the incident response plan.
• Confirm access to every tool in section 6.
• Shadow [two] shifts with an experienced primary.
• Complete a reverse shadow (you are primary, mentor watches).
• Run one game day or tabletop exercise.
• Added to the schedule by the on-call lead.

11. Post-incident duties

12. Review of this handbook

This handbook is reviewed every [quarter] at the on-call retrospective and after any incident that exposed a gap. Propose changes by [process, for example "pull request to the team handbook repository"].

Based on common site reliability engineering practice for on-call rotations and incident response. Original text written for this collection. Template licence: MIT.