# Game Day Plan: [Game day name]

> Plan for a game day: a scheduled exercise in which a team practises
> detecting, responding to and recovering from realistic failures, then
> learns from the result. Covers objectives, scenarios, roles, schedule,
> safety controls, communication, scoring of detection and response, and
> the debrief. Based on common site reliability engineering practice.
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- A game day tests people, processes and tools together. Write objectives
  about response (detect, diagnose, communicate, recover), not only about
  whether the system survives.
- Keep scenario details secret from responders when you want to test
  detection, but share the date, scope and safety rules with everyone.
- Safety first: every injected fault needs a tested stop and someone with
  authority to call it off.
- The debrief is the point of the exercise. Book it before the game day and
  turn findings into tracked actions.
-->

| Field | Value |
|---|---|
| Game day ID | [GD-2026-05] |
| Title | [Regional failover and dependency outage] |
| Services in scope | [Service A], [Service B], [shared platform] |
| Date and time | YYYY-MM-DD, [09:30-13:00 UTC] |
| Environment | [Staging / Production with limited blast radius] |
| Facilitator | [@facilitator] |
| Approver | [@engineering-director or SRE lead] |
| Status | [Draft / Approved / Completed / Actions closed] |
| Related | [Incident response plan], [Runbooks], [Previous game day GD-2026-04], [Chaos experiment plans] |

## 1. Objectives

<!-- Guidance: Three to five measurable objectives. Each one should be checked during scoring in section 7. -->

| Objective ID | Objective | Success measure |
|---|---|---|
| OBJ-01 | *Example:* Validate that the primary on-call detects a database primary failure through alerts, not through customers | Page received within 5 min of injection |
| OBJ-02 | *Example:* Practise regional failover using the runbook only | Failover complete within 30 min (RTO) with no runbook gaps |
| OBJ-03 | *Example:* Exercise incident communication | Status page updated within 15 min; stakeholder update every 30 min |
| OBJ-04 | *Example:* Train two new on-call engineers as incident commander | Each leads one scenario with a shadow |
| [OBJ-NN] | [Objective] | [Measure] |

## 2. Scope and assumptions

<!-- Guidance: State what may be touched and what must not. List assumptions so surprises during the day can be traced back. -->

| In scope | Out of scope |
|---|---|
| *Example:* Checkout and payment services in [Region A] | *Example:* Customer-facing traffic in [Region B] |
| *Example:* Primary database cluster for checkout | *Example:* Data deletion or corruption scenarios |
| *Example:* Paging, chat and status page tooling | *Example:* Real external customer notifications |

Assumptions:

- [Assumption, for example: the DR runbook version 3.2 is the current approved version.]
- [Assumption, for example: production traffic is below 40% of peak during the window.]

## 3. Scenarios

<!-- Guidance: Base scenarios on real risks: recent incidents, FMEA rows, new architecture. Write the injection and the expected response; keep the inject sheet from responders if detection is being tested. -->

| Scenario ID | Scenario | Injection | Expected detection | Expected response | Linked risk |
|---|---|---|---|---|---|
| SC-01 | *Example:* Checkout database primary fails | Stop primary instance in [Region A] | Replica lag and error-rate alerts page primary on-call | Promote replica per runbook RB-DB-04 | FMEA FM-011 |
| SC-02 | *Example:* Payment provider slow | Add 2 s latency to provider calls via proxy | Latency SLO burn alert | Enable degraded mode flag; communicate | INC-2291 |
| SC-03 | *Example:* Expired TLS certificate on internal gateway | Deploy expired certificate to staging gateway | Synthetic check failure | Rotate certificate with runbook RB-SEC-02 | Audit finding A-17 |
| SC-04 | *Example:* Loss of observability (metrics backend down) | Block metrics ingestion for 15 min | Meta-monitoring alert | Switch to logs and fallback dashboard | New risk |
| [SC-NN] | [Scenario] | [Injection] | [Detection] | [Response] | [Risk] |

## 4. Roles and responsibilities

<!-- Guidance: Separate the people running the exercise from those responding. Observers record facts; they do not help responders unless the facilitator allows it. -->

| Role | Name | Responsibilities |
|---|---|---|
| Facilitator | [@name] | Runs the schedule, releases injects, decides to pause or stop, leads debrief |
| Injector / operator | [@name] | Executes fault injections and rollbacks on the facilitator's signal |
| Safety officer | [@name] | Watches abort criteria and real customer impact; can stop at any time |
| Responders | [@primary], [@secondary], [@incident-commander] | Respond as in a real incident using normal tools and runbooks |
| Observers / scribes | [@name], [@name] | Record timeline, decisions, gaps and quotes |
| Stakeholder liaison | [@name] | Keeps support, product and leadership informed |
| Real-incident on-call | [@name] | Not part of the exercise; handles any genuine incident |

## 5. Schedule

<!-- Guidance: Leave buffer between scenarios for recovery and notes. Plan to finish early rather than late. -->

| Time (UTC) | Activity | Lead |
|---|---|---|
| 09:30 | *Example:* Briefing: objectives, safety rules, code word for real incidents | Facilitator |
| 09:45 | *Example:* Baseline check of dashboards and steady state | Observers |
| 10:00 | *Example:* SC-01 inject | Injector |
| 10:45 | *Example:* SC-01 stop, recover, notes | Facilitator |
| 11:00 | *Example:* SC-02 inject | Injector |
| 11:40 | *Example:* SC-04 inject | Injector |
| 12:20 | *Example:* All faults removed; verify steady state | Safety officer |
| 12:30 | *Example:* Hot wash (first impressions, 30 min) | Facilitator |
| YYYY-MM-DD | Formal debrief (section 8) | Facilitator |

## 6. Safety controls and communication

<!-- Guidance: Define how to stop, how to tell a real incident from the exercise, and who must be told before, during and after. -->

### 6.1 Safety controls

- Abort criteria: [real customer error rate above 1% for 2 min], [any data loss signal], [genuine SEV-1 or SEV-2 elsewhere], [safety officer judgement].
- Every injection has a tested rollback with the expected time recorded in the inject sheet.
- Code phrase for a real incident: "[REAL INCIDENT]". Any message containing it ends the exercise for the affected team.
- Changes made by responders during the exercise are logged and reviewed before the window closes.

### 6.2 Communication plan

| When | Audience | Channel | Message owner |
|---|---|---|---|
| 5 working days before | Engineering, support, product | Email and [channel] | Facilitator |
| Start and end | All participants | [#gameday channel] | Facilitator |
| During | Responders | Normal incident channel tagged [EXERCISE] | Incident commander |
| During (production only) | Customer support | [Support channel] | Stakeholder liaison |
| After | Engineering leadership | Debrief summary | Facilitator |

## 7. Scoring detection and response

<!-- Guidance: Score against the objectives with times taken from the observer timeline. Score the system and process, never individuals. -->

Times are measured from the moment of injection.

| Scenario | Time to detect | Detected by | Time to engage IC | Time to mitigate | Time to recover | Runbook followed | Comms on time | Score (1-5) |
|---|---|---|---|---|---|---|---|---|
| SC-01 | *Example:* 3 min | Alert | 6 min | 19 min | 27 min | Yes, one gap | Yes | 4 |
| SC-02 | *Example:* 14 min | Customer report simulated by facilitator | 16 min | 22 min | 30 min | No runbook existed | Late (25 min) | 2 |
| SC-04 | *Example:* 9 min | Meta-monitoring | 12 min | 15 min | 15 min | Yes | Yes | 4 |
| [SC-NN] | [min] | [Source] | [min] | [min] | [min] | [Yes/No] | [Yes/No] | [1-5] |

Score guide: 5 = met every success measure; 4 = met with minor gaps;
3 = met after significant delay; 2 = partly met; 1 = not met.

## 8. Debrief and actions

<!-- Guidance: Hold a blameless debrief within five working days. Ask what happened, what surprised us, what helped, what got in the way. Every gap becomes an action with an owner. -->

### 8.1 Findings

| Finding ID | Finding | Evidence | Type |
|---|---|---|---|
| GD-F01 | *Example:* No alert for provider latency below the timeout | SC-02 detected after 14 min | Detection gap |
| GD-F02 | *Example:* Runbook RB-DB-04 step 6 refers to a removed command | Observer note 10:21 | Runbook gap |
| GD-F03 | *Example:* New IC handled comms well using the template | SC-01 status updates on time | What went well |
| [GD-FNN] | [Finding] | [Evidence] | [Type] |

### 8.2 Actions

| Action ID | Action | Finding | Owner | Due | Ticket | Status |
|---|---|---|---|---|---|---|
| GD-A01 | *Example:* Add latency SLO burn alert for provider calls | GD-F01 | [@sre] | YYYY-MM-DD | [OPS-520] | Open |
| GD-A02 | *Example:* Fix RB-DB-04 step 6 and add runbook test to quarterly review | GD-F02 | [@dba] | YYYY-MM-DD | [OPS-521] | Open |
| [GD-ANN] | [Action] | [Finding] | [@owner] | YYYY-MM-DD | [Ticket] | [Status] |

### 8.3 Next game day

Proposed date [YYYY-MM-DD] with focus on [repeat SC-02 after fixes / new
scenario]. Objectives not met today are carried forward.

## Standard and licence note

Based on common site reliability engineering practice for game days and
disaster recovery exercises. All text is original. Template licence: MIT.