Game Day Plan Template

Game day exercise: objectives, scenarios, roles, schedule, safety controls, comms, detection and response scoring, debrief

운영표준game day plan업데이트됨 2026. 10. 20.

에디터에서 열기.md 다운로드

사용법: Use it to plan a team exercise that practises detecting and recovering from realistic failures. Set measurable objectives and safety controls first, then scenarios linked to real risks. The main pitfall is skipping the debrief, which is where the findings and actions come from.

미리보기

Game Day Plan: [Game day name]

Plan for a game day: a scheduled exercise in which a team practises detecting, responding to and recovering from realistic failures, then learns from the result. Covers objectives, scenarios, roles, schedule, safety controls, communication, scoring of detection and response, and the debrief. Based on common site reliability engineering practice. Replace every [bracketed] value and every row marked Example.

Field	Value
Game day ID	[GD-2026-05]
Title	[Regional failover and dependency outage]
Services in scope	[Service A], [Service B], [shared platform]
Date and time	YYYY-MM-DD, [09:30-13:00 UTC]
Environment	[Staging / Production with limited blast radius]
Facilitator	[@facilitator]
Approver	[@engineering-director or SRE lead]
Status	[Draft / Approved / Completed / Actions closed]
Related	[Incident response plan], [Runbooks], [Previous game day GD-2026-04], [Chaos experiment plans]

1. Objectives

Objective ID	Objective	Success measure
OBJ-01	Example: Validate that the primary on-call detects a database primary failure through alerts, not through customers	Page received within 5 min of injection
OBJ-02	Example: Practise regional failover using the runbook only	Failover complete within 30 min (RTO) with no runbook gaps
OBJ-03	Example: Exercise incident communication	Status page updated within 15 min; stakeholder update every 30 min
OBJ-04	Example: Train two new on-call engineers as incident commander	Each leads one scenario with a shadow
[OBJ-NN]	[Objective]	[Measure]

2. Scope and assumptions

In scope	Out of scope
Example: Checkout and payment services in [Region A]	Example: Customer-facing traffic in [Region B]
Example: Primary database cluster for checkout	Example: Data deletion or corruption scenarios
Example: Paging, chat and status page tooling	Example: Real external customer notifications

Assumptions:

[Assumption, for example: the DR runbook version 3.2 is the current approved version.]
[Assumption, for example: production traffic is below 40% of peak during the window.]

3. Scenarios

Scenario ID	Scenario	Injection	Expected detection	Expected response	Linked risk
SC-01	Example: Checkout database primary fails	Stop primary instance in [Region A]	Replica lag and error-rate alerts page primary on-call	Promote replica per runbook RB-DB-04	FMEA FM-011
SC-02	Example: Payment provider slow	Add 2 s latency to provider calls via proxy	Latency SLO burn alert	Enable degraded mode flag; communicate	INC-2291
SC-03	Example: Expired TLS certificate on internal gateway	Deploy expired certificate to staging gateway	Synthetic check failure	Rotate certificate with runbook RB-SEC-02	Audit finding A-17
SC-04	Example: Loss of observability (metrics backend down)	Block metrics ingestion for 15 min	Meta-monitoring alert	Switch to logs and fallback dashboard	New risk
[SC-NN]	[Scenario]	[Injection]	[Detection]	[Response]	[Risk]

4. Roles and responsibilities

Role	Name	Responsibilities
Facilitator	[@name]	Runs the schedule, releases injects, decides to pause or stop, leads debrief
Injector / operator	[@name]	Executes fault injections and rollbacks on the facilitator's signal
Safety officer	[@name]	Watches abort criteria and real customer impact; can stop at any time
Responders	[@primary], [@secondary], [@incident-commander]	Respond as in a real incident using normal tools and runbooks
Observers / scribes	[@name], [@name]	Record timeline, decisions, gaps and quotes
Stakeholder liaison	[@name]	Keeps support, product and leadership informed
Real-incident on-call	[@name]	Not part of the exercise; handles any genuine incident

5. Schedule

Time (UTC)	Activity	Lead
09:30	Example: Briefing: objectives, safety rules, code word for real incidents	Facilitator
09:45	Example: Baseline check of dashboards and steady state	Observers
10:00	Example: SC-01 inject	Injector
10:45	Example: SC-01 stop, recover, notes	Facilitator
11:00	Example: SC-02 inject	Injector
11:40	Example: SC-04 inject	Injector
12:20	Example: All faults removed; verify steady state	Safety officer
12:30	Example: Hot wash (first impressions, 30 min)	Facilitator
YYYY-MM-DD	Formal debrief (section 8)	Facilitator

6. Safety controls and communication

6.1 Safety controls

Abort criteria: [real customer error rate above 1% for 2 min], [any data loss signal], [genuine SEV-1 or SEV-2 elsewhere], [safety officer judgement].
Every injection has a tested rollback with the expected time recorded in the inject sheet.
Code phrase for a real incident: "[REAL INCIDENT]". Any message containing it ends the exercise for the affected team.
Changes made by responders during the exercise are logged and reviewed before the window closes.

6.2 Communication plan

When	Audience	Channel	Message owner
5 working days before	Engineering, support, product	Email and [channel]	Facilitator
Start and end	All participants	[#gameday channel]	Facilitator
During	Responders	Normal incident channel tagged [EXERCISE]	Incident commander
During (production only)	Customer support	[Support channel]	Stakeholder liaison
After	Engineering leadership	Debrief summary	Facilitator

7. Scoring detection and response

Times are measured from the moment of injection.

Scenario	Time to detect	Detected by	Time to engage IC	Time to mitigate	Time to recover	Runbook followed	Comms on time	Score (1-5)
SC-01	Example: 3 min	Alert	6 min	19 min	27 min	Yes, one gap	Yes	4
SC-02	Example: 14 min	Customer report simulated by facilitator	16 min	22 min	30 min	No runbook existed	Late (25 min)	2
SC-04	Example: 9 min	Meta-monitoring	12 min	15 min	15 min	Yes	Yes	4
[SC-NN]	[min]	[Source]	[min]	[min]	[min]	[Yes/No]	[Yes/No]	[1-5]

Score guide: 5 = met every success measure; 4 = met with minor gaps; 3 = met after significant delay; 2 = partly met; 1 = not met.

8. Debrief and actions

8.1 Findings

Finding ID	Finding	Evidence	Type
GD-F01	Example: No alert for provider latency below the timeout	SC-02 detected after 14 min	Detection gap
GD-F02	Example: Runbook RB-DB-04 step 6 refers to a removed command	Observer note 10:21	Runbook gap
GD-F03	Example: New IC handled comms well using the template	SC-01 status updates on time	What went well
[GD-FNN]	[Finding]	[Evidence]	[Type]

8.2 Actions

Action ID	Action	Finding	Owner	Due	Ticket	Status
GD-A01	Example: Add latency SLO burn alert for provider calls	GD-F01	[@sre]	YYYY-MM-DD	[OPS-520]	Open
GD-A02	Example: Fix RB-DB-04 step 6 and add runbook test to quarterly review	GD-F02	[@dba]	YYYY-MM-DD	[OPS-521]	Open
[GD-ANN]	[Action]	[Finding]	[@owner]	YYYY-MM-DD	[Ticket]	[Status]

8.3 Next game day

Proposed date [YYYY-MM-DD] with focus on [repeat SC-02 after fixes / new scenario]. Objectives not met today are carried forward.

Standard and licence note

Based on common site reliability engineering practice for game days and disaster recovery exercises. All text is original. Template licence: MIT.