Chaos Experiment Plan Template

Chaos engineering experiment: steady state, hypothesis, faults, blast radius, abort conditions, runbook, results, actions

운영표준Principles of Chaos Engineeringchaos experiment업데이트됨 2026. 10. 20.

에디터에서 열기.md 다운로드

사용법: Use it for each fault-injection experiment, in staging or production. Write the steady-state metrics, hypothesis and abort conditions first and get them approved before the run. The main pitfall is starting with a large blast radius or changing the hypothesis after seeing the results.

미리보기

Chaos Experiment Plan: [Experiment name]

Plan and record for a single chaos engineering experiment: the steady state you expect, the hypothesis, the faults you inject, how you limit the blast radius and when you stop, then what you observed and what you will change. Based on the experiment process described in the Principles of Chaos Engineering (principlesofchaos.org, last updated March 2019). Replace every [bracketed] value and every row marked Example.

Field	Value
Experiment ID	[CE-SERVICE-012]
Title	[Kill one ledger writer instance during peak load]
System / service	[Service name], [version]
Experiment owner	[@owner]
Approver	[@service-owner or SRE lead]
Planned date and window	YYYY-MM-DD, [14:00-15:00 UTC]
Environment	[Staging / Production canary / Production]
Status	[Draft / Approved / Run / Analysed / Closed]
Related	[FMEA FM-003], [Runbook], [Dashboards], [Previous run CE-SERVICE-011]

1. Purpose and background

This experiment tests whether [system] continues to deliver [function] when [fault]. It is motivated by [reason].

Example: Incident INC-2291 showed that duplicate ledger writes can occur when a writer crashes between commit and acknowledgement. Action ACT-01 in the payment FMEA added idempotency keys. This experiment checks that the fix holds under realistic load and that the duplicate-write alert fires if it does not.

2. Steady state and metrics

The steady state is defined by the following metrics, measured over the last [14] days at comparable load.

Metric ID	Metric	Source	Normal range	Tolerance during experiment
SS-01	Example: Successful authorisations per second	[Metrics query link]	950 to 1,150 at peak	No drop greater than 5% versus control
SS-02	Example: Authorisation p99 latency	[Query]	280 to 340 ms	Below 400 ms
SS-03	Example: Duplicate ledger entries	Reconciliation job	0	0
SS-04	Example: Ledger lag	[Query]	Under 2 s	Under 5 s
[SS-NN]	[Metric]	[Source]	[Range]	[Tolerance]

3. Hypothesis

Hypothesis: When [fault] occurs in [scope], the steady-state metrics SS-01 to SS-04 for the experimental group will stay within tolerance and will not differ from the control group by more than [5%].

Example: When one of the six ledger writer pods is terminated every 5 minutes for 30 minutes during peak load, successful authorisations, latency and ledger lag stay within tolerance, and no duplicate ledger entries are created.

Group	Definition	Size
Control	Example: Partitions 0-29 handled by unaffected writers	50% of traffic
Experimental	Example: Partitions 30-59 handled by writers targeted for termination	50% of traffic

4. Variables and faults injected

Fault ID	Fault type	Target	Magnitude	Duration / frequency	Tool / method
F-01	Example: Process termination	One ledger writer pod in experimental group	SIGKILL	Every 5 min for 30 min	[Chaos tool] pod-kill action
F-02	Example: Network latency	Calls to fraud provider	+800 ms on 20% of calls	15 min	[Chaos tool] network delay
F-03	Example: Dependency unavailable	Message broker node 2	Node stopped	10 min	Platform runbook step
[F-NN]	[Type]	[Target]	[Magnitude]	[Duration]	[Tool]

Faults are introduced one at a time unless the hypothesis is about a combination. Each fault has a tested way to stop it immediately (see 5.3).

5. Blast radius and safety

5.1 Blast radius

Dimension	Limit
Users or traffic affected	Example: At most 50% of authorisations for partitions 30-59
Regions / zones	Example: [Region A] only
Customers excluded	Example: Merchants on the high-value list
Data at risk	Example: None expected; ledger is reconciled after the run
Maximum duration	Example: 30 minutes of fault injection plus 15 minutes observation

5.2 Abort conditions

Stop the experiment immediately and roll back when any of the following is true for more than [1 minute]:

SS-01 drops more than [10%] below control.
SS-02 exceeds [600 ms].
SS-03 shows any duplicate entry.
Any unrelated incident of severity [SEV-2] or higher is declared.
The experiment owner or the on-call engineer asks to stop for any reason.

5.3 Rollback

Step	Action	Expected time
1	Example: Halt the chaos tool run with [command or button]	Under 10 s
2	Example: Confirm terminated pods are rescheduled and ready	Under 2 min
3	Example: Remove network fault rules and verify with probe	Under 1 min
4	Example: Announce stop in [channel] and record time	Immediate

6. Environment and prerequisites

Prerequisite	Status	Checked by
Example: Dashboards for SS-01 to SS-04 split by control and experimental group	[Ready]	[@name]
Example: On-call engineer informed and available	[Ready]	[@name]
Example: No change freeze or major event in the window	[Ready]	[@name]
Example: Rollback tested in staging on YYYY-MM-DD	[Ready]	[@name]
Example: Customer support informed (production only)	[Ready]	[@name]
[Prerequisite]	[Status]	[@name]

7. Runbook

Time (T+)	Step	Who	Check
T-30 min	Example: Go/no-go check of prerequisites; announce start in [channel]	Owner	All prerequisites ready
T-10 min	Example: Record baseline values for SS-01 to SS-04	Observer	Within normal range
T+0	Example: Start fault F-01	Operator	Tool shows run active
T+0 to T+30	Example: Watch abort conditions every minute	Observer	No abort condition met
T+30	Example: Stop fault injection	Operator	Tool shows run complete
T+30 to T+45	Example: Observe recovery; run reconciliation job	Owner	Metrics back to baseline
T+45	Example: Announce end in [channel]	Owner	Message posted

8. Observations

Time (UTC)	Observation	Metric values	Recorded by
Example: 14:05	First writer pod killed; replacement ready after 38 s	SS-04 peaked at 3.1 s	[@name]
Example: 14:10	Second kill; broker redelivered 214 messages	SS-03 = 0	[@name]
Example: 14:22	Alert "ledger lag high" did not fire although lag reached 4.6 s	Threshold set at 10 s	[@name]
[HH:MM]	[Observation]	[Values]	[@name]

9. Results

Outcome: [Hypothesis held / Hypothesis disproved / Inconclusive]

Metric	Control	Experimental	Within tolerance
SS-01	Example: 1,042/s	1,031/s	Yes
SS-02	Example: 310 ms	352 ms	Yes
SS-03	Example: 0	0	Yes
SS-04	Example: 1.2 s	4.6 s peak	Yes, but close to limit

Example: The hypothesis held. The idempotency fix prevented duplicates under 214 redeliveries. However, ledger lag approached the tolerance and the lag alert threshold is too high to warn operators in time.

10. Follow-up actions

Action ID	Action	Owner	Due	Ticket	Status
CE-12-A1	Example: Lower ledger lag alert threshold from 10 s to 4 s	[@sre]	YYYY-MM-DD	[OPS-412]	Open
CE-12-A2	Example: Reduce writer start-up time by preloading partition map	[@dev]	YYYY-MM-DD	[PAY-901]	Open
CE-12-A3	Example: Automate this experiment weekly in staging	[@sre]	YYYY-MM-DD	[OPS-413]	Open
CE-12-A4	Example: Update FMEA FM-003 detection score with this evidence	[@owner]	YYYY-MM-DD	[Link]	Open

Next run: [Repeat in production canary / Increase to two pods / Automate / Close].

Standard and licence note

Based on the experiment process and advanced principles published in the Principles of Chaos Engineering (https://principlesofchaos.org/): defining steady state, forming a hypothesis, introducing real-world variables, looking for a difference between control and experimental groups, and minimising the blast radius. No licence is stated for that page, so this template uses only its general concepts and credits the source; all text here is original. Template licence: MIT.