# Chaos Experiment Plan: [Experiment name]

> Plan and record for a single chaos engineering experiment: the steady
> state you expect, the hypothesis, the faults you inject, how you limit
> the blast radius and when you stop, then what you observed and what you
> will change. Based on the experiment process described in the Principles
> of Chaos Engineering (principlesofchaos.org, last updated March 2019).
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- One plan per experiment. A series of related experiments can share an
  index page that links to each plan.
- Write the hypothesis and abort conditions before the run and get them
  approved. Never change them during the run to make the result look better.
- Start small: a single instance, a small share of traffic, a test
  environment. Grow the blast radius only after earlier runs pass.
- An experiment that disproves the hypothesis is a success: it found a
  weakness before customers did. Record it honestly and fund the fix.
-->

| Field | Value |
|---|---|
| Experiment ID | [CE-SERVICE-012] |
| Title | [Kill one ledger writer instance during peak load] |
| System / service | [Service name], [version] |
| Experiment owner | [@owner] |
| Approver | [@service-owner or SRE lead] |
| Planned date and window | YYYY-MM-DD, [14:00-15:00 UTC] |
| Environment | [Staging / Production canary / Production] |
| Status | [Draft / Approved / Run / Analysed / Closed] |
| Related | [FMEA FM-003], [Runbook], [Dashboards], [Previous run CE-SERVICE-011] |

## 1. Purpose and background

<!-- Guidance: Explain why this experiment matters now. Link it to a risk, an incident, an FMEA row or a new design element. Experiments without a reason tend to test what is already known. -->

This experiment tests whether [system] continues to deliver [function]
when [fault]. It is motivated by [reason].

*Example:* Incident INC-2291 showed that duplicate ledger writes can occur
when a writer crashes between commit and acknowledgement. Action ACT-01 in
the payment FMEA added idempotency keys. This experiment checks that the
fix holds under realistic load and that the duplicate-write alert fires if
it does not.

## 2. Steady state and metrics

<!-- Guidance: Steady state is a measurable output that shows the system is behaving normally, usually a business or user-facing metric, not CPU. Record the normal range from recent data so the comparison is objective. -->

The steady state is defined by the following metrics, measured over the
last [14] days at comparable load.

| Metric ID | Metric | Source | Normal range | Tolerance during experiment |
|---|---|---|---|---|
| SS-01 | *Example:* Successful authorisations per second | [Metrics query link] | 950 to 1,150 at peak | No drop greater than 5% versus control |
| SS-02 | *Example:* Authorisation p99 latency | [Query] | 280 to 340 ms | Below 400 ms |
| SS-03 | *Example:* Duplicate ledger entries | Reconciliation job | 0 | 0 |
| SS-04 | *Example:* Ledger lag | [Query] | Under 2 s | Under 5 s |
| [SS-NN] | [Metric] | [Source] | [Range] | [Tolerance] |

## 3. Hypothesis

<!-- Guidance: State the hypothesis as "steady state will continue in both the control and experimental groups when X happens". Make it falsifiable with the metrics above. -->

**Hypothesis:** When [fault] occurs in [scope], the steady-state metrics
SS-01 to SS-04 for the experimental group will stay within tolerance and
will not differ from the control group by more than [5%].

*Example:* When one of the six ledger writer pods is terminated every 5
minutes for 30 minutes during peak load, successful authorisations,
latency and ledger lag stay within tolerance, and no duplicate ledger
entries are created.

| Group | Definition | Size |
|---|---|---|
| Control | *Example:* Partitions 0-29 handled by unaffected writers | 50% of traffic |
| Experimental | *Example:* Partitions 30-59 handled by writers targeted for termination | 50% of traffic |

## 4. Variables and faults injected

<!-- Guidance: Describe each fault precisely: type, target, magnitude, duration and tool. Prefer faults that reflect real events seen in incidents or in the FMEA. -->

| Fault ID | Fault type | Target | Magnitude | Duration / frequency | Tool / method |
|---|---|---|---|---|---|
| F-01 | *Example:* Process termination | One ledger writer pod in experimental group | SIGKILL | Every 5 min for 30 min | [Chaos tool] pod-kill action |
| F-02 | *Example:* Network latency | Calls to fraud provider | +800 ms on 20% of calls | 15 min | [Chaos tool] network delay |
| F-03 | *Example:* Dependency unavailable | Message broker node 2 | Node stopped | 10 min | Platform runbook step |
| [F-NN] | [Type] | [Target] | [Magnitude] | [Duration] | [Tool] |

Faults are introduced one at a time unless the hypothesis is about a
combination. Each fault has a tested way to stop it immediately (see 5.3).

## 5. Blast radius and safety

<!-- Guidance: Define the largest possible impact and how you keep it small. Abort conditions must be specific and measurable, and someone must be watching them the whole time. -->

### 5.1 Blast radius

| Dimension | Limit |
|---|---|
| Users or traffic affected | *Example:* At most 50% of authorisations for partitions 30-59 |
| Regions / zones | *Example:* [Region A] only |
| Customers excluded | *Example:* Merchants on the high-value list |
| Data at risk | *Example:* None expected; ledger is reconciled after the run |
| Maximum duration | *Example:* 30 minutes of fault injection plus 15 minutes observation |

### 5.2 Abort conditions

Stop the experiment immediately and roll back when any of the following is
true for more than [1 minute]:

- SS-01 drops more than [10%] below control.
- SS-02 exceeds [600 ms].
- SS-03 shows any duplicate entry.
- Any unrelated incident of severity [SEV-2] or higher is declared.
- The experiment owner or the on-call engineer asks to stop for any reason.

### 5.3 Rollback

| Step | Action | Expected time |
|---|---|---|
| 1 | *Example:* Halt the chaos tool run with [command or button] | Under 10 s |
| 2 | *Example:* Confirm terminated pods are rescheduled and ready | Under 2 min |
| 3 | *Example:* Remove network fault rules and verify with probe | Under 1 min |
| 4 | *Example:* Announce stop in [channel] and record time | Immediate |

## 6. Environment and prerequisites

<!-- Guidance: Confirm the environment is ready and that the observability needed to judge the hypothesis exists. An experiment you cannot observe is not worth running. -->

| Prerequisite | Status | Checked by |
|---|---|---|
| *Example:* Dashboards for SS-01 to SS-04 split by control and experimental group | [Ready] | [@name] |
| *Example:* On-call engineer informed and available | [Ready] | [@name] |
| *Example:* No change freeze or major event in the window | [Ready] | [@name] |
| *Example:* Rollback tested in staging on YYYY-MM-DD | [Ready] | [@name] |
| *Example:* Customer support informed (production only) | [Ready] | [@name] |
| [Prerequisite] | [Status] | [@name] |

## 7. Runbook

<!-- Guidance: A step-by-step script for the run with timings and who does what. Keep a timestamped log while running. -->

| Time (T+) | Step | Who | Check |
|---|---|---|---|
| T-30 min | *Example:* Go/no-go check of prerequisites; announce start in [channel] | Owner | All prerequisites ready |
| T-10 min | *Example:* Record baseline values for SS-01 to SS-04 | Observer | Within normal range |
| T+0 | *Example:* Start fault F-01 | Operator | Tool shows run active |
| T+0 to T+30 | *Example:* Watch abort conditions every minute | Observer | No abort condition met |
| T+30 | *Example:* Stop fault injection | Operator | Tool shows run complete |
| T+30 to T+45 | *Example:* Observe recovery; run reconciliation job | Owner | Metrics back to baseline |
| T+45 | *Example:* Announce end in [channel] | Owner | Message posted |

## 8. Observations

<!-- Guidance: Record facts with timestamps during the run. Interpretation goes in section 9. -->

| Time (UTC) | Observation | Metric values | Recorded by |
|---|---|---|---|
| *Example:* 14:05 | First writer pod killed; replacement ready after 38 s | SS-04 peaked at 3.1 s | [@name] |
| *Example:* 14:10 | Second kill; broker redelivered 214 messages | SS-03 = 0 | [@name] |
| *Example:* 14:22 | Alert "ledger lag high" did not fire although lag reached 4.6 s | Threshold set at 10 s | [@name] |
| [HH:MM] | [Observation] | [Values] | [@name] |

## 9. Results

<!-- Guidance: State clearly whether the hypothesis held. Compare against the control group, not against hopes. List surprises; they are often the most valuable output. -->

**Outcome:** [Hypothesis held / Hypothesis disproved / Inconclusive]

| Metric | Control | Experimental | Within tolerance |
|---|---|---|---|
| SS-01 | *Example:* 1,042/s | 1,031/s | Yes |
| SS-02 | *Example:* 310 ms | 352 ms | Yes |
| SS-03 | *Example:* 0 | 0 | Yes |
| SS-04 | *Example:* 1.2 s | 4.6 s peak | Yes, but close to limit |

*Example:* The hypothesis held. The idempotency fix prevented duplicates
under 214 redeliveries. However, ledger lag approached the tolerance and
the lag alert threshold is too high to warn operators in time.

## 10. Follow-up actions

<!-- Guidance: Turn weaknesses into tracked work with owners. Decide whether to repeat the experiment, widen its scope or automate it. -->

| Action ID | Action | Owner | Due | Ticket | Status |
|---|---|---|---|---|---|
| CE-12-A1 | *Example:* Lower ledger lag alert threshold from 10 s to 4 s | [@sre] | YYYY-MM-DD | [OPS-412] | Open |
| CE-12-A2 | *Example:* Reduce writer start-up time by preloading partition map | [@dev] | YYYY-MM-DD | [PAY-901] | Open |
| CE-12-A3 | *Example:* Automate this experiment weekly in staging | [@sre] | YYYY-MM-DD | [OPS-413] | Open |
| CE-12-A4 | *Example:* Update FMEA FM-003 detection score with this evidence | [@owner] | YYYY-MM-DD | [Link] | Open |

**Next run:** [Repeat in production canary / Increase to two pods / Automate / Close].

## Standard and licence note

Based on the experiment process and advanced principles published in the
Principles of Chaos Engineering (https://principlesofchaos.org/): defining
steady state, forming a hypothesis, introducing real-world variables,
looking for a difference between control and experimental groups, and
minimising the blast radius. No licence is stated for that page, so this
template uses only its general concepts and credits the source; all text
here is original. Template licence: MIT.