# DR Test Report: [Test name] [DRT-YYYY-NN]

> Evidence report for one disaster recovery (DR) test or exercise: how
> the event is classified, scope and objectives, scenario, participants,
> the timeline, recovery time and recovery point achieved against target
> for every system, data integrity and functional checks, deviations
> from the plan, findings, corrective actions and sign-off. Event types
> and terms follow NIST SP 800-84 (Guide to Test, Training, and Exercise
> Programs for IT Plans and Capabilities, September 2006) and NIST SP
> 800-34 Rev. 1 (Contingency Planning Guide for Federal Information
> Systems, May 2010, updated November 2010). Evidence is mapped to NIST
> SP 800-53 Rev. 5 (release 5.2.0) controls CP-4, CP-9 and CP-10 and to
> the clause structure of ISO 22301:2019 (with Amendment 1:2024).
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Write one report per test event. The plan that the test exercises is
  the IT Disaster Recovery Plan (`ops/disaster-recovery-plan`); its
  section 9 schedules tests and lists the results this report proves.
- Fill section 2 (classification and measurement rules) before the
  test, so that nobody argues afterwards about when the clock started.
- Record timestamps live, in one time zone, from one source (the
  incident channel or a scribe log). Reconstructing them later from
  memory is the most common reason a DR test report fails an audit.
- Recompute every derived number in section 7 from the raw timestamps
  in section 6. A reviewer must be able to repeat the arithmetic.
- A failed objective is a useful result, not a reason to soften the
  report. State it plainly and track the action to closure.
- Related documents: IT Disaster Recovery Plan
  (`ops/disaster-recovery-plan`), Backup and Restore Procedure
  (`ops/backup-restore-procedure`), Backup Policy (`ops/backup-policy`),
  Resilience Tiering, HA and DR Design
  (`architecture/resilience-tiering-ha-dr-design`), Game Day Plan
  (`ops/game-day-plan`), Tabletop Exercise (`security/tabletop-exercise`),
  Business Continuity Plan (`compliance/business-continuity`),
  Incident Severity Matrix (`ops/incident-severity-matrix`).
- Worked example: Corvid Commerce, a fictional SaaS company, tests a
  failover of its order-taking service from region A to region B. All
  rows marked *Example* belong to that fictional test.
-->

## Document control

<!-- Guidance: Identify the event and the plan version it tested. A test of an out-of-date plan version proves little; record the exact plan version used. -->

| Field | Value |
|---|---|
| Document ID | [DRT-YYYY-NN] |
| Test name | [Test name] |
| Plan tested | [IT Disaster Recovery Plan ID and version] |
| Event type (section 2.1) | [Tabletop exercise / Functional exercise / Full-scale exercise / Component test / System test / Comprehensive test] |
| Test date and window | [YYYY-MM-DD HH:MM to HH:MM time zone] |
| Environment | [Production / Production-like isolated / Staging] |
| Test director | [@test-director] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Actions closed] |
| Owner | [@dr-coordinator] |
| Approvers/reviewers | [@service-owner], [@head-of-infrastructure], [@ciso], [@internal-audit] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal / Confidential] |
| Overall result | [Pass / Pass with findings / Fail] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Test director | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| Service owner | [@name] | [Decision] | YYYY-MM-DD |
| Head of infrastructure | [@name] | [Decision] | YYYY-MM-DD |
| Business continuity manager | [@name] | [Decision] | YYYY-MM-DD |
| Internal audit (observer) | [@name] | [Noted] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Draft written within 2 business days of the test |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Integrity reconciliation results added (section 8) |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved; actions DRT-A-01 to DRT-A-06 opened |

## Contents

<!-- Guidance: Keep this list aligned with the headings below. -->

1. [Executive summary](#1-executive-summary)
2. [Test classification and measurement method](#2-test-classification-and-measurement-method)
3. [Scope and objectives](#3-scope-and-objectives)
4. [Scenario, assumptions and safety controls](#4-scenario-assumptions-and-safety-controls)
5. [Participants and roles](#5-participants-and-roles)
6. [Timeline](#6-timeline)
7. [Recovery objectives: target versus achieved](#7-recovery-objectives-target-versus-achieved)
8. [Data integrity and functional validation](#8-data-integrity-and-functional-validation)
9. [Deviations from the plan](#9-deviations-from-the-plan)
10. [Findings](#10-findings)
11. [Corrective actions](#11-corrective-actions)
12. [Evidence and control mapping](#12-evidence-and-control-mapping)
13. [Conclusion and sign-off](#13-conclusion-and-sign-off)
14. [Glossary and references](#14-glossary-and-references)

## 1. Executive summary

<!-- Guidance: Write this last, in five to eight sentences a director can read in one minute. Questions: did the business service come back within its objective, which systems missed, what is the single most important fix? Pitfall: reporting only per-system results when the business cares about the end-to-end service. -->

[Summarise the purpose of the test, what was failed over or restored,
the overall result, the systems that missed an objective, the most
important finding and the date by which the fix is due.]

*Example:* On 2026-09-19 Corvid Commerce ran a live failover combining
system tests of four systems with a functional exercise of activation
and notification (event types in section 2.1): the order-taking service
was failed over from region A to region B in production, during a
pre-announced low-traffic window, by isolating region A at the network
layer. Three of the four systems in scope met their recovery time
objective (RTO) and all four met their recovery point objective (RPO).
The payments ledger service missed its 60-minute RTO by 12 minutes,
mainly because its encryption key policy in region B did not grant the
recovery role access and a break-glass procedure was needed, and also
because its runbook started 5 minutes after its database was ready. Because
order taking needs the payments ledger, the end-to-end business service
was restored after 72 minutes against a 60-minute objective. The test
result is therefore **Pass with findings**. The key policy fix
(DRT-A-01) is due on 2026-10-17, the dependency-ordered recovery
checklist (DRT-A-05) on 2026-11-07, and a targeted retest of the payments
ledger (DRT-A-06) is booked for 2026-11-14.

| Headline measure | Target | Achieved | Result |
|---|---|---|---|
| End-to-end service "place an order" restored *(Example)* | 60 min | 72 min | Missed by 12 min |
| Systems meeting RTO *(Example)* | 4 of 4 | 3 of 4 | Missed |
| Systems meeting RPO *(Example)* | 4 of 4 | 4 of 4 | Met |
| Integrity checks passed *(Example)* | 6 of 6 | 6 of 6 | Met |
| Findings raised *(Example)* | n/a | 6 (1 High, 2 Medium, 2 Low, 1 Observation) | n/a |
| Failback completed inside the window *(Example)* | Yes | Yes | Met |

## 2. Test classification and measurement method

<!-- Guidance: Agree how the event is classified and how every number is measured before the test starts. Questions: is this a discussion, a simulated execution or a real recovery? Where does the clock start and stop? Pitfall: measuring RTO from the moment the team started the runbook, which hides detection and decision time. -->

### 2.1 Event type

NIST SP 800-84 separates exercises, which validate a plan and the
people who carry it out, from tests, which evaluate systems or
components. Classify the event with the terms below and record the
classification in Document control. Many organisations also use the
industry terms walkthrough, simulation, parallel test and full
interruption test; the right-hand column shows how they usually map.

| Event type (SP 800-84 term) | What happens | Typical evidence | Common industry term |
|---|---|---|---|
| Tabletop exercise | Participants discuss roles and decisions against a scenario; no systems are touched | Facilitator notes, decisions, gaps in the plan | Walkthrough, tabletop |
| Functional exercise | Participants perform their duties in a simulated operational environment for one or more functions of the plan | Timed actions, inject log, observer notes | Simulation |
| Full-scale exercise | A functional exercise that covers every element of the plan, with senior and operational staff | All of the above for the whole plan | Full simulation |
| Component test | One hardware or software component is tested, for example one backup restore | Restore log, checksum output | Restore test |
| System test | A complete system is recovered and checked against its requirements | Runbook log, RTO and RPO measured | Parallel test (recovery site runs beside production) |
| Comprehensive test | All systems and components that support the plan are tested together | Full timeline, end-to-end checks | Full interruption or cutover test |

A discussion-based exercise belongs in a Tabletop Exercise
(`security/tabletop-exercise`) record; a fault-injection event that
tests detection and response behaviour belongs in a Game Day Plan
(`ops/game-day-plan`). Use this report when systems are actually
recovered or restored and recovery objectives are measured.

### 2.2 Measurement rules

| ID | Rule | Value for this test |
|---|---|---|
| MR-01 | Clock source | [NTP-synchronised time from the incident channel, UTC] |
| MR-02 | Disruption time, T0 | [Moment the fault is injected or the outage begins] |
| MR-03 | Recovery time for a system | [T0 to the moment the system passes its validation checks in section 8] |
| MR-04 | Recovery time for a business service | [T0 to the moment the last hard dependency of the service is validated] |
| MR-05 | Data loss window | [Time of failure minus the commit time of the last transaction present after recovery] |
| MR-06 | Pass criteria | [Achieved recovery time at or below RTO and data loss window at or below RPO] |
| MR-07 | Low margin flag | [Margin below 10% of the objective is reported as an Observation even when met] |

The formulas used in section 7 are:

```text
Recovery time (min)  = T(validated) - T0
Variance (min)       = Recovery time - RTO          (negative is better)
RTO margin (%)       = (RTO - Recovery time) / RTO x 100
Data loss window (s) = T0 - T(last committed transaction present after recovery)
RPO margin (%)       = (RPO - Data loss window) / RPO x 100
```

*Example:* Corvid Commerce measures RTO from T0, not from the
declaration of disaster, because its customers experience the outage
from T0. Detection and decision time are therefore part of the measured
recovery time and appear as separate phases in section 7.3.

## 3. Scope and objectives

<!-- Guidance: List exactly which systems, data sets and business services the test covers, and which it deliberately does not. Questions: does the scope include every hard dependency of the business service? Which objectives come from the resilience tier? Pitfall: testing a database failover while the identity provider or key management service it depends on stays in the healthy region. -->

### 3.1 Systems in scope

| System ID | System | Resilience tier | RTO | RPO | Recovery method under test | Owner |
|---|---|---|---|---|---|---|
| SYS-01 *(Example)* | Checkout API | T1 | 60 min | n/a (stateless) | Redeploy and scale out in region B | [@name] |
| SYS-02 *(Example)* | Orders database (PostgreSQL) | T1 | 60 min | 5 min | Promote asynchronous cross-region replica | [@name] |
| SYS-05 *(Example)* | Payments ledger service | T1 | 60 min | 1 min | Fail over to synchronous standby in region B | [@name] |
| SYS-07 *(Example)* | Catalogue search index | T3 (stricter RTO from the BIA) | 4 h | 24 h | Restore nightly snapshot, then rebuild deltas | [@name] |
| [SYS-NN] | [System] | [Tier] | [RTO] | [RPO] | [Method] | [@owner] |

System IDs, tiers and objectives come from section 3.1 of the IT
Disaster Recovery Plan (`ops/disaster-recovery-plan`) and must match it
exactly; in the worked example they are copied from the Corvid Commerce
rows of that section. If a value differs, the plan wins and the report
is corrected before approval. Tier definitions come from
Resilience Tiering, HA and DR Design
(`architecture/resilience-tiering-ha-dr-design`).

### 3.2 Out of scope

| Item | Reason | When it will be tested |
|---|---|---|
| SYS-03 Reporting warehouse *(Example)* | T3, backup tier B2: full timed restore tested separately every year, as section 8 of the Backup Policy (`ops/backup-policy`) requires | [DRT-YYYY-NN], YYYY-MM-DD |
| Customer support tooling (SaaS) *(Example)* | Vendor-operated; vendor DR evidence reviewed instead | Annual supplier review |
| [Item] | [Reason] | [Plan] |

### 3.3 Objectives

| ID | Objective | Success criterion | Result |
|---|---|---|---|
| DRT-OBJ-01 *(Example)* | Restore order taking in region B within the business service RTO | MR-04 at or below 60 min | Not met (72 min) |
| DRT-OBJ-02 *(Example)* | Confirm data loss stays within RPO for every stateful system | MR-05 at or below RPO for SYS-02, SYS-05, SYS-07 | Met |
| DRT-OBJ-03 *(Example)* | Prove recovered data is complete and consistent | All checks INT-01 to INT-06 pass | Met |
| DRT-OBJ-04 *(Example)* | Validate the activation and notification procedure | Activation decision within 15 min of T0; contact tree completed | Met with finding DRT-F-05 |
| DRT-OBJ-05 *(Example)* | Fail back to region A without data loss inside the window | Failback validated before 16:00 UTC | Met |
| [DRT-OBJ-NN] | [Objective] | [Criterion] | [Met / Not met] |

## 4. Scenario, assumptions and safety controls

<!-- Guidance: Describe the scenario precisely enough that the test can be repeated. Questions: how was the failure created, what did responders know in advance, and what would have stopped the test? Pitfall: a scenario that quietly leaves one shared dependency healthy, so the test passes but a real disaster would not. -->

### 4.1 Scenario

[Describe the disruption being simulated, how it was injected, and what
responders were told before the start.]

*Example:* Region A becomes unreachable for all Corvid Commerce
workloads. The platform team injects the fault at T0 = 09:00:00 UTC by
applying a deny-all network policy between region A and the internet
and between region A and region B. The on-call engineers know the date
and window but not the exact time or the failure mode. Region A data is
treated as lost for the duration of the test: nobody may copy data out
of region A to help the recovery.

### 4.2 Assumptions and constraints

| ID | Assumption or constraint | Impact on the result |
|---|---|---|
| AS-01 *(Example)* | Test ran in a low-traffic window (Saturday 09:00 UTC, about 18% of weekday peak) | Region B capacity was not stressed; capacity under peak remains unproven |
| AS-02 *(Example)* | Payment service provider was told in advance and allowed traffic from region B addresses | A real event would also need the provider to accept the new addresses |
| AS-03 *(Example)* | Identity provider is a SaaS outside both regions | Identity was not part of the failure |
| [AS-NN] | [Assumption] | [Impact] |

### 4.3 Safety controls and abort criteria

| ID | Control | Owner | Triggered |
|---|---|---|---|
| SC-01 *(Example)* | Abort and restore region A routing if customer-facing error rate above 5% for 10 min after the planned recovery time | Test director | No |
| SC-02 *(Example)* | Fresh backups of SYS-02 and SYS-05 taken and verified before T0 | Database lead | n/a (precondition met) |
| SC-03 *(Example)* | Status page notice published 48 h before the window | Communications lead | n/a |
| SC-04 *(Example)* | Change record approved for the test window | Change manager | n/a |
| [SC-NN] | [Control] | [@owner] | [Yes / No] |

## 5. Participants and roles

<!-- Guidance: Record who did what, so findings about people and process can be traced. Questions: were the people who would respond in a real event the ones who responded? Was there an independent observer? Pitfall: the engineer who wrote the runbook executes it, so gaps in the written steps are never exposed. -->

| Role | Name | Responsibility during the test |
|---|---|---|
| Test director | [@name] | Owns go, pause and abort decisions; declares end of test |
| Incident commander | [@name] | Runs the response as in a real event; role defined in the Incident Commander Handbook (`ops/incident-commander-handbook`) |
| Recovery engineers | [@name], [@name] | Execute runbooks for each system |
| Scribe | [@name] | Records timestamps and decisions in the timeline |
| Observer or evaluator | [@name] | Independent; checks measurement rules and records deviations |
| Business validator | [@name] | Confirms business functions work after recovery |
| Communications lead | [@name] | Internal and customer notices |

*Example:* At Corvid Commerce the runbooks were executed by the
secondary on-call engineers, not by their authors, and an engineer from
the security team acted as independent observer.

## 6. Timeline

<!-- Guidance: One row per material event, with the exact timestamp. Questions: when was the failure detected, when was the decision taken, when did each system start and finish recovery, and when was it validated? Pitfall: recording only start and end times, so the phase that consumed the time cannot be found. -->

| Time (UTC) | Elapsed from T0 | Event | System | Source |
|---|---|---|---|---|
| 09:00:00 | 0 min | *Example:* Fault injected, region A isolated (T0) | All | Platform change log |
| 09:03 | 3 min | *Example:* Availability alert fires for checkout | SYS-01 | Alerting system |
| 09:09 | 9 min | *Example:* Incident declared, incident commander assigned | All | Incident channel |
| 09:14 | 14 min | *Example:* DR activation decided by incident commander and service owner | All | Incident channel |
| 09:16 | 16 min | *Example:* Replica promotion runbook started | SYS-02 | Runbook log |
| 09:20 | 20 min | *Example:* Snapshot restore of search index started in region B | SYS-07 | Runbook log |
| 09:31 | 31 min | *Example:* Replica promoted; runbook step 7 command failed, corrected (DEV-01) | SYS-02 | Runbook log |
| 09:47 | 47 min | *Example:* Integrity checks INT-01 and INT-02 passed; SYS-02 validated | SYS-02 | Validation log |
| 09:49 | 49 min | *Example:* Checkout API scaled out in region B | SYS-01 | Deployment log |
| 09:50 | 50 min | *Example:* DNS weighted record for checkout switched to region B (DEV-04) | SYS-01 | DNS change log |
| 09:52 | 52 min | *Example:* Payments ledger failover started | SYS-05 | Runbook log |
| 09:58 | 58 min | *Example:* Checkout smoke tests passed; SYS-01 validated | SYS-01 | Validation log |
| 09:59 | 59 min | *Example:* Ledger fails to start: key access denied in region B (DRT-F-01) | SYS-05 | Service logs |
| 10:07 | 67 min | *Example:* Break-glass key access granted by security on-call | SYS-05 | Key management audit log |
| 10:12 | 72 min | *Example:* Ledger reconciliation INT-03 passed; SYS-05 validated; order taking restored end to end | SYS-05 | Validation log |
| 11:41 | 161 min | *Example:* Search index rebuilt and validated | SYS-07 | Validation log |
| 12:30 | 210 min | *Example:* Failback to region A started | All | Change log |
| 15:10 | 370 min | *Example:* Failback validated, region B returned to standby | All | Validation log |
| 15:30 | 390 min | *Example:* Test director declares end of test | All | Incident channel |
| [HH:MM] | [N min] | [Event] | [SYS-NN] | [Source] |

```mermaid
gantt
    title Example recovery timeline (minutes from T0)
    dateFormat mm
    axisFormat %M
    section Detect and decide
    Detection           :d1, 00, 3m
    Declare and decide  :d2, after d1, 11m
    section SYS-02 Orders database
    Promote and validate :s2, 16, 31m
    section SYS-01 Checkout API
    Scale out and validate :s1, 49, 9m
    section SYS-05 Payments ledger
    Fail over and validate :s5, 52, 20m
```

## 7. Recovery objectives: target versus achieved

<!-- Guidance: This is the core evidence. Compute every value with the formulas in section 2.2 and the timestamps in section 6. Questions: which systems missed, by how much, and which phase consumed the time? Pitfall: copying "achieved" values from the runbook author's estimate instead of the timeline. -->

### 7.1 Recovery time

| System ID | RTO | T(validated) | Recovery time | Variance | RTO margin | Result |
|---|---|---|---|---|---|---|
| SYS-02 *(Example)* | 60 min | 09:47 | 47 min | -13 min | 21.7% | Met |
| SYS-01 *(Example)* | 60 min | 09:58 | 58 min | -2 min | 3.3% | Met, low margin (DRT-F-04) |
| SYS-05 *(Example)* | 60 min | 10:12 | 72 min | +12 min | -20.0% | Not met (DRT-F-01) |
| SYS-07 *(Example)* | 240 min | 11:41 | 161 min | -79 min | 32.9% | Met |
| Business service "place an order" (SYS-01, SYS-02, SYS-05) *(Example)* | 60 min | 10:12 | 72 min | +12 min | -20.0% | Not met |
| [SYS-NN] | [RTO] | [HH:MM] | [N min] | [+/- N min] | [N%] | [Met / Not met] |

Check of the arithmetic for SYS-05 *(Example)*: T(validated) 10:12
minus T0 09:00 is 72 minutes; 72 minus the 60-minute RTO gives a
variance of +12 minutes; (60 - 72) / 60 = -20.0%. The business service
time equals the latest validation among its hard dependencies, which is
SYS-05 at 72 minutes.

### 7.2 Recovery point

| System ID | RPO | Last transaction present after recovery | Data loss window | RPO margin | Result |
|---|---|---|---|---|---|
| SYS-02 *(Example)* | 300 s | 08:59:23 | 37 s | 87.7% | Met |
| SYS-05 *(Example)* | 60 s | 09:00:00 (synchronous standby) | 0 s | 100.0% | Met |
| SYS-07 *(Example)* | 24 h | Snapshot 02:00:00, deltas replayed from SYS-02 | 7 h before replay, 37 s after | 70.8% before replay | Met |
| SYS-01 *(Example)* | n/a | Stateless | n/a | n/a | n/a |
| [SYS-NN] | [RPO] | [HH:MM:SS] | [N] | [N%] | [Met / Not met] |

SYS-02 *(Example)*: 09:00:00 minus 08:59:23 is 37 seconds;
(300 - 37) / 300 = 87.7%. SYS-07 *(Example)*: the snapshot was 7 hours
old at T0; (24 - 7) / 24 = 70.8%. Replaying deltas from the recovered
orders database reduced the effective loss to that of SYS-02.

### 7.3 Phase breakdown

| Phase | SYS-02 | SYS-01 | SYS-05 | Observation |
|---|---|---|---|---|
| Detection (T0 to alert) *(Example)* | 3 min | 3 min | 3 min | Within the 5-minute alerting objective |
| Declaration and decision *(Example)* | 11 min | 11 min | 11 min | Contact tree slowed declaration (DRT-F-05) |
| Wait for dependencies *(Example)* | 2 min | 35 min | 38 min | SYS-01 waited for SYS-02; SYS-05 waited for SYS-02 and then started late |
| Execution *(Example)* | 15 min | 7 min | 15 min | SYS-05 execution includes 8 minutes lost to the key policy (09:59 to 10:07) |
| Validation *(Example)* | 16 min | 2 min | 5 min | SYS-02 validation includes integrity checks |
| **Total** | **47 min** | **58 min** | **72 min** | Matches section 7.1 |

Totals *(Example)*: SYS-02 3 + 11 + 2 + 15 + 16 = 47; SYS-01
3 + 11 + 35 + 7 + 2 = 58; SYS-05 3 + 11 + 38 + 15 + 5 = 72. The
SYS-05 wait of 38 minutes runs from 09:14 to 09:52 and includes 5
minutes after SYS-02 was validated before the ledger runbook started,
which is a sequencing gap tracked as DRT-F-06. Removing only the 8
minutes lost to the key policy would still give 64 minutes: the RTO is
met only if both DRT-A-01 and DRT-A-05 are delivered, which is why the
retest DRT-A-06 depends on both. Removing both delays (8 + 5 minutes)
gives 72 - 13 = 59 minutes, a margin of (60 - 59) / 60 = 1.7%, which
MR-07 would still report as an Observation. The retest should therefore
also look for time to remove from the wait and validation phases.

## 8. Data integrity and functional validation

<!-- Guidance: A system is not recovered until its data is proven complete and consistent and the business confirms it works. Questions: which checks prove completeness, which prove consistency between systems, and who signed off the business functions? Pitfall: treating "the service answers health checks" as proof that the data is correct. -->

| Check ID | System | Check | Method | Expected | Observed | Result |
|---|---|---|---|---|---|---|
| INT-01 *(Example)* | SYS-02 | Row count of orders created before the last replicated commit | Count query against a checkpoint taken by the replica | 1,284,312 | 1,284,312 | Pass |
| INT-02 *(Example)* | SYS-02 | Structural consistency | Database consistency check and index validation | No errors | No errors | Pass |
| INT-03 *(Example)* | SYS-05 | Ledger balances against payment provider records for the day | Reconciliation job against provider settlement report | 2,140 matched | 2,140 matched, 0 unmatched | Pass |
| INT-04 *(Example)* | SYS-02, SYS-05 | Every paid order has a ledger entry and the reverse | Cross-system join on order ID | 0 orphans | 0 orphans | Pass |
| INT-05 *(Example)* | SYS-07 | Search returns products added after the snapshot | Query for 25 products created after 02:00 | 25 found | 25 found | Pass |
| INT-06 *(Example)* | SYS-01 | Synthetic order end to end, card payment test mode | Scripted journey from the synthetic monitoring suite | Order confirmed | Order confirmed | Pass |
| [INT-NN] | [SYS-NN] | [Check] | [Method] | [Expected] | [Observed] | [Pass / Fail] |

Business validation *(Example)*: the order operations lead placed and
cancelled two real orders with internal accounts at 10:20 and confirmed
the order appeared in the fulfilment queue. Record the name and time of
the business validator for each business service.

If any check fails, record whether the system was declared recovered
anyway and why. A recovery declared before integrity is proved must be
reported as a deviation in section 9 and as a finding in section 10.

## 9. Deviations from the plan

<!-- Guidance: Record every point where the team did something other than what the plan or runbook says, including improvisations that worked. Questions: was the runbook wrong, out of date or ignored? Did anyone use access or knowledge that would not exist in a real disaster? Pitfall: omitting successful improvisations, which then never reach the runbook. -->

| ID | Plan or runbook step | What happened instead | Reason | Effect on time or data | Linked finding |
|---|---|---|---|---|---|
| DEV-01 *(Example)* | SYS-02 runbook step 7: promote replica | Documented command refers to a tool version that was retired; engineer used the current command | Runbook not updated after a platform upgrade | About 4 min lost | DRT-F-03 |
| DEV-02 *(Example)* | SYS-05 runbook: use recovery role for key access | Break-glass role used instead | Recovery role lacked permission in region B | About 8 min lost | DRT-F-01 |
| DEV-03 *(Example)* | Plan section 5.3: notify via contact tree | Two contacts reached through a chat message after phone numbers failed | Contact list out of date | About 3 min lost | DRT-F-05 |
| DEV-04 *(Example)* | Plan section 6.3 step 2: switch DNS weighted record, time-to-live 60 s | Record switched, but the deployed record still had a 300 s time-to-live | Value changed outside infrastructure code during an earlier incident and never reverted | No effect on measured RTO (validation used the region B endpoint); some customers reached region A for up to 11 min | DRT-F-02 |
| [DEV-NN] | [Step] | [What happened] | [Reason] | [Effect] | [DRT-F-NN] |

## 10. Findings

<!-- Guidance: One row per distinct problem or observation, with severity and evidence. Use the organisation's finding scale, or the example below. Questions: what is the root cause, not just the symptom? Would this have been worse at peak load or at night? Pitfall: merging several causes into one vague finding, which then cannot be closed. -->

Severity scale *(Example)*: **High** means an objective was missed or
recovery depended on an unplanned workaround; **Medium** means an
objective was put at risk; **Low** means a documentation or minor
process gap; **Observation** means a point worth recording with no
action required now.

| Finding ID | Severity | Finding | Root cause | Evidence |
|---|---|---|---|---|
| DRT-F-01 *(Example)* | High | Payments ledger missed RTO by 12 min | Key access policy in region B did not include the recovery role; policy is managed separately from the region A policy | Timeline 09:59 to 10:07, key management audit log |
| DRT-F-02 *(Example)* | Medium | Some customers' clients kept resolving region A for up to 11 min after the DNS change | Record time-to-live of 300 s (plan 6.3 specifies 60 s; the deployed record still had 300 s, DEV-04) plus resolvers that ignore it; no client-side retry to the alternate endpoint | Synthetic monitoring from three networks, timeline 09:50 |
| DRT-F-03 *(Example)* | Low | SYS-02 runbook step 7 out of date | No runbook review step in the platform upgrade checklist | DEV-01 |
| DRT-F-04 *(Example)* | Observation | Checkout API met RTO with a 3.3% margin | Scale-out waits for the database; the warm pool in region B is minimal | Section 7.1 and 7.3 |
| DRT-F-05 *(Example)* | Medium | Declaration took 11 min after the alert | Contact list out of date; activation authority unclear for weekend shifts | DEV-03, timeline 09:03 to 09:14 |
| DRT-F-06 *(Example)* | Low | Ledger failover started 5 min after the database was ready | Runbooks are sequenced by hand in the incident channel, not by a recovery checklist with dependencies | Timeline 09:47 to 09:52 |
| [DRT-F-NN] | [Severity] | [Finding] | [Root cause] | [Evidence] |

## 11. Corrective actions

<!-- Guidance: Every High and Medium finding needs at least one action with an owner and a date. Questions: how will each action be verified, and does any fix need a retest? Pitfall: closing an action when the change is merged, instead of when a test proves it works. -->

| Action ID | Finding | Action | Owner | Due | Verification | Status |
|---|---|---|---|---|---|---|
| DRT-A-01 *(Example)* | DRT-F-01 | Manage key access policies for both regions from one definition and add a policy drift check | [@security-platform-lead] | 2026-10-17 | Drift check alerts on a seeded difference | [Open] |
| DRT-A-02 *(Example)* | DRT-F-02 | Restore the 60 s time-to-live that plan 6.3 specifies, manage the record in infrastructure code with a drift check, and add client retry to the alternate endpoint | [@edge-lead] | 2026-10-31 | Synthetic test during the retest | [Open] |
| DRT-A-03 *(Example)* | DRT-F-03 | Add "update affected runbooks" to the platform upgrade checklist; fix step 7 | [@database-lead] | 2026-10-03 | Peer review of the runbook | [Open] |
| DRT-A-04 *(Example)* | DRT-F-05 | Quarterly contact list verification; name weekend activation authority in the plan | [@dr-coordinator] | 2026-10-10 | Next IT Disaster Recovery Plan version approved | [Open] |
| DRT-A-05 *(Example)* | DRT-F-06 | Create a recovery checklist with dependency order that starts each runbook when its dependencies are validated | [@sre-lead] | 2026-11-07 | Used in the retest | [Open] |
| DRT-A-06 *(Example)* | DRT-F-01 | Targeted system test of SYS-05 failover after DRT-A-01 and DRT-A-05 | [@payments-lead] | 2026-11-14 | New DR test report shows RTO met | [Open] |
| [DRT-A-NN] | [DRT-F-NN] | [Action] | [@owner] | YYYY-MM-DD | [How proven] | [Open / Done / Verified] |

Feed every action into the improvement table of the IT Disaster
Recovery Plan (section 10.2 of `ops/disaster-recovery-plan`), and copy
plan or runbook changes into the next plan version.

### 11.1 Traceability

| Objective | Systems | Checks | Findings | Actions |
|---|---|---|---|---|
| DRT-OBJ-01 *(Example)* | SYS-01, SYS-02, SYS-05 | INT-06 | DRT-F-01, DRT-F-02, DRT-F-04, DRT-F-06 | DRT-A-01, DRT-A-02, DRT-A-05, DRT-A-06 |
| DRT-OBJ-02 *(Example)* | SYS-02, SYS-05, SYS-07 | INT-01, INT-03, INT-05 | None | None |
| DRT-OBJ-03 *(Example)* | SYS-01, SYS-02, SYS-05, SYS-07 | INT-01 to INT-06 | None | None |
| DRT-OBJ-04 *(Example)* | All | n/a | DRT-F-05 | DRT-A-04 |
| DRT-OBJ-05 *(Example)* | All | Failback validation | None | None |
| [DRT-OBJ-NN] | [SYS-NN] | [INT-NN] | [DRT-F-NN] | [DRT-A-NN] |

## 12. Evidence and control mapping

<!-- Guidance: Map the report to the requirements your auditors test, so one test produces evidence for several frameworks. Cite control and clause numbers only. Questions: which evidence item proves each requirement, and where is it stored? Pitfall: claiming a control is satisfied by a test that did not exercise it, for example claiming restore testing when only a replica was promoted. -->

| Requirement | Reference | Evidence in this report | Stored at |
|---|---|---|---|
| Contingency plan testing | NIST SP 800-53 Rev. 5 CP-4; CP-4(1) coordination with related plans; CP-4(2) alternate processing site; CP-4(4) full recovery and reconstitution | Sections 2, 6, 7, 13 | [Evidence repository path] |
| Backup reliability and integrity testing | NIST SP 800-53 Rev. 5 CP-9(1); CP-9(2) restoration from sampled backups | INT-05 snapshot restore; section 8 | [Path] |
| Recovery and reconstitution to a known state | NIST SP 800-53 Rev. 5 CP-10; CP-10(4) restore within time period | Sections 7 and 8, failback in section 6 | [Path] |
| Backups tested; integrity verified before and after restoration | NIST CSF 2.0 PR.DS-11, RC.RP-03, RC.RP-05 | INT-01 to INT-06 | [Path] |
| Improvements identified from tests and exercises | NIST CSF 2.0 ID.IM-02 | Sections 10 and 11 | [Path] |
| Exercise programme | ISO 22301:2019 clause 8.5 | Whole report, objectives in 3.3 | [Path] |
| Evaluation of business continuity capabilities | ISO 22301:2019 clause 8.6 | Sections 7 and 10 | [Path] |
| Monitoring, measurement and evaluation | ISO 22301:2019 clause 9.1 | Section 7 | [Path] |
| Nonconformity and corrective action | ISO 22301:2019 clause 10.1 | Section 11 | [Path] |
| [Requirement] | [Reference] | [Section] | [Path] |

Keep the raw evidence (logs, query output, screenshots of dashboards)
for at least as long as your audit cycle requires, and reference it by
path instead of pasting it into this report.

## 13. Conclusion and sign-off

<!-- Guidance: State the overall result using the rule agreed in section 2, the conditions attached to it, and the date of the next test. Questions: does the organisation accept the residual risk until the actions close? Pitfall: approving "Pass" when a business service objective was missed. -->

### 13.1 Overall result rule

| Result | Rule *(Example)* |
|---|---|
| Pass | All objectives met, no High finding |
| Pass with findings | Every stateful system met RPO, integrity checks passed, at most one system missed RTO (a business service that misses only because of that system counts with it) by no more than 25%, and the cause has a dated fix and a booked retest |
| Fail | Any RPO missed, any integrity check failed, more than one system missed RTO, a business service missed RTO for a reason other than one system's miss, or any miss above 25% |

*Example:* SYS-05 missed RTO by 20.0%, which is within the 25%
allowance; the business service missed only because it waits for SYS-05,
so it counts with that one miss; all RPOs and integrity checks were met, and
DRT-A-06 books the retest. The result is **Pass with findings**. Until
DRT-A-06 is verified, the service owner accepts the risk that a regional
outage would restore order taking in about 72 minutes, not 60.

### 13.2 Next test

| Item | Value |
|---|---|
| Retest of failed objectives | [DRT-YYYY-NN], YYYY-MM-DD |
| Next scheduled test from the plan | [Exercise ID from plan section 9.1], YYYY-MM-DD |
| Changes to scope for the next test | [For example, run at peak load or with region B scaled to production size] |

### 13.3 Sign-off

| Role | Name | Statement | Signature or approval reference | Date |
|---|---|---|---|---|
| Test director | [@name] | The test was run and measured as described | [Reference] | YYYY-MM-DD |
| Independent observer | [@name] | The measurement rules were followed; deviations are complete | [Reference] | YYYY-MM-DD |
| Service owner | [@name] | I accept the result and the residual risk until the actions close | [Reference] | YYYY-MM-DD |
| Business continuity manager | [@name] | The result is recorded in the exercise programme | [Reference] | YYYY-MM-DD |

## 14. Glossary and references

<!-- Guidance: Define the terms your readers will be audited on and list the exact editions you rely on. -->

### 14.1 Glossary

| Term | Meaning in this report |
|---|---|
| T0 | The moment the disruption begins or the fault is injected; the start of every recovery clock |
| RTO (recovery time objective) | The longest acceptable time from T0 until a system or service is restored and validated |
| RPO (recovery point objective) | The longest acceptable period of data that may be lost, measured back from the time of failure |
| Data loss window | The measured period between the last transaction present after recovery and the time of failure |
| Validation | Integrity and functional checks that must pass before a system counts as recovered |
| Failback | Returning service to the primary location after recovery at the alternate location |
| Exercise | An event that validates a plan and the people who carry it out (SP 800-84 sense) |
| Test | An event that evaluates systems or components (SP 800-84 sense) |
| Deviation | Any action during the test that differed from the plan or runbook |

### 14.2 References

- NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT
  Plans and Capabilities, September 2006:
  https://csrc.nist.gov/pubs/sp/800/84/final
- NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal
  Information Systems, May 2010, updated 11 November 2010:
  https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
- NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information
  Systems and Organizations, release 5.2.0 (August 2025):
  https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final
- NIST Cybersecurity Framework (CSF) 2.0, February 2024:
  https://www.nist.gov/cyberframework
- ISO 22301:2019, Security and resilience: Business continuity
  management systems: Requirements, with Amendment 1:2024 (climate
  action changes): https://www.iso.org/standard/75106.html
- Related templates: IT Disaster Recovery Plan
  (`ops/disaster-recovery-plan`), Backup Policy (`ops/backup-policy`),
  Backup and Restore Procedure (`ops/backup-restore-procedure`),
  Resilience Tiering, HA and DR Design
  (`architecture/resilience-tiering-ha-dr-design`), Game Day Plan
  (`ops/game-day-plan`), Chaos Experiment (`ops/chaos-experiment`),
  Tabletop Exercise (`security/tabletop-exercise`), Business Continuity
  Plan (`compliance/business-continuity`), Postmortem
  (`dev/postmortem`).

## Standard and licence note

Based on NIST SP 800-84 (September 2006) for the classification of
tests and exercises, NIST SP 800-34 Rev. 1 for contingency plan testing,
and NIST SP 800-53 Rev. 5 (release 5.2.0) and NIST CSF 2.0 for control
references; NIST publications are works of the US Government and are
not subject to copyright in the United States. ISO 22301:2019 and its
Amendment 1:2024 are copyrighted by ISO: this template cites clause
numbers only, with original one-line descriptions, and reproduces no
ISO text. The industry terms walkthrough, simulation, parallel test and
full interruption test are common practice and are not owned by any
standard. All text, the worked example and the example organisation
Corvid Commerce are original and fictional. Check the current official
version of each source before relying on it. Template licence: MIT.