# Performance Test Plan and Report: [System or release name]

> Combined performance test plan and results report. Part A plans the
> testing (objectives, workload model, test types, environment, criteria,
> scenarios); Part B records results, bottlenecks and recommendations.
> Based on common industry practice for load and performance engineering.
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Start from the service level objectives (section 2). A performance test
  without a numeric target cannot pass or fail.
- Build the workload model from production data (logs, analytics, APM),
  not from guesses. If the system is new, state the assumptions and their source.
- Keep Part A frozen once approved; fill Part B per test run. Re-baseline
  Part A when the workload model or environment changes.
- Report percentiles, not averages. An average hides the slow requests users notice.
-->

## Document control

<!-- Guidance: Identify the plan and the system version under test. -->

| Field | Value |
|---|---|
| Document ID | [PTP-SYSTEM-001] |
| System / release | [System name] [version] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Results final] |
| Owner | [@performance-engineer] |
| Classification | [Internal / Confidential] |
| Last updated | YYYY-MM-DD |
| Related documents | [Test plan], [SLO document], [Architecture description], [Capacity plan] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Performance engineer | [@name] | [Approved] | YYYY-MM-DD |
| Engineering lead | [@name] | [Approved / Rejected / Approved with comments] | YYYY-MM-DD |
| SRE / operations lead | [@name] | [Decision] | YYYY-MM-DD |
| Product owner | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Record plan changes and each results issue separately. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Plan draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Plan approved |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Results of load and stress runs added to Part B |

## Contents

<!-- Guidance: Keep in sync with headings. -->

Part A: Plan

1. [Introduction](#1-introduction)
2. [Objectives and service level targets](#2-objectives-and-service-level-targets)
3. [Workload model](#3-workload-model)
4. [Test types](#4-test-types)
5. [Test environment and data](#5-test-environment-and-data)
6. [Tools and monitoring](#6-tools-and-monitoring)
7. [Entry and exit criteria](#7-entry-and-exit-criteria)
8. [Test scenarios](#8-test-scenarios)
9. [Risks, assumptions and schedule](#9-risks-assumptions-and-schedule)

Part B: Report

10. [Results summary](#10-results-summary)
11. [Detailed results](#11-detailed-results)
12. [Bottleneck analysis](#12-bottleneck-analysis)
13. [Recommendations](#13-recommendations)

## 1. Introduction

<!-- Guidance: Describe the system, why performance testing is needed now, and what is in and out of scope. -->

### 1.1 Purpose

This document plans and reports performance testing for [system name]
[version] ahead of [launch / seasonal peak / migration], to show that the
system meets its latency, throughput and resource targets at forecast
peak load plus headroom, and to find the limits before users do.

### 1.2 Scope

| In scope | Out of scope |
|---|---|
| *Example:* Public web storefront and Orders API | Back-office reporting (batch, separate test) |
| *Example:* Checkout, search, product pages | Third-party payment provider capacity (contractual SLA) |
| *Example:* Database and cache tier | CDN edge performance (vendor-managed) |

### 1.3 System under test

[One paragraph or diagram reference describing the architecture: entry
points, services, data stores, caches, queues and external dependencies,
with the instance sizes used in production.]

## 2. Objectives and service level targets

<!-- Guidance: Each target has a metric, a percentile, a load level and a threshold. Agree them with the product owner and operations before testing. -->

| ID | Transaction or resource | Metric | Target | At load |
|---|---|---|---|---|
| SLA-01 | *Example:* Product page | Server response time p95 | <= 300 ms | Peak (100%) |
| SLA-02 | *Example:* Search | Response time p95 / p99 | <= 500 ms / <= 1,200 ms | Peak |
| SLA-03 | *Example:* Checkout submit | Response time p95 | <= 1,500 ms | Peak |
| SLA-04 | *Example:* All transactions | Error rate | < 0.5% | Peak |
| SLA-05 | *Example:* System | Sustained throughput | >= 250 requests/s | 120% of peak |
| SLA-06 | *Example:* Orders API pods | CPU utilisation | < 70% average | Peak |
| SLA-07 | *Example:* Database | Connection pool usage | < 80% | Peak |
| [SLA-nn] | [Transaction] | [Metric] | [Target] | [Load] |

Objectives beyond pass or fail:

- Find the breaking point (load at which an SLA is first violated) and the failure mode.
- Confirm the system recovers within [5 minutes] after overload ends.
- Detect memory or connection leaks over a [8-hour] soak.

## 3. Workload model

<!-- Guidance: The workload model says what virtual users do, how often and in what mix. Base it on production analytics for the peak hour you are designing for. -->

### 3.1 Source data

Peak hour analysed: [YYYY-MM-DD HH:00, from APM and web analytics].
Forecast growth to the test date: [+35%]. Headroom applied: [20%].

### 3.2 Transaction mix

| Transaction | Share of requests | Peak rate (req/s) | Think time (s) | Notes |
|---|---|---|---|---|
| *Example:* Home page | 18% | 36 | 5 | Cached at CDN for anonymous users; test bypasses CDN |
| *Example:* Search | 25% | 50 | 8 | Query terms from top 2,000 production searches |
| *Example:* Product page | 32% | 64 | 10 | 50,000 SKUs, Zipf distribution |
| *Example:* Add to cart | 12% | 24 | 6 | |
| *Example:* Checkout (3 steps) | 5% | 10 | 15 | Payment stub with 400 ms delay |
| *Example:* Login | 8% | 16 | 4 | |
| **Total** | **100%** | **200** | | |

### 3.3 Concurrency estimate

The number of concurrent users needed to produce the target throughput
follows from Little's law:

$$N = X \cdot R$$

where $N$ is the average number of users in the system, $X$ is throughput
(requests per second) and $R$ is the average time each user spends per
request cycle, including think time.

*Example:* With $X = 200$ req/s and an average cycle of response time
0.3 s plus think time 7.7 s, $R = 8$ s, so $N = 200 \times 8 = 1{,}600$
virtual users at peak. At 120% of peak, the test needs $1{,}920$ users.

### 3.4 User journeys

| Journey | Steps | Share of virtual users |
|---|---|---|
| Browser | Home, search, 3 product pages | 55% |
| Buyer | Login, search, product page, add to cart, checkout | 15% |
| Returning customer | Login, order history, product page | 20% |
| API partner | Stock query batch via API | 10% |

## 4. Test types

<!-- Guidance: Pick the types that answer your objectives. Describe the load profile of each so it can be reproduced. -->

| Test type | Purpose | Load profile | Duration | Pass criteria |
|---|---|---|---|---|
| Baseline | Reference figures at low load | 10% of peak | 15 min | Recorded, no criteria |
| Load | Verify SLAs at expected peak | Ramp to 100% over 10 min, hold | 60 min | SLA-01 to SLA-07 met |
| Stress | Find breaking point and failure mode | Increase 20% every 10 min until SLA breach or errors over 5% | Until break | Breaking point documented; graceful degradation |
| Soak (endurance) | Detect leaks and degradation over time | 80% of peak, constant | 8 h | No upward trend in memory, latency or errors |
| Spike | Check behaviour under sudden surge | 30% to 250% of peak in 30 s, hold 5 min, back to 30% | 30 min | No crash; recovery within 5 min |
| Scalability | Check autoscaling | Step load with autoscaling enabled | 45 min | New capacity serves traffic within 3 min |

```mermaid
flowchart LR
  B["Baseline"] --> L["Load"]
  L --> S["Stress"]
  L --> K["Soak"]
  L --> P["Spike"]
  S --> R["Report and tune"]
  K --> R
  P --> R
```

Tests run in the order shown; stress, soak and spike start only after the load test passes or its failures are understood.

## 5. Test environment and data

<!-- Guidance: Document every difference from production. Results only transfer to production if differences are known and accounted for. -->

| Component | Production | Test environment | Difference and effect |
|---|---|---|---|
| *Example:* Orders API | 6 pods, 2 vCPU / 4 GB | 6 pods, 2 vCPU / 4 GB | None |
| *Example:* Database | 8 vCPU primary + 2 replicas | 8 vCPU primary + 1 replica | Read capacity lower; read-heavy results conservative |
| *Example:* Cache | 3-node cluster | 3-node cluster | None |
| *Example:* Payment provider | Live | Stub with fixed 400 ms latency | Real latency variance not modelled |
| [Component] | [Production] | [Test] | [Difference] |

**Data**: database populated with [2 million orders, 500,000 customers,
50,000 SKUs] of synthetic data matching production distributions. No
personal data. Caches warmed for [10 minutes] before measured runs.

**Load generators**: [4 instances in the same region, 8 vCPU each];
generator CPU kept under 70% so the generator is never the bottleneck.

## 6. Tools and monitoring

<!-- Guidance: List load tools and every metric source. You cannot analyse a bottleneck you did not monitor. -->

| Purpose | Tool | Notes |
|---|---|---|
| Load generation | [k6 / JMeter / Gatling / Locust] | Scripts in `perf/` in the repository |
| Application metrics and traces | [APM tool] | Per-endpoint latency, traces for slow requests |
| Infrastructure metrics | [Prometheus and Grafana / cloud monitoring] | CPU, memory, network, disk per node |
| Database | [Query statistics, slow query log] | Top queries by total time |
| Logs | [Log platform] | Error rates and exceptions |

Metrics captured for every run: response time percentiles (p50, p90,
p95, p99, max) per transaction; throughput; error rate by type; CPU,
memory, garbage collection; connection pools; queue depths; database
locks and slow queries.

## 7. Entry and exit criteria

<!-- Guidance: Entry criteria protect test validity; exit criteria define when testing is done. -->

| Entry criteria | Exit criteria |
|---|---|
| Functionally stable build deployed (smoke test passes) | All planned test types executed |
| Environment matches section 5; differences approved | SLAs met in load test, or deviations accepted by product owner |
| Test data loaded and verified | Breaking point and failure mode documented |
| Monitoring dashboards in place for all components | No leak trend in soak test |
| Scripts reviewed and validated at low load | Report reviewed and approved |

## 8. Test scenarios

<!-- Guidance: One row per run you intend to execute. Run IDs link Part A to Part B. -->

| Run ID | Test type | Journeys and mix | Users / rate | Duration | Build | Status |
|---|---|---|---|---|---|---|
| PT-01 | Baseline | All, mix 3.2 | 160 users | 15 min | [build] | [Planned] |
| PT-02 | Load | All, mix 3.2 | 1,600 users | 60 min | [build] | [Planned] |
| PT-03 | Load (120%) | All, mix 3.2 | 1,920 users | 60 min | [build] | [Planned] |
| PT-04 | Stress | All, mix 3.2 | +320 users per 10 min | Until break | [build] | [Planned] |
| PT-05 | Soak | All, mix 3.2 | 1,280 users | 8 h | [build] | [Planned] |
| PT-06 | Spike | All, mix 3.2 | 480 to 4,000 users | 30 min | [build] | [Planned] |

## 9. Risks, assumptions and schedule

<!-- Guidance: List what could invalidate results and when each run happens. -->

| ID | Risk or assumption | Mitigation |
|---|---|---|
| R-01 | *Example:* Shared environment used by other teams during runs | Book exclusive windows; check for foreign traffic in dashboards |
| R-02 | *Example:* Payment stub hides provider latency spikes | Add variable delay profile in a second checkout run |
| R-03 | *Example:* Production traffic mix shifts after marketing campaign | Re-validate mix one week before launch |

Schedule: scripts ready YYYY-MM-DD; PT-01 to PT-03 YYYY-MM-DD; PT-04 to
PT-06 YYYY-MM-DD; report YYYY-MM-DD; retest window YYYY-MM-DD.

## 10. Results summary

<!-- Guidance: Part B starts here. Give the verdict per SLA first, then the evidence. -->

| SLA | Target | Result (run) | Verdict |
|---|---|---|---|
| SLA-01 Product page p95 | <= 300 ms | *Example:* 212 ms (PT-02) | Pass |
| SLA-02 Search p95 / p99 | <= 500 / 1,200 ms | *Example:* 640 / 1,850 ms (PT-02) | Fail |
| SLA-03 Checkout p95 | <= 1,500 ms | *Example:* 1,180 ms (PT-02) | Pass |
| SLA-04 Error rate | < 0.5% | *Example:* 0.12% (PT-02) | Pass |
| SLA-05 Throughput at 120% | >= 250 req/s | *Example:* 241 req/s (PT-03) | Fail |
| [SLA] | [Target] | [Result] | [Pass / Fail] |

Overall verdict: **[Pass / Pass with conditions / Fail]**. *Example:* Fail
on search latency and throughput at 120%; see 12.1. Breaking point at
about 2,250 users, where search errors exceeded 5%.

## 11. Detailed results

<!-- Guidance: One table per run. Keep raw result files and link them; summarise here. -->

### 11.1 Run PT-02 (load, 100% of peak)

| Transaction | Samples | p50 (ms) | p90 (ms) | p95 (ms) | p99 (ms) | Max (ms) | Errors |
|---|---|---|---|---|---|---|---|
| *Example:* Home page | 129,600 | 85 | 140 | 168 | 290 | 1,020 | 0.02% |
| *Example:* Search | 180,000 | 310 | 520 | 640 | 1,850 | 4,900 | 0.31% |
| *Example:* Product page | 230,400 | 120 | 180 | 212 | 410 | 1,600 | 0.05% |
| *Example:* Checkout submit | 36,000 | 690 | 1,020 | 1,180 | 1,700 | 3,300 | 0.20% |
| [Transaction] | [n] | [ms] | [ms] | [ms] | [ms] | [ms] | [%] |

| Resource | Average | Peak | Target |
|---|---|---|---|
| *Example:* Orders API CPU | 54% | 71% | < 70% average |
| *Example:* Search service CPU | 83% | 97% | < 70% average |
| *Example:* DB connections | 61% | 78% | < 80% |

Raw results: [link to result files and dashboards snapshot].

### 11.2 Other runs

| Run | Key observation | Evidence |
|---|---|---|
| PT-04 Stress | *Example:* Search degrades first; at 2,250 users p95 reaches 3 s and errors 5.4%; other transactions within SLA | [Link] |
| PT-05 Soak | *Example:* Heap stable; latency flat; DB connections stable | [Link] |
| PT-06 Spike | *Example:* Autoscaling added 4 pods in 2 min 40 s; 1.8% errors during first 60 s; full recovery in 3 min | [Link] |

## 12. Bottleneck analysis

<!-- Guidance: For each failed SLA, show the symptom, the evidence and the root cause. Separate confirmed causes from hypotheses. -->

| ID | Symptom | Evidence | Root cause | Confidence |
|---|---|---|---|---|
| BN-01 | *Example:* Search p95 640 ms, p99 1,850 ms at peak | Search CPU 97% peak; traces show 70% of time in relevance scoring | Search service under-provisioned; scoring function not cached | Confirmed |
| BN-02 | *Example:* Throughput plateaus at 241 req/s | Search thread pool saturated; queue wait 400 ms | Thread pool size 50, fixed | Confirmed |
| [BN-nn] | [Symptom] | [Evidence] | [Cause] | [Confirmed / Hypothesis] |

### 12.1 Capacity projection

With utilisation $U = X \cdot S$ (throughput times service demand), the
search service at $S = 16$ ms CPU per request on 1 vCPU saturates near
$X = 1 / 0.016 \approx 62$ req/s, and at the peak search rate of 50 req/s
it runs at $U = 50 \times 0.016 = 0.8$, which matches the 83% observed. Meeting 120% of peak search load with CPU
under 70% needs roughly $\lceil 60 \times 0.016 / 0.7 \rceil = 2$ vCPU for
search alone, so [describe the sizing change].

## 13. Recommendations

<!-- Guidance: Each recommendation names an owner, an expected effect and a retest. -->

| ID | Recommendation | Addresses | Expected effect | Owner | Retest run |
|---|---|---|---|---|---|
| REC-01 | *Example:* Cache relevance scores for top 2,000 queries for 5 minutes | BN-01 | Search p95 below 400 ms | [@search-lead] | PT-02R |
| REC-02 | *Example:* Raise search thread pool to 120 and add 2 pods | BN-02 | Throughput above 280 req/s | [@sre] | PT-03R |
| REC-03 | *Example:* Add search p99 alert at 1,000 ms in production | Monitoring gap | Earlier detection | [@sre] | Not applicable |
| [REC-nn] | [Recommendation] | [BN] | [Effect] | [@name] | [Run] |

Release recommendation: [Release / Release after REC-01 and REC-02 are
verified by retest / Do not release].

## Standard and licence note

Based on common industry practice for performance test planning and
reporting, including workload modelling and queueing relationships such
as Little's law and the utilisation law. This template is original text
written for this collection. Template licence: MIT.