# Problem Record: [PRB-NNNN] [Short problem title]

> IT service management record for one problem, from detection to closure: problem statement, linked incidents, impact and priority, investigation and root cause analysis, known-error entry with its workaround, the change that removes the cause, verification that the fix worked, and closure. Structure aligned with ISO/IEC 20000-1:2018 with Amendment 1:2024 (the current edition as of October 2026), clause 8.6.3 problem management, with links to clause 8.6.1 incident management, clause 8.5.1 change management and clause 10 improvement. The practice name "problem management" and the terms "known error" and "workaround" follow common ITIL usage and are cited by name only. Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- One record per problem. A problem is the unknown underlying cause of one or more incidents (reactive) or of incidents that have not happened yet (proactive). An incident is restored; a problem is understood and removed.
- Open a problem record when an incident recurs, when a major incident is closed without a confirmed cause, when monitoring or trend analysis shows a pattern, or when a supplier announces a defect that affects your services.
- Fill sections 1 to 3 at logging. Sections 4 to 6 grow during investigation. Publish the known error (section 6) as soon as a workaround exists, even if the root cause is still open: the service desk needs it now, not at closure.
- Write facts, evidence and causes, never blame. A cause such as "human error" is where the analysis starts, not where it ends.
- Related templates: Incident Management Procedure (`ops/incident-management-procedure`) for the incidents linked here, Change Request (`ops/change-request`) for the fix, Postmortem (`dev/postmortem`) for the narrative review of a major incident, Root Cause Analysis (`business/root-cause-analysis`) for non-IT problems, CMDB Data Model (`ops/cmdb-data-model`) for configuration item IDs, Service Review Report (`ops/service-review-report`) where problem trends are reported.
- Example organisation: Corvid Commerce, a fictional online retailer, investigating repeated checkout timeouts. Delete every *Example* row before use.
-->

## Document control

<!-- Guidance: The problem ID links incidents, known errors and changes in the service management tool. Keep status and owner current: an unowned problem is a closed problem in practice. -->

| Field | Value |
|---|---|
| Problem ID | [PRB-NNNN] *Example:* PRB-0042 |
| Title | [Short, specific title] *Example:* Checkout requests time out after database failover |
| Version | [0.1] |
| Status | [Logged / Under investigation / Known error / Fix in progress / Resolved, monitoring / Closed / Closed without fix] |
| Owner (problem owner) | [@problem-owner] |
| Problem manager | [@problem-manager] |
| Approvers/reviewers | [@service-owner], [@problem-manager] |
| Service(s) affected | [Service name and ID from the service catalogue] *Example:* Online checkout (SVC-CHK) |
| Priority | [P1 / P2 / P3 / P4], see section 3 |
| Detection type | [Reactive (from incidents) / Proactive (trend, supplier notice, review)] |
| Date logged | YYYY-MM-DD |
| Target date for root cause | YYYY-MM-DD |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related records | [Incidents INC-], [Known error KE-], [Change CR-], [Postmortem PM-] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Problem owner | [@name] | [Root cause accepted / Closure accepted] | YYYY-MM-DD |
| Service owner | [@name] | [Fix accepted / Risk of no fix accepted] | YYYY-MM-DD |
| Problem manager | [@name] | [Closure confirmed] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Problem logged |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Workaround verified, known error KE-0019 published |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Root cause confirmed in staging reproduction; fix raised as CR-2231 |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Fix verified over 30 days with no recurrence; problem closed |

## Contents

<!-- Guidance: Keep aligned with the headings below. -->

1. [Problem statement](#1-problem-statement)
2. [Linked incidents and evidence](#2-linked-incidents-and-evidence)
3. [Impact and priority](#3-impact-and-priority)
4. [Investigation plan and timeline](#4-investigation-plan-and-timeline)
5. [Root cause analysis](#5-root-cause-analysis)
6. [Known error and workaround](#6-known-error-and-workaround)
7. [Resolution and linked changes](#7-resolution-and-linked-changes)
8. [Verification and closure](#8-verification-and-closure)
9. [Lessons and improvement actions](#9-lessons-and-improvement-actions)
10. [Problem lifecycle and status rules](#10-problem-lifecycle-and-status-rules)
11. [Standards mapping and references](#11-standards-mapping-and-references)

## 1. Problem statement

<!-- Guidance: Describe the symptom precisely enough that two people would recognise the same problem. Use the "what, where, when, extent" frame and include what is NOT affected: the contrast narrows the search. Pitfall: writing the suspected cause as the problem statement, which stops the investigation early. -->

| Item | Is | Is not (contrast) |
|---|---|---|
| What fails | *Example:* Checkout API returns HTTP 504 after 30 s for order submission | *Example:* Basket, search and order history are not affected |
| Where | *Example:* All three application nodes of checkout-api (CI-APP-0201) | *Example:* The mobile app's separate payment flow |
| When | *Example:* Within 2 to 20 minutes after an orders database failover or replica promotion | *Example:* Not during normal operation or planned restarts of checkout-api |
| Extent | *Example:* 4% to 100% of checkout requests during the window, until checkout-api is restarted | *Example:* No data loss; no duplicate orders found |
| First seen | *Example:* 2026-07-14 (INC-3107) | *Example:* Not before the database cluster topology change in June 2026 |

Problem statement *(Example)*: After a failover of the orders database, checkout-api keeps sending requests over connections to the former primary until the connection pool is exhausted, and order submissions time out until the application is restarted.

## 2. Linked incidents and evidence

<!-- Guidance: Link every incident that shares the symptom, including ones resolved with a workaround. Attach evidence references, not screenshots pasted into the record. Questions: are there incidents classified under a different category that are really this problem? -->

### 2.1 Linked incidents

| Incident ID | Date | Priority | Symptom summary | Duration of impact | Resolved by |
|---|---|---|---|---|---|
| INC-3107 *(Example)* | 2026-07-14 | P3 | Intermittent checkout timeouts for 4% of requests | 26 min | Restart of checkout-api node 2 |
| INC-3112 *(Example)* | 2026-08-03 | P2 | Checkout timeouts for about 30% of requests after maintenance failover | 22 min | Rolling restart of checkout-api |
| INC-3125 *(Example)* | 2026-08-21 | P1 | Checkout unavailable after unplanned failover | 41 min | Rolling restart of checkout-api (the workaround later published as KE-0019) |
| [INC-] | YYYY-MM-DD | [P] | [Symptom] | [Minutes] | [Action] |

Total customer-facing impact across linked incidents *(Example)*: 89 minutes (26 + 22 + 41).

### 2.2 Evidence

| Evidence ID | Type | Location | What it shows |
|---|---|---|---|
| EV-01 *(Example)* | Application metrics | Dashboard snapshot, link | Pool "active connections" flat at maximum (50) while throughput falls to zero |
| EV-02 *(Example)* | Database logs | Log query reference, link | Connections still open to the former primary IP for 18 minutes after failover |
| EV-03 *(Example)* | Configuration | Repository commit reference | Pool validation on borrow disabled since a performance change in May 2026 |
| EV-04 *(Example)* | Reproduction | Staging test record TST-904 | Symptom reproduced 3 out of 3 times by forcing a failover |

## 3. Impact and priority

<!-- Guidance: Priority decides how fast the problem is worked on. Assess impact if the problem recurs, not only what has happened so far. Use the same impact and urgency scales as the incident procedure so that the service desk and problem team speak one language. -->

| Factor | Assessment *(Example)* |
|---|---|
| Business impact if it recurs | High: checkout is revenue-bearing; one occurrence cost about 41 minutes of sales |
| Likelihood of recurrence | High: any failover triggers it, and unplanned failovers happened twice this quarter |
| Customers or users affected | All online customers in the checkout journey |
| Service level exposure | Monthly availability target 99.9% breached in August 2026 (see `ops/service-review-report`) |
| Security or compliance impact | None found |
| Workaround available? | Yes, manual restart within 5 minutes of a failover (section 6) |

| Priority | Rule *(Example policy)* | Target for root cause | Review cadence |
|---|---|---|---|
| P1 | High impact and high likelihood, no workaround | 5 business days | Daily |
| P2 | High impact with workaround, or medium impact without | 15 business days | Weekly |
| P3 | Medium impact with workaround | 30 business days | Fortnightly |
| P4 | Low impact | Best effort, reviewed quarterly | Quarterly |

Selected priority: [P2] *Example:* P2, high impact with a working but manual workaround. Reason recorded by [@problem-manager] on YYYY-MM-DD.

## 4. Investigation plan and timeline

<!-- Guidance: Name the hypotheses, the test for each and who runs it. A problem without a plan drifts into the backlog. Keep the timeline factual and in one time zone. Pitfall: investigating only in production logs when a safe reproduction would answer the question in an hour. -->

### 4.1 Team and hypotheses

| Role | Name |
|---|---|
| Problem owner | [@name] *Example:* Checkout platform lead |
| Investigators | [@name], [@name] *Example:* Database engineer, checkout developer |
| Supplier contact (if any) | [Supplier, case number] |

| ID | Hypothesis | Test | Owner | Result |
|---|---|---|---|---|
| H1 *(Example)* | Pool keeps stale connections after failover | Force failover in staging with pool metrics on | [@dev] | Confirmed (EV-04) |
| H2 *(Example)* | DNS cache in the runtime holds the old primary address | Check DNS TTL and runtime cache settings | [@sre] | Contributing: runtime cached DNS for 60 s, not 18 min |
| H3 *(Example)* | Database driver bug in reconnect logic | Compare with driver release notes and test newer driver | [@dev] | Rejected: newer driver shows the same behaviour |

### 4.2 Timeline

| Date and time (UTC) | Event | Source |
|---|---|---|
| 2026-08-21 09:12 *(Example)* | INC-3125 closed; problem PRB-0042 opened from three matching incidents | Service management tool |
| 2026-08-22 14:00 *(Example)* | Workaround documented and tested; KE-0019 published to the service desk | KE-0019 |
| 2026-08-27 11:30 *(Example)* | Reproduction in staging confirms H1 | TST-904 |
| 2026-09-02 10:00 *(Example)* | Root cause accepted by problem owner; CR-2231 raised | CR-2231 |
| 2026-09-09 02:00 *(Example)* | CR-2231 implemented | Change record |
| 2026-10-09 *(Example)* | 30-day verification period ends with no recurrence across 2 planned failovers | Section 8 |

## 5. Root cause analysis

<!-- Guidance: Pick a method that fits the problem and record the reasoning, not only the answer. Separate the root cause (removing it prevents recurrence) from contributing factors (they made it worse or more likely) and from the trigger (what set it off this time). Pitfall: stopping at the first technical cause and missing the process gap that let it reach production. -->

### 5.1 Method

| Method | When to use it | Used here? |
|---|---|---|
| Five whys | Single causal chain, quick first pass | [Yes / No] *Example:* Yes |
| Cause and effect (fishbone) diagram | Many possible causes across people, process, technology and suppliers | [Yes / No] |
| Fault tree analysis | Several conditions must combine for the failure | [Yes / No] |
| Change analysis | The problem started after a known change | [Yes / No] *Example:* Yes, May 2026 pool change |
| Timeline and contrast analysis ("is / is not") | Intermittent problems | [Yes / No] *Example:* Yes, section 1 |

### 5.2 Causal chain *(Example)*

1. Why did checkout time out? Every pooled connection was waiting on a database host that no longer accepted writes.
2. Why were the connections still pointing to the old host? The pool reused open connections and did not check them before handing them out.
3. Why was the check off? Validation on borrow was disabled in May 2026 to cut 2 ms of latency per request.
4. Why did that change not show the risk? The change test plan covered steady-state load only; no failover test was part of the change model.
5. Why was failover not in the change model? Failover testing was not a required check for configuration changes to data access layers.

```mermaid
flowchart LR
  T["Trigger:<br/>database failover"] --> S["Stale connections<br/>to former primary"]
  C1["Root cause:<br/>pool validation disabled"] --> S
  C2["Contributing:<br/>no failover test in change model"] --> C1
  C3["Contributing:<br/>no pool exhaustion alert"] --> D["Slow detection"]
  S --> P["Pool exhausted"]
  P --> I["Checkout timeouts<br/>INC-3107, INC-3112, INC-3125"]
  D --> I
```

### 5.3 Conclusion

| Item | Finding |
|---|---|
| Root cause | *Example:* Connection pool validation disabled, so connections to a demoted database host were reused until the pool was exhausted |
| Contributing factors | *Example:* Change model for data access configuration had no failover test; no alert on pool exhaustion; restart was the only recovery |
| Trigger | *Example:* Planned or unplanned database failover |
| Confidence | [Confirmed by reproduction / Probable / Unconfirmed] *Example:* Confirmed by reproduction (EV-04) |
| Accepted by | [@problem-owner], YYYY-MM-DD |

## 6. Known error and workaround

<!-- Guidance: A known error is a problem whose cause or symptom pattern is understood well enough to give the service desk a reliable workaround. Publish it in the known error database the moment the workaround is tested; do not wait for the root cause. Write the workaround as steps a first-line analyst can follow. -->

| Field | Value |
|---|---|
| Known error ID | [KE-NNNN] *Example:* KE-0019 |
| Linked problem | [PRB-NNNN] *Example:* PRB-0042 |
| Symptoms for matching | *Example:* Checkout 504 errors starting within 20 minutes after a database failover alert; pool active connections at maximum |
| Workaround | *Example:* 1. Confirm a failover alert in the last 30 minutes. 2. Run the rolling restart job for checkout-api from the operations console. 3. Confirm error rate below 1% within 5 minutes. 4. Link the incident to KE-0019. |
| Workaround tested by and date | [@name], YYYY-MM-DD |
| Workaround side effects | *Example:* In-flight checkouts on the restarting node fail; customers retry |
| Time to apply | *Example:* 5 minutes |
| Visible to | [Service desk / Support partners / Customers via status page] |
| Published | YYYY-MM-DD |
| Retired | YYYY-MM-DD, when the fix is verified (section 8) |

Matching rule for the service desk: link a new incident to this known error only when the matching symptoms are all present; otherwise log it as a new incident and tell the problem owner.

## 7. Resolution and linked changes

<!-- Guidance: The fix is delivered through change management, never as an unrecorded hotfix. Compare options with cost, risk and residual risk. If the decision is not to fix, record who accepted the risk and when it will be reviewed. -->

### 7.1 Options

| Option | Description | Cost or effort | Risk | Residual risk | Decision |
|---|---|---|---|---|---|
| OPT-1 *(Example)* | Re-enable validation on borrow and set a maximum connection lifetime of 10 minutes | 2 days | Low; 2 ms latency added per request | Low | Selected |
| OPT-2 *(Example)* | Move to the database proxy that handles failover transparently | 4 weeks | Medium; new component | Very low | Planned for next year, PRJ-118 |
| OPT-3 *(Example)* | Keep workaround only | None | None now | High; manual restart at each failover | Rejected |

### 7.2 Linked changes and other actions

| ID | Type | Description | Owner | Status | Date |
|---|---|---|---|---|---|
| CR-2231 *(Example)* | Normal change | Enable pool validation and maximum lifetime | [@dev-lead] | Implemented, closure code Successful | 2026-09-09 |
| CR-2240 *(Example)* | Standard change model update | Add failover test to the data access change model | [@change-manager] | Implemented | 2026-09-16 |
| ALR-077 *(Example)* | Monitoring action | Alert when pool active connections stay at maximum for 2 minutes | [@sre] | Done | 2026-09-12 |
| [ID] | [Type] | [Description] | [@name] | [Status] | YYYY-MM-DD |

If no fix will be made: risk accepted by [@service-owner] on YYYY-MM-DD, review date YYYY-MM-DD, known error stays published.

## 8. Verification and closure

<!-- Guidance: Close only when evidence shows the cause is gone, not when the change is deployed. Set the verification period and test in advance. Pitfall: closing on deployment day and reopening a month later as a new problem with no history. -->

| Check | Criterion *(Example)* | Result |
|---|---|---|
| Recurrence watch | No incident matching KE-0019 for 30 days after CR-2231 | *Example:* None in 30 days |
| Controlled test | Failover in staging and at least one production failover with no timeouts | *Example:* Two planned production failovers, 0 checkout errors above baseline |
| Monitoring in place | Pool exhaustion alert active and tested | *Example:* Tested 2026-09-12 |
| Known error retired | KE-0019 marked retired, service desk informed | *Example:* Retired 2026-10-09 |
| Records updated | CMDB attributes, runbooks and change model updated | *Example:* Yes |

| Closure field | Value |
|---|---|
| Closure code | [Resolved, root cause removed / Resolved, risk reduced / Closed without fix, risk accepted / Duplicate of PRB- / Cancelled] |
| Closed by and date | [@problem-manager], YYYY-MM-DD |
| Reopen rule | Reopen this record if a matching incident occurs within [90] days of closure |

## 9. Lessons and improvement actions

<!-- Guidance: Capture what the organisation learns beyond this fix: process gaps, detection gaps and knowledge gaps. Each action needs an owner and a date and goes into the improvement register. -->

| Action ID | Lesson | Action | Owner | Due | Status |
|---|---|---|---|---|---|
| IMP-061 *(Example)* | Performance tuning removed a safety check without anyone noticing | Add "removes a safety check?" question to the change risk assessment | [@change-manager] | YYYY-MM-DD | Done |
| IMP-062 *(Example)* | Restart was the only recovery; no one had looked for a cause after the first incident | Rule: open a problem after the second matching incident within 90 days | [@problem-manager] | YYYY-MM-DD | Done |
| [IMP-] | [Lesson] | [Action] | [@name] | YYYY-MM-DD | [Status] |

## 10. Problem lifecycle and status rules

<!-- Guidance: Use these statuses consistently across all problem records so that reports and audits can count them. Adapt the names to your tool, but keep one meaning per status. -->

```mermaid
stateDiagram-v2
  [*] --> Logged
  Logged --> UnderInvestigation: owner assigned
  UnderInvestigation --> KnownError: workaround tested
  UnderInvestigation --> FixInProgress: cause found, no workaround needed
  KnownError --> FixInProgress: fix approved as a change
  KnownError --> ClosedWithoutFix: risk accepted
  FixInProgress --> ResolvedMonitoring: change implemented
  ResolvedMonitoring --> Closed: verification passed
  ResolvedMonitoring --> UnderInvestigation: recurrence
  Closed --> [*]
  ClosedWithoutFix --> [*]
```

| Status | Meaning | Who may set it |
|---|---|---|
| Logged | Recorded, priority set, owner not yet confirmed | Anyone in the service management team |
| Under investigation | Owner assigned, investigation plan in section 4 | Problem manager |
| Known error | Cause or symptom pattern understood, tested workaround published | Problem owner |
| Fix in progress | A change or project to remove the cause is approved | Problem owner |
| Resolved, monitoring | Fix implemented, verification period running | Problem owner |
| Closed | Verification passed, known error retired | Problem manager |
| Closed without fix | Risk formally accepted; known error stays published and is reviewed | Problem manager, with service owner acceptance |

## 11. Standards mapping and references

<!-- Guidance: Use this mapping when the problem process is audited. Clause numbers refer to ISO/IEC 20000-1:2018; check them against the edition you are certified to. -->

| Section of this record | ISO/IEC 20000-1:2018 clause | Topic (short original label) |
|---|---|---|
| 1, 2, 3 Statement, incidents, priority | 8.6.3 Problem management | Identify problems and assess their impact |
| 4, 5 Investigation and root cause | 8.6.3 | Analyse causes and decide actions |
| 6 Known error and workaround | 8.6.3, 8.6.1 Incident management | Record known errors and make workarounds available to incident handling |
| 7 Resolution | 8.5.1 Change management | Deliver the fix through controlled change |
| 2, 7 Configuration items | 8.2.6 Configuration management | Link records to configuration items |
| 8, 9 Verification and lessons | 9.1 Monitoring and measurement, 10.2 Continual improvement | Check effectiveness and improve |
| Trend reporting | 9.4 Service reporting | Report problem trends to stakeholders |

| Reference | Version or date checked | Use |
|---|---|---|
| ISO/IEC 20000-1:2018, Information technology, Service management, Part 1: Service management system requirements, with Amd 1:2024 | Current edition as of October 2026 (no newer edition found) | Clause numbers above |
| ITIL, PeopleCert | ITIL 4 available; ITIL (Version 5) released in phases from 2026; checked 2026-10-30 | Practice name and the terms known error and workaround, by name only |

Glossary:

- **Problem**: the cause, or potential cause, of one or more incidents.
- **Known error**: a problem with an understood cause or symptom pattern and a documented workaround.
- **Workaround**: a tested way to reduce or remove the impact of an incident while the cause remains.
- **Root cause**: the cause whose removal prevents recurrence.
- **Trigger**: the event that set off the failure this time.

## Standard and licence note

Structure aligned with ISO/IEC 20000-1:2018 with Amendment 1:2024, clause 8.6.3 problem management and the related clauses listed in section 11. ISO/IEC 20000-1 is a copyrighted standard: this template cites clause numbers with short original labels only; no text, table or definition is copied. Official text: ISO, purchase or licensed access: https://www.iso.org/standard/70636.html, also sold by national standards bodies. The terms problem management, known error and workaround are used as in common ITIL usage, by name only; ITIL is published by PeopleCert, which released ITIL (Version 5) in phases during 2026 while ITIL 4 remains available (https://www.peoplecert.org/news-and-announcements/itil-version-5-explained); ITIL is a registered trademark of the PeopleCert group. The priority rules, targets and worked example are illustrative, not taken from any standard. All text is original, written for this collection. Check the current edition of any standard you certify against. Template licence: MIT.