Problem Record: [PRB-NNNN] [Short problem title]
IT service management record for one problem, from detection to closure: problem statement, linked incidents, impact and priority, investigation and root cause analysis, known-error entry with its workaround, the change that removes the cause, verification that the fix worked, and closure. Structure aligned with ISO/IEC 20000-1:2018 with Amendment 1:2024 (the current edition as of October 2026), clause 8.6.3 problem management, with links to clause 8.6.1 incident management, clause 8.5.1 change management and clause 10 improvement. The practice name "problem management" and the terms "known error" and "workaround" follow common ITIL usage and are cited by name only. Replace every [bracketed] value and every row marked Example.
Document control
| Field | Value |
|---|
| Problem ID | [PRB-NNNN] Example: PRB-0042 |
| Title | [Short, specific title] Example: Checkout requests time out after database failover |
| Version | [0.1] |
| Status | [Logged / Under investigation / Known error / Fix in progress / Resolved, monitoring / Closed / Closed without fix] |
| Owner (problem owner) | [@problem-owner] |
| Problem manager | [@problem-manager] |
| Approvers/reviewers | [@service-owner], [@problem-manager] |
| Service(s) affected | [Service name and ID from the service catalogue] Example: Online checkout (SVC-CHK) |
| Priority | [P1 / P2 / P3 / P4], see section 3 |
| Detection type | [Reactive (from incidents) / Proactive (trend, supplier notice, review)] |
| Date logged | YYYY-MM-DD |
| Target date for root cause | YYYY-MM-DD |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related records | [Incidents INC-], [Known error KE-], [Change CR-], [Postmortem PM-] |
Approvals
| Role | Name | Decision | Date |
|---|
| Problem owner | [@name] | [Root cause accepted / Closure accepted] | YYYY-MM-DD |
| Service owner | [@name] | [Fix accepted / Risk of no fix accepted] | YYYY-MM-DD |
| Problem manager | [@name] | [Closure confirmed] | YYYY-MM-DD |
Revision history
| Version | Date | Author | Change summary |
|---|
| 0.1 | YYYY-MM-DD | [@name] | Problem logged |
| 0.2 | YYYY-MM-DD | [@name] | Example: Workaround verified, known error KE-0019 published |
| 0.3 | YYYY-MM-DD | [@name] | Example: Root cause confirmed in staging reproduction; fix raised as CR-2231 |
| 1.0 | YYYY-MM-DD | [@name] | Example: Fix verified over 30 days with no recurrence; problem closed |
Contents
1. Problem statement
| Item | Is | Is not (contrast) |
|---|
| What fails | Example: Checkout API returns HTTP 504 after 30 s for order submission | Example: Basket, search and order history are not affected |
| Where | Example: All three application nodes of checkout-api (CI-APP-0201) | Example: The mobile app's separate payment flow |
| When | Example: Within 2 to 20 minutes after an orders database failover or replica promotion | Example: Not during normal operation or planned restarts of checkout-api |
| Extent | Example: 4% to 100% of checkout requests during the window, until checkout-api is restarted | Example: No data loss; no duplicate orders found |
| First seen | Example: 2026-07-14 (INC-3107) | Example: Not before the database cluster topology change in June 2026 |
Problem statement (Example): After a failover of the orders database, checkout-api keeps sending requests over connections to the former primary until the connection pool is exhausted, and order submissions time out until the application is restarted.
2. Linked incidents and evidence
2.1 Linked incidents
| Incident ID | Date | Priority | Symptom summary | Duration of impact | Resolved by |
|---|
| INC-3107 (Example) | 2026-07-14 | P3 | Intermittent checkout timeouts for 4% of requests | 26 min | Restart of checkout-api node 2 |
| INC-3112 (Example) | 2026-08-03 | P2 | Checkout timeouts for about 30% of requests after maintenance failover | 22 min | Rolling restart of checkout-api |
| INC-3125 (Example) | 2026-08-21 | P1 | Checkout unavailable after unplanned failover | 41 min | Rolling restart of checkout-api (the workaround later published as KE-0019) |
| [INC-] | YYYY-MM-DD | [P] | [Symptom] | [Minutes] | [Action] |
Total customer-facing impact across linked incidents (Example): 89 minutes (26 + 22 + 41).
2.2 Evidence
| Evidence ID | Type | Location | What it shows |
|---|
| EV-01 (Example) | Application metrics | Dashboard snapshot, link | Pool "active connections" flat at maximum (50) while throughput falls to zero |
| EV-02 (Example) | Database logs | Log query reference, link | Connections still open to the former primary IP for 18 minutes after failover |
| EV-03 (Example) | Configuration | Repository commit reference | Pool validation on borrow disabled since a performance change in May 2026 |
| EV-04 (Example) | Reproduction | Staging test record TST-904 | Symptom reproduced 3 out of 3 times by forcing a failover |
3. Impact and priority
| Factor | Assessment (Example) |
|---|
| Business impact if it recurs | High: checkout is revenue-bearing; one occurrence cost about 41 minutes of sales |
| Likelihood of recurrence | High: any failover triggers it, and unplanned failovers happened twice this quarter |
| Customers or users affected | All online customers in the checkout journey |
| Service level exposure | Monthly availability target 99.9% breached in August 2026 (see ops/service-review-report) |
| Security or compliance impact | None found |
| Workaround available? | Yes, manual restart within 5 minutes of a failover (section 6) |
| Priority | Rule (Example policy) | Target for root cause | Review cadence |
|---|
| P1 | High impact and high likelihood, no workaround | 5 business days | Daily |
| P2 | High impact with workaround, or medium impact without | 15 business days | Weekly |
| P3 | Medium impact with workaround | 30 business days | Fortnightly |
| P4 | Low impact | Best effort, reviewed quarterly | Quarterly |
Selected priority: [P2] Example: P2, high impact with a working but manual workaround. Reason recorded by [@problem-manager] on YYYY-MM-DD.
4. Investigation plan and timeline
4.1 Team and hypotheses
| Role | Name |
|---|
| Problem owner | [@name] Example: Checkout platform lead |
| Investigators | [@name], [@name] Example: Database engineer, checkout developer |
| Supplier contact (if any) | [Supplier, case number] |
| ID | Hypothesis | Test | Owner | Result |
|---|
| H1 (Example) | Pool keeps stale connections after failover | Force failover in staging with pool metrics on | [@dev] | Confirmed (EV-04) |
| H2 (Example) | DNS cache in the runtime holds the old primary address | Check DNS TTL and runtime cache settings | [@sre] | Contributing: runtime cached DNS for 60 s, not 18 min |
| H3 (Example) | Database driver bug in reconnect logic | Compare with driver release notes and test newer driver | [@dev] | Rejected: newer driver shows the same behaviour |
4.2 Timeline
| Date and time (UTC) | Event | Source |
|---|
| 2026-08-21 09:12 (Example) | INC-3125 closed; problem PRB-0042 opened from three matching incidents | Service management tool |
| 2026-08-22 14:00 (Example) | Workaround documented and tested; KE-0019 published to the service desk | KE-0019 |
| 2026-08-27 11:30 (Example) | Reproduction in staging confirms H1 | TST-904 |
| 2026-09-02 10:00 (Example) | Root cause accepted by problem owner; CR-2231 raised | CR-2231 |
| 2026-09-09 02:00 (Example) | CR-2231 implemented | Change record |
| 2026-10-09 (Example) | 30-day verification period ends with no recurrence across 2 planned failovers | Section 8 |
5. Root cause analysis
5.1 Method
| Method | When to use it | Used here? |
|---|
| Five whys | Single causal chain, quick first pass | [Yes / No] Example: Yes |
| Cause and effect (fishbone) diagram | Many possible causes across people, process, technology and suppliers | [Yes / No] |
| Fault tree analysis | Several conditions must combine for the failure | [Yes / No] |
| Change analysis | The problem started after a known change | [Yes / No] Example: Yes, May 2026 pool change |
| Timeline and contrast analysis ("is / is not") | Intermittent problems | [Yes / No] Example: Yes, section 1 |
5.2 Causal chain (Example)
- Why did checkout time out? Every pooled connection was waiting on a database host that no longer accepted writes.
- Why were the connections still pointing to the old host? The pool reused open connections and did not check them before handing them out.
- Why was the check off? Validation on borrow was disabled in May 2026 to cut 2 ms of latency per request.
- Why did that change not show the risk? The change test plan covered steady-state load only; no failover test was part of the change model.
- Why was failover not in the change model? Failover testing was not a required check for configuration changes to data access layers.
5.3 Conclusion
| Item | Finding |
|---|
| Root cause | Example: Connection pool validation disabled, so connections to a demoted database host were reused until the pool was exhausted |
| Contributing factors | Example: Change model for data access configuration had no failover test; no alert on pool exhaustion; restart was the only recovery |
| Trigger | Example: Planned or unplanned database failover |
| Confidence | [Confirmed by reproduction / Probable / Unconfirmed] Example: Confirmed by reproduction (EV-04) |
| Accepted by | [@problem-owner], YYYY-MM-DD |
6. Known error and workaround
| Field | Value |
|---|
| Known error ID | [KE-NNNN] Example: KE-0019 |
| Linked problem | [PRB-NNNN] Example: PRB-0042 |
| Symptoms for matching | Example: Checkout 504 errors starting within 20 minutes after a database failover alert; pool active connections at maximum |
| Workaround | Example: 1. Confirm a failover alert in the last 30 minutes. 2. Run the rolling restart job for checkout-api from the operations console. 3. Confirm error rate below 1% within 5 minutes. 4. Link the incident to KE-0019. |
| Workaround tested by and date | [@name], YYYY-MM-DD |
| Workaround side effects | Example: In-flight checkouts on the restarting node fail; customers retry |
| Time to apply | Example: 5 minutes |
| Visible to | [Service desk / Support partners / Customers via status page] |
| Published | YYYY-MM-DD |
| Retired | YYYY-MM-DD, when the fix is verified (section 8) |
Matching rule for the service desk: link a new incident to this known error only when the matching symptoms are all present; otherwise log it as a new incident and tell the problem owner.
7. Resolution and linked changes
7.1 Options
| Option | Description | Cost or effort | Risk | Residual risk | Decision |
|---|
| OPT-1 (Example) | Re-enable validation on borrow and set a maximum connection lifetime of 10 minutes | 2 days | Low; 2 ms latency added per request | Low | Selected |
| OPT-2 (Example) | Move to the database proxy that handles failover transparently | 4 weeks | Medium; new component | Very low | Planned for next year, PRJ-118 |
| OPT-3 (Example) | Keep workaround only | None | None now | High; manual restart at each failover | Rejected |
7.2 Linked changes and other actions
| ID | Type | Description | Owner | Status | Date |
|---|
| CR-2231 (Example) | Normal change | Enable pool validation and maximum lifetime | [@dev-lead] | Implemented, closure code Successful | 2026-09-09 |
| CR-2240 (Example) | Standard change model update | Add failover test to the data access change model | [@change-manager] | Implemented | 2026-09-16 |
| ALR-077 (Example) | Monitoring action | Alert when pool active connections stay at maximum for 2 minutes | [@sre] | Done | 2026-09-12 |
| [ID] | [Type] | [Description] | [@name] | [Status] | YYYY-MM-DD |
If no fix will be made: risk accepted by [@service-owner] on YYYY-MM-DD, review date YYYY-MM-DD, known error stays published.
8. Verification and closure
| Check | Criterion (Example) | Result |
|---|
| Recurrence watch | No incident matching KE-0019 for 30 days after CR-2231 | Example: None in 30 days |
| Controlled test | Failover in staging and at least one production failover with no timeouts | Example: Two planned production failovers, 0 checkout errors above baseline |
| Monitoring in place | Pool exhaustion alert active and tested | Example: Tested 2026-09-12 |
| Known error retired | KE-0019 marked retired, service desk informed | Example: Retired 2026-10-09 |
| Records updated | CMDB attributes, runbooks and change model updated | Example: Yes |
| Closure field | Value |
|---|
| Closure code | [Resolved, root cause removed / Resolved, risk reduced / Closed without fix, risk accepted / Duplicate of PRB- / Cancelled] |
| Closed by and date | [@problem-manager], YYYY-MM-DD |
| Reopen rule | Reopen this record if a matching incident occurs within [90] days of closure |
9. Lessons and improvement actions
| Action ID | Lesson | Action | Owner | Due | Status |
|---|
| IMP-061 (Example) | Performance tuning removed a safety check without anyone noticing | Add "removes a safety check?" question to the change risk assessment | [@change-manager] | YYYY-MM-DD | Done |
| IMP-062 (Example) | Restart was the only recovery; no one had looked for a cause after the first incident | Rule: open a problem after the second matching incident within 90 days | [@problem-manager] | YYYY-MM-DD | Done |
| [IMP-] | [Lesson] | [Action] | [@name] | YYYY-MM-DD | [Status] |
10. Problem lifecycle and status rules
| Status | Meaning | Who may set it |
|---|
| Logged | Recorded, priority set, owner not yet confirmed | Anyone in the service management team |
| Under investigation | Owner assigned, investigation plan in section 4 | Problem manager |
| Known error | Cause or symptom pattern understood, tested workaround published | Problem owner |
| Fix in progress | A change or project to remove the cause is approved | Problem owner |
| Resolved, monitoring | Fix implemented, verification period running | Problem owner |
| Closed | Verification passed, known error retired | Problem manager |
| Closed without fix | Risk formally accepted; known error stays published and is reviewed | Problem manager, with service owner acceptance |
11. Standards mapping and references
| Section of this record | ISO/IEC 20000-1:2018 clause | Topic (short original label) |
|---|
| 1, 2, 3 Statement, incidents, priority | 8.6.3 Problem management | Identify problems and assess their impact |
| 4, 5 Investigation and root cause | 8.6.3 | Analyse causes and decide actions |
| 6 Known error and workaround | 8.6.3, 8.6.1 Incident management | Record known errors and make workarounds available to incident handling |
| 7 Resolution | 8.5.1 Change management | Deliver the fix through controlled change |
| 2, 7 Configuration items | 8.2.6 Configuration management | Link records to configuration items |
| 8, 9 Verification and lessons | 9.1 Monitoring and measurement, 10.2 Continual improvement | Check effectiveness and improve |
| Trend reporting | 9.4 Service reporting | Report problem trends to stakeholders |
| Reference | Version or date checked | Use |
|---|
| ISO/IEC 20000-1:2018, Information technology, Service management, Part 1: Service management system requirements, with Amd 1:2024 | Current edition as of October 2026 (no newer edition found) | Clause numbers above |
| ITIL, PeopleCert | ITIL 4 available; ITIL (Version 5) released in phases from 2026; checked 2026-10-30 | Practice name and the terms known error and workaround, by name only |
- Problem: the cause, or potential cause, of one or more incidents.
- Known error: a problem with an understood cause or symptom pattern and a documented workaround.
- Workaround: a tested way to reduce or remove the impact of an incident while the cause remains.
- Root cause: the cause whose removal prevents recurrence.
- Trigger: the event that set off the failure this time.
Standard and licence note
Structure aligned with ISO/IEC 20000-1:2018 with Amendment 1:2024, clause 8.6.3 problem management and the related clauses listed in section 11. ISO/IEC 20000-1 is a copyrighted standard: this template cites clause numbers with short original labels only; no text, table or definition is copied. Official text: ISO, purchase or licensed access: https://www.iso.org/standard/70636.html, also sold by national standards bodies. The terms problem management, known error and workaround are used as in common ITIL usage, by name only; ITIL is published by PeopleCert, which released ITIL (Version 5) in phases during 2026 while ITIL 4 remains available (https://www.peoplecert.org/news-and-announcements/itil-version-5-explained); ITIL is a registered trademark of the PeopleCert group. The priority rules, targets and worked example are illustrative, not taken from any standard. All text is original, written for this collection. Check the current edition of any standard you certify against. Template licence: MIT.