# LLM and Agent Incident Postmortem: [Incident title]

> Blameless postmortem for an incident involving a language-model
> application or an AI agent: harmful or wrong output, unintended tool
> actions, data exposure, safety filter failures or quality regressions.
> It extends a classic software postmortem with what makes AI incidents
> different: trace replay, non-determinism, evaluation regression
> analysis, an AI failure classification and a check of AI-specific
> notification duties. Incident handling follows the lifecycle in NIST SP
> 800-61 Rev. 3 (April 2025); the notification check points to the EU AI
> Act (Regulation (EU) 2024/1689, Article 73) and the GDPR (Articles 33
> and 34), with the regulatory report itself kept in the AI Incident
> Report (`ai-data/ai-incident-report`). Replace every `[bracketed]` value
> and every row marked *Example*. This is a documentation format, not
> legal advice. Worked examples use a billing support agent that, after a
> context change, showed some customers another account's invoice details
> and promised refunds outside policy.

<!--
Guidance: How to use this template
- Blameless means you look for conditions that made the failure
  possible, not for a person to blame. Write "the release checklist did
  not require an injection test", not "Alex forgot to test".
- Freeze evidence first: traces, prompts, model snapshot, tool versions,
  retrieved documents and evaluation results. AI incidents are hard to
  reproduce once the context or the model changes.
- Run the notification check in section 11 within the first hours, not
  at the end. Legal deadlines start when you become aware of the
  incident.
- Use the general Postmortem (`dev/postmortem`) for incidents with no AI
  component; use this one when a model, prompt, retrieval or agent action
  contributed.
-->

## Document control

<!-- Guidance: Record the exact versions in production during the incident. Many AI incidents are caused by a change nobody thought of as a release, such as a prompt edit or a provider model update. -->

| Field | Value |
|---|---|
| Document ID | [PM-AI-2026-001] |
| Version | [0.1] |
| Status | [Draft / In review / Final] |
| Owner (postmortem author) | [@author] |
| Approvers / reviewers | [@incident-commander], [@agent-owner], [@security-lead], [@privacy-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Confidential] |
| Incident ID | [INC-2026-0412] |
| Severity | [SEV1 / SEV2 / SEV3] |
| Incident window (UTC) | [YYYY-MM-DD HH:MM] to [YYYY-MM-DD HH:MM] |
| System and versions | [Application version], [model and snapshot], [prompt or context spec version], [tool catalogue version], [retrieval index version] |
| Linked records | [AI incident report], [security incident ticket], [privacy breach record] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Incident commander | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Agent or product owner | [@name] | [Decision] | YYYY-MM-DD |
| Privacy or legal lead | [@name] | [Decision on section 11] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft within 48 hours |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added replay results and evaluation gap analysis |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Final after review meeting |

## Contents

<!-- Guidance: Keep this list in step with the headings below. -->

1. [Summary](#1-summary)
2. [Impact](#2-impact)
3. [Timeline](#3-timeline)
4. [Detection](#4-detection)
5. [Response and containment](#5-response-and-containment)
6. [Trace replay and reproduction](#6-trace-replay-and-reproduction)
7. [Evaluation regression analysis](#7-evaluation-regression-analysis)
8. [Contributing factors](#8-contributing-factors)
9. [AI failure classification](#9-ai-failure-classification)
10. [Corrective and preventive actions](#10-corrective-and-preventive-actions)
11. [Regulatory and contractual notification check](#11-regulatory-and-contractual-notification-check)
12. [Lessons learned](#12-lessons-learned)
13. [Glossary and references](#13-glossary-and-references)

## 1. Summary

<!-- Guidance: Five or six sentences a busy executive can read: what happened, who was affected, how it was found, how it was stopped, the main contributing factor and the main fix. -->

*Example:* On [date], between 09:12 and 12:05 UTC, the billing support
agent showed invoice details belonging to other accounts in 37
conversations and promised refunds outside policy in 212 conversations.
A context change released at 09:05 moved retrieved past tickets into an
application-side cache of assembled context blocks keyed without the
account id, so examples from one account appeared in another account's
context, and the new ordering placed refund policy
after those examples. A customer reported the exposure through support
at 11:31; the on-call engineer disabled the past-tickets source and the
refund tool at 12:05. The context change had passed the task evaluation
suite, which contained no multi-account cases and no check on refund
promises.

## 2. Impact

<!-- Guidance: Quantify impact by type. AI incidents often have several kinds at once: wrong answers, harmful content, unauthorised actions and data exposure. Count from logs, not from memory, and say how you counted. -->

### 2.1 Users and business

| Measure | Value | How measured |
|---|---|---|
| Conversations in the incident window | *Example:* 1,240 | Session logs, 09:12 to 12:05 UTC |
| Conversations with out-of-policy refund promises | *Example:* 212 (17.1%) | Classifier over transcripts, then manual review of all flagged items |
| Unintended agent actions | *Example:* 0 refunds created; 212 refund promises made in text | Tool call logs |
| Customer remediation cost | *Example:* USD 9,800 in goodwill credits honoured | Finance report |
| Support load | *Example:* 96 follow-up tickets | Ticketing system |

### 2.2 Harmful or unsafe output

| Category | Count | Example (redacted) | Severity |
|---|---|---|---|
| Out-of-policy commitments | *Example:* 212 | "I have approved a full refund of your annual plan" | Medium |
| Disclosure of another person's data | *Example:* 37 | Invoice number, plan name and amount of another account | High |
| Abusive, dangerous or discriminatory content | *Example:* 0 | Not applicable | Not applicable |

### 2.3 Data exposure

| Item | Value |
|---|---|
| Data categories exposed | *Example:* Invoice numbers, plan names, amounts, company names; no payment card data |
| Data subjects affected | *Example:* 29 accounts whose data appeared (some appeared more than once); 37 recipients |
| Exposure channel | *Example:* Agent replies shown in the support chat; also stored in chat transcripts |
| Was data sent to a third party | *Example:* Model provider processed it under the existing data processing agreement; no new recipient |
| Containment of copies | *Example:* Affected transcripts purged from analytics exports; retention hold on originals for the investigation |

*Example:* 37 of 1,240 conversations is 3.0% of conversations in the
window.

## 3. Timeline

<!-- Guidance: Use UTC and one line per event. Include changes before the incident (deploys, prompt edits, provider model updates) and every decision during response. Mark the moment of awareness, because notification deadlines count from it. -->

| Time (UTC) | Event | Source |
|---|---|---|
| *Example:* D-2 14:00 | Context change CTX-2026-001 v0.3 approved after task suite passed | Pull request |
| *Example:* 09:05 | Context change deployed to 100% of traffic | Deploy log |
| *Example:* 09:12 | First conversation with another account's invoice details | Trace replay |
| *Example:* 10:20 | Context block cache hit rate rises from 41% to 88%; no alert configured | Metrics |
| *Example:* 11:31 | Customer reports seeing another company's invoice | Support ticket |
| *Example:* 11:40 | Support lead pages on-call; incident declared SEV2 | Paging tool |
| *Example:* 11:52 | **Awareness of possible personal data breach recorded** | Incident channel |
| *Example:* 12:05 | Past-tickets source disabled; `refund.quote` and `refund.create` removed from tool set | Feature flags |
| *Example:* 12:30 | Severity raised to SEV1 after confirming cross-account exposure | Incident channel |
| *Example:* 15:30 | Rollback to context v0.2 complete; tools restored | Deploy log |
| *Example:* D+1 10:00 | Privacy lead decision on notification (section 11) | Decision record |

## 4. Detection

<!-- Guidance: Say how the incident was found, how long it took, and which signal should have caught it earlier. Customer reports as the first signal usually mean monitoring gaps. -->

| Item | Value |
|---|---|
| First signal | *Example:* Customer report |
| Time to detect | *Example:* 2 hours 19 minutes from first bad output (09:12 to 11:31) |
| Signals that existed but did not alert | *Example:* Cache hit rate jump; rise in "refund" phrases in replies; output scanner flagged 4 replies as containing foreign invoice ids but only logged them |
| Signals that did not exist | *Example:* No per-conversation check that every invoice id in a reply belongs to the session account |

Monitoring improvements belong in LLM Observability Specification
(`ai-data/llm-observability-spec`) and Guardrails Specification
(`ai-data/guardrails-spec`); list the actions in section 10.

## 5. Response and containment

<!-- Guidance: Record what was done to stop harm, how fast, and what side effects the containment had. Agents need containment options beyond rollback: disabling tools, lowering autonomy, or a kill switch. -->

| Time to | Value (*Example*) |
|---|---|
| Acknowledge | 9 minutes (11:31 to 11:40) |
| Contain | 34 minutes after report (11:31 to 12:05) |
| Recover | 3 hours 59 minutes after report (11:31 to 15:30) |

Containment options used and their effect:

- *Example:* Disabled the past-tickets retrieval source by flag. Stopped
  new exposure immediately; answer quality dropped slightly.
- *Example:* Removed refund tools from the tool set, so the agent handed
  refund requests to staff. Raised staff queue by 40 tickets.
- *Example:* Rolled back the context version. Restored normal behaviour.

Containment options that were missing: *Example:* no single switch to
lower the agent to suggest-only mode; this is an action in section 10.
Oversight modes are designed in Human Oversight Design
(`ai-data/human-oversight-design`).

## 6. Trace replay and reproduction

<!-- Guidance: Rebuild the exact context of affected calls and replay them. Model output varies between runs, so replay many times and report a rate, not a yes or no. -->

| Item | Value |
|---|---|
| Traces preserved | *Example:* All 1,240 sessions; full context hashes and block contents for 48 sampled sessions |
| Replay environment | *Example:* Same model snapshot, same context v0.3, cache disabled and enabled |
| Sampling settings | *Example:* Production settings (temperature 0.3) |
| Runs per case | *Example:* 50 |

| Case | Condition | Reproduced | Rate |
|---|---|---|---|
| RP-01 *(Example)* | Cross-account data, cache enabled, v0.3 | 31 of 50 | 62% |
| RP-02 *(Example)* | Same, cache disabled, v0.3 | 0 of 50 | 0% |
| RP-03 *(Example)* | Out-of-policy refund promise, v0.3 | 22 of 50 | 44% |
| RP-04 *(Example)* | Out-of-policy refund promise, v0.2 | 1 of 50 | 2% |

Interpretation (*Example*): RP-01 and RP-02 show the data exposure came
from cache sharing, not from the model. RP-03 and RP-04 show the
reordering raised the out-of-policy promise rate from 2% to 44%. The 2%
baseline is itself a defect that existed before the incident. Zero of 50
does not prove a zero rate: by the rule of three it only bounds the true
rate below about 6% (3 / 50) at 95% confidence, which is why EV-REG-01
uses 500 runs (bound below about 0.6%).

Non-determinism notes:

- Record the sampling parameters, seed support, and whether the provider
  guarantees identical output for identical input. Most do not.
- Check whether the provider changed the model behind the snapshot or
  alias during the window.
- When a case does not reproduce, say so and keep it in the regression
  set anyway; a low-rate failure is still a failure at volume.

## 7. Evaluation regression analysis

<!-- Guidance: Ask why the pre-release evaluation passed. The answer is usually a missing case, a missing metric, or a threshold that hid a change. This section is unique to AI incidents and is often the most valuable. -->

| Question | Finding (*Example*) |
|---|---|
| Which evaluation gated the change | Task suite (420 cases) and injection suite (60 cases) |
| Did the failure appear in any existing case | No: every case used a single account and an empty cache |
| Which metric would have caught it | A policy-compliance metric on refund replies; a cross-account leakage check |
| Were thresholds hiding a shift | Groundedness fell from 0.94 to 0.92, inside the 0.03 tolerance |
| Was production monitoring an evaluation gap | Yes: output scanner findings were logged but not evaluated |

New regression cases (add to the LLM Evaluation Plan,
`ai-data/llm-evaluation-plan`, and the Agent Evaluation Plan,
`ai-data/agent-evaluation-plan`):

| ID | Case | Metric | Pass criterion |
|---|---|---|---|
| EV-REG-01 *(Example)* | Two accounts in parallel sessions with cache enabled | Foreign identifier in reply | Zero occurrences in 500 runs |
| EV-REG-02 *(Example)* | Refund request above policy | Out-of-policy promise rate | Below the agreed bar in 200 runs |
| EV-REG-03 *(Example)* | Context reorder of any block | Full suite rerun with cache enabled | No metric drop beyond tolerance |

## 8. Contributing factors

<!-- Guidance: List conditions, not culprits. Use several categories so you do not stop at the first technical cause. Each factor should lead to at least one action in section 10. -->

| ID | Factor | Category | Evidence |
|---|---|---|---|
| CF-01 *(Example)* | Context design placed per-account examples in an application cache block keyed without the account id | Design | Context spec v0.3, section 7 |
| CF-02 *(Example)* | Policy text moved after examples, weakening its effect on refund replies | Context ordering | RP-03, RP-04 |
| CF-03 *(Example)* | Evaluation had no multi-account or cache-enabled cases | Evaluation coverage | Section 7 |
| CF-04 *(Example)* | Output scanner findings were logged without alerting | Monitoring | Section 4 |
| CF-05 *(Example)* | Context changes were not treated as releases needing staged rollout | Process | Deploy log shows 100% at once |
| CF-06 *(Example)* | No suggest-only mode for fast containment | Oversight design | Section 5 |

## 9. AI failure classification

<!-- Guidance: Classify the incident so trends become visible across postmortems. Pick a primary class and any secondary classes. Use the same taxonomy every time; adapt this one if your organisation already has one. -->

| Code | Class | Description | This incident |
|---|---|---|---|
| AF-01 | Specification | Instructions, policy or scope were wrong or ambiguous | No |
| AF-02 | Context assembly | Wrong, missing, stale, leaked or misordered context | **Primary** *(Example)* |
| AF-03 | Retrieval | Wrong or poisoned documents retrieved, or right ones missed | Secondary *(Example)* |
| AF-04 | Model behaviour | Fabrication, policy non-compliance, refusal or tone failure with correct context | Secondary *(Example)* |
| AF-05 | Tool selection or arguments | Wrong tool, wrong arguments or wrong sequence | No |
| AF-06 | Tool or integration defect | Tool, API or permission check behaved incorrectly | No |
| AF-07 | Adversarial input | Prompt injection, jailbreak, data or memory poisoning | No |
| AF-08 | Oversight failure | Human approval or review missed or approved a harmful action | No |
| AF-09 | Evaluation gap | Pre-release tests could not detect the failure | Secondary *(Example)* |
| AF-10 | Change and configuration | Model, prompt, index or provider change not controlled | Secondary *(Example)* |

For adversarial causes, add the attack class from NIST AI 100-2 E2025
and the matching OWASP Top 10 for Agentic Applications 2026 category (for
example ASI01 Agent Goal Hijack or ASI06 Memory & Context Poisoning),
and update the Agentic AI Threat Model (`ai-data/agentic-threat-model`).

## 10. Corrective and preventive actions

<!-- Guidance: Each action has an owner, a due date and a way to verify it worked. Prefer actions that change systems over actions that ask people to be more careful. -->

| ID | Action | Addresses | Type | Owner | Due | Verification |
|---|---|---|---|---|---|---|
| CA-01 *(Example)* | Add the account id to every context block cache key; add a test that fails if per-account content enters a shared block | CF-01 | Prevent | [@agent-owner] | YYYY-MM-DD | Unit test plus EV-REG-01 |
| CA-02 *(Example)* | Restore policy block before examples; add policy-compliance metric | CF-02 | Prevent | [@prompt-owner] | YYYY-MM-DD | EV-REG-02 |
| CA-03 *(Example)* | Add multi-account and cache-enabled cases to the release gate | CF-03 | Detect | [@evaluation-lead] | YYYY-MM-DD | Gate run attached to next release |
| CA-04 *(Example)* | Alert when a reply contains an identifier that does not belong to the session account; block the reply | CF-04 | Detect and prevent | [@platform-owner] | YYYY-MM-DD | Synthetic test in production |
| CA-05 *(Example)* | Roll out context changes to 5%, then 25%, then 100%, with automated comparison | CF-05 | Prevent | [@release-owner] | YYYY-MM-DD | Next two context changes follow it |
| CA-06 *(Example)* | Add a suggest-only switch for the agent | CF-06 | Contain | [@agent-owner] | YYYY-MM-DD | Game day drill |
| CA-07 *(Example)* | Fix the 2% baseline out-of-policy promise rate seen in RP-04 | RP-04 | Prevent | [@prompt-owner] | YYYY-MM-DD | EV-REG-02 on v0.2 lineage |

Record each action in the Corrective and Preventive Action Plan if your
organisation keeps one, and link it here.

## 11. Regulatory and contractual notification check

<!-- Guidance: Decide early and record the decision with reasons, even when the answer is "no notification". The deadlines below are summaries; check the consolidated legal text and your counsel. This section is a checklist, not legal advice. -->

| Duty | Applies? | Trigger summary | Deadline summary | Decision and date | Owner |
|---|---|---|---|---|---|
| GDPR Article 33, notify the supervisory authority | *Example:* Yes | Personal data breach, unless unlikely to result in a risk to people | Without undue delay and, where feasible, within 72 hours of awareness | *Example:* Notified D+1 11:00, after the decision at D+1 10:00 | [@privacy-lead] |
| GDPR Article 34, inform affected people | *Example:* Assessed: no | Breach likely to result in a high risk to people | Without undue delay | *Example:* Not required: low sensitivity, recorded reasons | [@privacy-lead] |
| EU AI Act Article 73, serious incident (high-risk AI systems) | *Example:* No: support agent is not a high-risk system under Annex III | Serious incident as defined in Article 3(49) | Immediately once a causal link is established or reasonably likely, at the latest 15 days after awareness; shorter limits apply for death and for widespread infringement or serious critical infrastructure disruption (check the consolidated text and current application dates) | *Example:* Recorded D+1 | [@legal] |
| Sector or national rules | [Yes / No] | [Rule] | [Deadline] | [Decision] | [@legal] |
| Customer contracts and data processing agreements | [Yes / No] | [Clause] | [Deadline] | [Decision] | [@account-owner] |
| Model or tool provider terms | [Yes / No] | [Clause on misuse or security reports] | [Deadline] | [Decision] | [@vendor-owner] |

When any regulatory notification applies to the AI system, complete the
AI Incident Report (`ai-data/ai-incident-report`), which holds the
regulator-facing fields, and link it in Document control. Security
handling follows the Incident Response Plan
(`compliance/incident-response-plan`).

## 12. Lessons learned

<!-- Guidance: Write what the organisation learned, including what went well and where you were lucky. Lessons without actions are fine here; actions without lessons usually miss the point. -->

What went well (*Example*):

- Support recognised the report as a possible data exposure and paged
  within 9 minutes.
- Feature flags allowed containment without a full rollback.
- Full context hashes made exact replay possible.

What went poorly (*Example*):

- A context change was treated as a configuration tweak, not a release.
- The evaluation suite tested the model, not the assembled system with
  its cache and multiple users.
- Signals existed (cache hit rate, scanner findings) but nobody watched
  them.

Where we were lucky (*Example*):

- The refund tools required a quote that the server validated, so no
  money moved even though the agent promised refunds.
- The exposed data did not include payment details.

Share the lessons with other teams running agents; the context caching
pattern in CF-01 is likely to exist elsewhere.

## 13. Glossary and references

<!-- Guidance: Define terms for readers outside the AI team, and cite the legal and standard sources you relied on with the version you checked. -->

| Term | Meaning |
|---|---|
| Blameless postmortem | A review that looks for system conditions behind a failure rather than individual fault |
| Trace replay | Rebuilding the exact inputs of a recorded model call and running it again |
| Non-determinism | Variation in model output between runs with the same input |
| Evaluation regression | A behaviour change that pre-release tests failed to detect or allowed |
| Context block cache | Application-side cache of assembled context blocks; distinct from provider prompt caching, which reuses processing of an identical prefix and does not add content to a request |
| Awareness | The moment the organisation has reasonable certainty an incident occurred; many legal deadlines count from it |

References:

- NIST SP 800-61 Rev. 3, Incident Response Recommendations and
  Considerations for Cybersecurity Risk Management: A CSF 2.0 Community
  Profile, April 2025: https://csrc.nist.gov/pubs/sp/800/61/r3/final
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and
  Terminology of Attacks and Mitigations, March 2025:
  https://csrc.nist.gov/pubs/ai/100/2/e2025/final
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 73,
  as amended; consolidated text on EUR-Lex:
  https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
- Regulation (EU) 2016/679 (General Data Protection Regulation),
  Articles 33 and 34: https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
- OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications
  for 2026, December 2025:
  https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
- Related templates: Postmortem (`dev/postmortem`), AI Incident Report
  (`ai-data/ai-incident-report`), Context Engineering Specification
  (`ai-data/context-engineering-spec`), Agent Safety Case
  (`ai-data/agent-safety-case`).

## Standard and licence note

Based on NIST SP 800-61 Rev. 3 (April 2025) for the incident handling
lifecycle, on Regulation (EU) 2024/1689 Article 73 and Regulation (EU)
2016/679 Articles 33 and 34 for the notification check (paraphrased in
original words), and on NIST AI 100-2 E2025 for adversarial attack
classes. OWASP Top 10 for Agentic Applications 2026 IDs and short names
are cited for mapping with attribution to the OWASP GenAI Security
Project (CC BY-SA 4.0). The AI failure classification, structure and all
wording are original. Laws and their application dates change: check
the consolidated texts and the current official versions before use.
This is a documentation format, not legal advice. Template licence: MIT.