LLM and Agent Incident Postmortem Template

LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

AI 및 데이터전체NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025llm incident postmortem업데이트됨 2026. 10. 22.

에디터에서 열기.md 다운로드

사용법: Use it for a blameless review when a model, prompt, retrieval step or agent action contributed to an incident. Freeze traces and versions first and run the notification check within hours, then replay affected calls many times. The main pitfall is stopping at the model's bad output instead of asking why pre-release evaluation passed.

미리보기

LLM and Agent Incident Postmortem: [Incident title]

Blameless postmortem for an incident involving a language-model application or an AI agent: harmful or wrong output, unintended tool actions, data exposure, safety filter failures or quality regressions. It extends a classic software postmortem with what makes AI incidents different: trace replay, non-determinism, evaluation regression analysis, an AI failure classification and a check of AI-specific notification duties. Incident handling follows the lifecycle in NIST SP 800-61 Rev. 3 (April 2025); the notification check points to the EU AI Act (Regulation (EU) 2024/1689, Article 73) and the GDPR (Articles 33 and 34), with the regulatory report itself kept in the AI Incident Report (ai-data/ai-incident-report). Replace every [bracketed] value and every row marked Example. This is a documentation format, not legal advice. Worked examples use a billing support agent that, after a context change, showed some customers another account's invoice details and promised refunds outside policy.

Document control

Field	Value
Document ID	[PM-AI-2026-001]
Version	[0.1]
Status	[Draft / In review / Final]
Owner (postmortem author)	[@author]
Approvers / reviewers	[@incident-commander], [@agent-owner], [@security-lead], [@privacy-lead]
Last updated	YYYY-MM-DD
Classification	[Confidential]
Incident ID	[INC-2026-0412]
Severity	[SEV1 / SEV2 / SEV3]
Incident window (UTC)	[YYYY-MM-DD HH:MM] to [YYYY-MM-DD HH:MM]
System and versions	[Application version], [model and snapshot], [prompt or context spec version], [tool catalogue version], [retrieval index version]
Linked records	[AI incident report], [security incident ticket], [privacy breach record]

Approvals

Role	Name	Decision	Date
Incident commander	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Agent or product owner	[@name]	[Decision]	YYYY-MM-DD
Privacy or legal lead	[@name]	[Decision on section 11]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft within 48 hours
0.2	YYYY-MM-DD	[@name]	Example: Added replay results and evaluation gap analysis
1.0	YYYY-MM-DD	[@name]	Example: Final after review meeting

Summary
Impact
Timeline
Detection
Response and containment
Trace replay and reproduction
Evaluation regression analysis
Contributing factors
AI failure classification
Corrective and preventive actions
Regulatory and contractual notification check
Lessons learned
Glossary and references

1. Summary

Example: On [date], between 09:12 and 12:05 UTC, the billing support agent showed invoice details belonging to other accounts in 37 conversations and promised refunds outside policy in 212 conversations. A context change released at 09:05 moved retrieved past tickets into an application-side cache of assembled context blocks keyed without the account id, so examples from one account appeared in another account's context, and the new ordering placed refund policy after those examples. A customer reported the exposure through support at 11:31; the on-call engineer disabled the past-tickets source and the refund tool at 12:05. The context change had passed the task evaluation suite, which contained no multi-account cases and no check on refund promises.

2. Impact

2.1 Users and business

Measure	Value	How measured
Conversations in the incident window	Example: 1,240	Session logs, 09:12 to 12:05 UTC
Conversations with out-of-policy refund promises	Example: 212 (17.1%)	Classifier over transcripts, then manual review of all flagged items
Unintended agent actions	Example: 0 refunds created; 212 refund promises made in text	Tool call logs
Customer remediation cost	Example: USD 9,800 in goodwill credits honoured	Finance report
Support load	Example: 96 follow-up tickets	Ticketing system

2.2 Harmful or unsafe output

Category	Count	Example (redacted)	Severity
Out-of-policy commitments	Example: 212	"I have approved a full refund of your annual plan"	Medium
Disclosure of another person's data	Example: 37	Invoice number, plan name and amount of another account	High
Abusive, dangerous or discriminatory content	Example: 0	Not applicable	Not applicable

2.3 Data exposure

Item	Value
Data categories exposed	Example: Invoice numbers, plan names, amounts, company names; no payment card data
Data subjects affected	Example: 29 accounts whose data appeared (some appeared more than once); 37 recipients
Exposure channel	Example: Agent replies shown in the support chat; also stored in chat transcripts
Was data sent to a third party	Example: Model provider processed it under the existing data processing agreement; no new recipient
Containment of copies	Example: Affected transcripts purged from analytics exports; retention hold on originals for the investigation

Example: 37 of 1,240 conversations is 3.0% of conversations in the window.

3. Timeline

Time (UTC)	Event	Source
Example: D-2 14:00	Context change CTX-2026-001 v0.3 approved after task suite passed	Pull request
Example: 09:05	Context change deployed to 100% of traffic	Deploy log
Example: 09:12	First conversation with another account's invoice details	Trace replay
Example: 10:20	Context block cache hit rate rises from 41% to 88%; no alert configured	Metrics
Example: 11:31	Customer reports seeing another company's invoice	Support ticket
Example: 11:40	Support lead pages on-call; incident declared SEV2	Paging tool
Example: 11:52	Awareness of possible personal data breach recorded	Incident channel
Example: 12:05	Past-tickets source disabled; `refund.quote` and `refund.create` removed from tool set	Feature flags
Example: 12:30	Severity raised to SEV1 after confirming cross-account exposure	Incident channel
Example: 15:30	Rollback to context v0.2 complete; tools restored	Deploy log
Example: D+1 10:00	Privacy lead decision on notification (section 11)	Decision record

4. Detection

Item	Value
First signal	Example: Customer report
Time to detect	Example: 2 hours 19 minutes from first bad output (09:12 to 11:31)
Signals that existed but did not alert	Example: Cache hit rate jump; rise in "refund" phrases in replies; output scanner flagged 4 replies as containing foreign invoice ids but only logged them
Signals that did not exist	Example: No per-conversation check that every invoice id in a reply belongs to the session account

Monitoring improvements belong in LLM Observability Specification (ai-data/llm-observability-spec) and Guardrails Specification (ai-data/guardrails-spec); list the actions in section 10.

5. Response and containment

Time to	Value (Example)
Acknowledge	9 minutes (11:31 to 11:40)
Contain	34 minutes after report (11:31 to 12:05)
Recover	3 hours 59 minutes after report (11:31 to 15:30)

Containment options used and their effect:

Example: Disabled the past-tickets retrieval source by flag. Stopped new exposure immediately; answer quality dropped slightly.
Example: Removed refund tools from the tool set, so the agent handed refund requests to staff. Raised staff queue by 40 tickets.
Example: Rolled back the context version. Restored normal behaviour.

Containment options that were missing: Example: no single switch to lower the agent to suggest-only mode; this is an action in section 10. Oversight modes are designed in Human Oversight Design (ai-data/human-oversight-design).

6. Trace replay and reproduction

Item	Value
Traces preserved	Example: All 1,240 sessions; full context hashes and block contents for 48 sampled sessions
Replay environment	Example: Same model snapshot, same context v0.3, cache disabled and enabled
Sampling settings	Example: Production settings (temperature 0.3)
Runs per case	Example: 50

Case	Condition	Reproduced	Rate
RP-01 (Example)	Cross-account data, cache enabled, v0.3	31 of 50	62%
RP-02 (Example)	Same, cache disabled, v0.3	0 of 50	0%
RP-03 (Example)	Out-of-policy refund promise, v0.3	22 of 50	44%
RP-04 (Example)	Out-of-policy refund promise, v0.2	1 of 50	2%

Interpretation (Example): RP-01 and RP-02 show the data exposure came from cache sharing, not from the model. RP-03 and RP-04 show the reordering raised the out-of-policy promise rate from 2% to 44%. The 2% baseline is itself a defect that existed before the incident. Zero of 50 does not prove a zero rate: by the rule of three it only bounds the true rate below about 6% (3 / 50) at 95% confidence, which is why EV-REG-01 uses 500 runs (bound below about 0.6%).

Non-determinism notes:

Record the sampling parameters, seed support, and whether the provider guarantees identical output for identical input. Most do not.
Check whether the provider changed the model behind the snapshot or alias during the window.
When a case does not reproduce, say so and keep it in the regression set anyway; a low-rate failure is still a failure at volume.

7. Evaluation regression analysis

Question	Finding (Example)
Which evaluation gated the change	Task suite (420 cases) and injection suite (60 cases)
Did the failure appear in any existing case	No: every case used a single account and an empty cache
Which metric would have caught it	A policy-compliance metric on refund replies; a cross-account leakage check
Were thresholds hiding a shift	Groundedness fell from 0.94 to 0.92, inside the 0.03 tolerance
Was production monitoring an evaluation gap	Yes: output scanner findings were logged but not evaluated

New regression cases (add to the LLM Evaluation Plan, ai-data/llm-evaluation-plan, and the Agent Evaluation Plan, ai-data/agent-evaluation-plan):

ID	Case	Metric	Pass criterion
EV-REG-01 (Example)	Two accounts in parallel sessions with cache enabled	Foreign identifier in reply	Zero occurrences in 500 runs
EV-REG-02 (Example)	Refund request above policy	Out-of-policy promise rate	Below the agreed bar in 200 runs
EV-REG-03 (Example)	Context reorder of any block	Full suite rerun with cache enabled	No metric drop beyond tolerance

8. Contributing factors

ID	Factor	Category	Evidence
CF-01 (Example)	Context design placed per-account examples in an application cache block keyed without the account id	Design	Context spec v0.3, section 7
CF-02 (Example)	Policy text moved after examples, weakening its effect on refund replies	Context ordering	RP-03, RP-04
CF-03 (Example)	Evaluation had no multi-account or cache-enabled cases	Evaluation coverage	Section 7
CF-04 (Example)	Output scanner findings were logged without alerting	Monitoring	Section 4
CF-05 (Example)	Context changes were not treated as releases needing staged rollout	Process	Deploy log shows 100% at once
CF-06 (Example)	No suggest-only mode for fast containment	Oversight design	Section 5

9. AI failure classification

Code	Class	Description	This incident
AF-01	Specification	Instructions, policy or scope were wrong or ambiguous	No
AF-02	Context assembly	Wrong, missing, stale, leaked or misordered context	Primary (Example)
AF-03	Retrieval	Wrong or poisoned documents retrieved, or right ones missed	Secondary (Example)
AF-04	Model behaviour	Fabrication, policy non-compliance, refusal or tone failure with correct context	Secondary (Example)
AF-05	Tool selection or arguments	Wrong tool, wrong arguments or wrong sequence	No
AF-06	Tool or integration defect	Tool, API or permission check behaved incorrectly	No
AF-07	Adversarial input	Prompt injection, jailbreak, data or memory poisoning	No
AF-08	Oversight failure	Human approval or review missed or approved a harmful action	No
AF-09	Evaluation gap	Pre-release tests could not detect the failure	Secondary (Example)
AF-10	Change and configuration	Model, prompt, index or provider change not controlled	Secondary (Example)

For adversarial causes, add the attack class from NIST AI 100-2 E2025 and the matching OWASP Top 10 for Agentic Applications 2026 category (for example ASI01 Agent Goal Hijack or ASI06 Memory & Context Poisoning), and update the Agentic AI Threat Model (ai-data/agentic-threat-model).

10. Corrective and preventive actions

ID	Action	Addresses	Type	Owner	Due	Verification
CA-01 (Example)	Add the account id to every context block cache key; add a test that fails if per-account content enters a shared block	CF-01	Prevent	[@agent-owner]	YYYY-MM-DD	Unit test plus EV-REG-01
CA-02 (Example)	Restore policy block before examples; add policy-compliance metric	CF-02	Prevent	[@prompt-owner]	YYYY-MM-DD	EV-REG-02
CA-03 (Example)	Add multi-account and cache-enabled cases to the release gate	CF-03	Detect	[@evaluation-lead]	YYYY-MM-DD	Gate run attached to next release
CA-04 (Example)	Alert when a reply contains an identifier that does not belong to the session account; block the reply	CF-04	Detect and prevent	[@platform-owner]	YYYY-MM-DD	Synthetic test in production
CA-05 (Example)	Roll out context changes to 5%, then 25%, then 100%, with automated comparison	CF-05	Prevent	[@release-owner]	YYYY-MM-DD	Next two context changes follow it
CA-06 (Example)	Add a suggest-only switch for the agent	CF-06	Contain	[@agent-owner]	YYYY-MM-DD	Game day drill
CA-07 (Example)	Fix the 2% baseline out-of-policy promise rate seen in RP-04	RP-04	Prevent	[@prompt-owner]	YYYY-MM-DD	EV-REG-02 on v0.2 lineage

Record each action in the Corrective and Preventive Action Plan if your organisation keeps one, and link it here.

11. Regulatory and contractual notification check

Duty	Applies?	Trigger summary	Deadline summary	Decision and date	Owner
GDPR Article 33, notify the supervisory authority	Example: Yes	Personal data breach, unless unlikely to result in a risk to people	Without undue delay and, where feasible, within 72 hours of awareness	Example: Notified D+1 11:00, after the decision at D+1 10:00	[@privacy-lead]
GDPR Article 34, inform affected people	Example: Assessed: no	Breach likely to result in a high risk to people	Without undue delay	Example: Not required: low sensitivity, recorded reasons	[@privacy-lead]
EU AI Act Article 73, serious incident (high-risk AI systems)	Example: No: support agent is not a high-risk system under Annex III	Serious incident as defined in Article 3(49)	Immediately once a causal link is established or reasonably likely, at the latest 15 days after awareness; shorter limits apply for death and for widespread infringement or serious critical infrastructure disruption (check the consolidated text and current application dates)	Example: Recorded D+1	[@legal]
Sector or national rules	[Yes / No]	[Rule]	[Deadline]	[Decision]	[@legal]
Customer contracts and data processing agreements	[Yes / No]	[Clause]	[Deadline]	[Decision]	[@account-owner]
Model or tool provider terms	[Yes / No]	[Clause on misuse or security reports]	[Deadline]	[Decision]	[@vendor-owner]

When any regulatory notification applies to the AI system, complete the AI Incident Report (ai-data/ai-incident-report), which holds the regulator-facing fields, and link it in Document control. Security handling follows the Incident Response Plan (compliance/incident-response-plan).

12. Lessons learned

What went well (Example):

Support recognised the report as a possible data exposure and paged within 9 minutes.
Feature flags allowed containment without a full rollback.
Full context hashes made exact replay possible.

What went poorly (Example):

A context change was treated as a configuration tweak, not a release.
The evaluation suite tested the model, not the assembled system with its cache and multiple users.
Signals existed (cache hit rate, scanner findings) but nobody watched them.

Where we were lucky (Example):

The refund tools required a quote that the server validated, so no money moved even though the agent promised refunds.
The exposed data did not include payment details.

Share the lessons with other teams running agents; the context caching pattern in CF-01 is likely to exist elsewhere.

13. Glossary and references

Term	Meaning
Blameless postmortem	A review that looks for system conditions behind a failure rather than individual fault
Trace replay	Rebuilding the exact inputs of a recorded model call and running it again
Non-determinism	Variation in model output between runs with the same input
Evaluation regression	A behaviour change that pre-release tests failed to detect or allowed
Context block cache	Application-side cache of assembled context blocks; distinct from provider prompt caching, which reuses processing of an identical prefix and does not add content to a request
Awareness	The moment the organisation has reasonable certainty an incident occurred; many legal deadlines count from it

References:

NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, April 2025: https://csrc.nist.gov/pubs/sp/800/61/r3/final
NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, March 2025: https://csrc.nist.gov/pubs/ai/100/2/e2025/final
Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 73, as amended; consolidated text on EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 33 and 34: https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications for 2026, December 2025: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
Related templates: Postmortem (dev/postmortem), AI Incident Report (ai-data/ai-incident-report), Context Engineering Specification (ai-data/context-engineering-spec), Agent Safety Case (ai-data/agent-safety-case).

Standard and licence note

Based on NIST SP 800-61 Rev. 3 (April 2025) for the incident handling lifecycle, on Regulation (EU) 2024/1689 Article 73 and Regulation (EU) 2016/679 Articles 33 and 34 for the notification check (paraphrased in original words), and on NIST AI 100-2 E2025 for adversarial attack classes. OWASP Top 10 for Agentic Applications 2026 IDs and short names are cited for mapping with attribution to the OWASP GenAI Security Project (CC BY-SA 4.0). The AI failure classification, structure and all wording are original. Laws and their application dates change: check the consolidated texts and the current official versions before use. This is a documentation format, not legal advice. Template licence: MIT.