Agent Safety Case Template (GSN, OWASP Agentic)

Agent safety case: autonomy, hazards, GSN-style claims, arguments, evidence, defeaters, residual risk, OWASP ASI and Singapore MGF mapping

IA y datosCompletaGSN Community Standard Version 3 (SCSC-141C, May 2021); OWASP Top 10 for Agentic Applications 2026; IMDA MGF for Agentic AICC BY-SA 4.0agent safety caseActualizada 22 oct 2026

Abrir en el editorDescargar .md

Cómo usar: Use it before raising an agent's autonomy or launching it in production, to argue with evidence that it is acceptably safe in a bounded envelope. Fill the autonomy level, operating envelope and hazards first, then the top claim and sub-claims. The main pitfall is a case with no open challenges, which usually means nobody looked for defeaters.

Vista previa

Agent Safety Case: [Agent name] [version]

Structured argument, supported by evidence, that an AI agent is acceptably safe and secure to operate at a stated autonomy level in a stated environment. It uses the claims, arguments and evidence pattern with element types and IDs in the style of the Goal Structuring Notation (GSN) Community Standard Version 3 (SCSC-141C, May 2021): goals (claims), strategies (arguments), solutions (evidence), context, assumptions, justifications, and challenges that can defeat a claim. It records the autonomy level, hazards, the top claim and sub-claims, the evidence from evaluations, red teaming and monitoring, open defeaters and residual risk, and maps the argument to the OWASP Top 10 for Agentic Applications 2026 and to the four dimensions of Singapore's Model AI Governance Framework for Agentic AI (IMDA). Replace every [bracketed] value and every row marked Example. Worked examples use an IT operations agent that reads incident tickets, queries logs and runs pre-approved remediation runbooks.

Document control

Field	Value
Document ID	[ASC-2026-001]
Version	[0.1]
Status	[Draft / In review / Accepted / Accepted with conditions / Withdrawn]
Owner	[@safety-case-owner]
Approvers / reviewers	[@agent-owner], [@security-lead], [@ai-governance], [@independent-reviewer]
Last updated	YYYY-MM-DD
Classification	[Confidential]
Agent and build	[Name], [version or commit]
Model(s) and snapshot	[Provider, model, snapshot date]
Tool set version	[Tool catalogue ID and version]
Autonomy level argued	[Level from section 2]
Notation	GSN Community Standard Version 3 element types (textual form)
Next review	YYYY-MM-DD or on any trigger in section 12

Approvals

Role	Name	Decision	Date
Agent owner	[@name]	[Accepted / Accepted with conditions / Rejected]	YYYY-MM-DD
Security lead	[@name]	[Decision]	YYYY-MM-DD
AI governance lead	[@name]	[Decision]	YYYY-MM-DD
Independent reviewer	[@name]	[Decision and review report link]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Added challenge CH-02 after red team finding RT-014
1.0	YYYY-MM-DD	[@name]	Example: Accepted with conditions for autonomy level L2

Scope and system description
Autonomy level and operating envelope
Hazards
Argument notation
Top claim and context
Sub-claims and arguments
Evidence register
Defeaters and challenges
Residual risk and confidence
Mapping to OWASP Top 10 for Agentic Applications
Mapping to the Singapore agentic AI framework
Maintenance and review triggers
Glossary and references

1. Scope and system description

Item	Value
Purpose	Example: Reduce time to mitigate routine production incidents by triaging tickets and running pre-approved runbooks
Users and principals	Example: On-call engineers; the agent acts with a dedicated service identity
Environment	Example: Production Kubernetes clusters in two regions; read access to logs and metrics; write access only through the runbook executor
Tools	Example: `ticket.read`, `logs.query`, `metrics.query`, `runbook.execute` (12 approved runbooks), `ticket.comment`
Other agents	Example: A log summarisation sub-agent with read-only tools
Not in scope	Example: Database schema changes, network policy changes, customer communication
Specification	AI Agent Specification (`ai-data/ai-agent-spec`) [ID and version]

2. Autonomy level and operating envelope

There is no single normative autonomy scale for agents. Define the scale you use and keep it consistent with Human Oversight Design (ai-data/human-oversight-design).

Level	Meaning (Example scale)
L0	Suggests only; a human performs every action
L1	Prepares actions; a human approves each one
L2	Acts alone within an allow-list and limits; a human approves anything else
L3	Acts alone across a broad domain; humans monitor and can stop it
L4	Acts alone, sets its own sub-goals, little routine human review

Envelope dimension	Bound (Example)
Autonomy level argued	L2
Actions without approval	Restart a deployment, scale a deployment within 50% to 200% of baseline, clear a named cache
Actions needing approval	Every other runbook; any action in a region already degraded by another change
Rate	At most 5 autonomous actions per incident and 20 per hour per cluster
Data	Logs and metrics; no customer payload data in prompts
Hours	Any time; a human on call is always reachable
Stop control	Kill switch in the operations console; disables `runbook.execute` within 60 seconds

3. Hazards

ID	Hazard	Cause examples	Worst credible consequence	Severity	Threat model link
HZ-01 (Example)	Agent runs a runbook that worsens an outage	Misdiagnosis; misleading log lines	Extended outage across a region	High	ATM-T-002
HZ-02 (Example)	Agent follows instructions planted in a ticket or log	Indirect prompt injection	Unauthorised action within its permissions	High	ATM-T-001
HZ-03 (Example)	Agent loops, repeating actions	Planning error; flapping metric	Resource churn, cascading restarts	Medium	ATM-T-007
HZ-04 (Example)	Engineer approves a harmful proposal	Plausible but wrong rationale; alert fatigue	As HZ-01, with human sign-off	High	ATM-T-009
HZ-05 (Example)	Sensitive data leaks into ticket comments	Secrets in logs copied to output	Credential exposure	High	ATM-T-004
HZ-06 (Example)	Agent identity misused outside the task	Token reuse; over-broad scope	Actions outside the envelope	High	ATM-T-003

Severity scale and risk scoring follow the AI Risk Assessment (ai-data/ai-risk-assessment) [ID] so that this case and the risk register use the same language.

4. Argument notation

The case uses GSN element types in a textual form. GSN Version 3 defines core elements and two relationships, plus optional extensions; this template uses the dialectic extension to record challenges.

Prefix	GSN element	Use in this case
G	Goal	A claim about the agent that is true or false
S	Strategy	The reasoning step that breaks a goal into sub-goals
Sn	Solution	A reference to an item of evidence
C	Context	Definitions, scope or reference information a claim relies on
A	Assumption	A statement taken as true without further support in this case
J	Justification	Why a strategy or claim is acceptable
CH	Challenge (recorded with the dialectic extension's "challenges" relationship)	A counter-argument or counter-evidence that challenges another element; resolved, open or defeating

Relationships: "supported by" links a goal to strategies, sub-goals or solutions; "in context of" links an element to context, assumptions and justifications. Mark a goal as undeveloped when its support is planned but not yet written, and never present an undeveloped goal as met.

5. Top claim and context

ID	Element	Statement
G1 (Example)	Goal	The IT operations agent [version] is acceptably safe and secure to operate at autonomy level L2 within the operating envelope in section 2
C1 (Example)	Context	Operating envelope, section 2
C2 (Example)	Context	Definition of "acceptably safe": no hazard in section 3 has a residual risk above the acceptance threshold in the AI risk assessment [ID]
C3 (Example)	Context	Hazard list, section 3; threat model ATM-2026-001
A1 (Example)	Assumption	The runbook executor enforces its own allow-list independently of the agent
A2 (Example)	Assumption	An on-call engineer is reachable within 15 minutes at all times
J1 (Example)	Justification	L2 is chosen because the allow-listed runbooks are reversible within minutes and rate-limited
S1 (Example)	Strategy	Argue over each hazard: show that it is prevented, bounded, or detected and contained

6. Sub-claims and arguments

ID	Sub-claim	Strategy or argument	Hazards	Evidence	Status
G2 (Example)	Harmful actions are prevented or bounded	Actions are limited by an allow-list, rate limits and blast-radius limits enforced outside the model (S2)	HZ-01, HZ-03, HZ-06	Sn1, Sn5, Sn6	Supported
G3 (Example)	Instructions planted in tickets or logs cannot cause actions outside policy	Untrusted content is labelled as data; tool permissions do not depend on model output; injection attacks were attempted and failed (S3)	HZ-02	Sn2, Sn7	Supported with challenge CH-02
G4 (Example)	Humans keep effective control over non-allow-listed actions	Approval required in code; approvals show evidence, not only the agent's rationale; approval quality measured (S4)	HZ-04	Sn3, Sn8	Challenged (CH-01)
G5 (Example)	Failures are detected and contained before harm spreads	Monitoring alerts on anomalous action patterns; kill switch tested; loop guard (S5)	HZ-01, HZ-03	Sn4, Sn9	Supported
G6 (Example)	Sensitive data is not exposed through outputs	Secret redaction before prompts and before comments; output scanning (S6)	HZ-05	Sn10	Undeveloped (evidence due YYYY-MM-DD)
G7 (Example)	The model and tools in production are the ones evaluated	Release pipeline pins model snapshot and tool versions; drift check (S7)	All	Sn11	Supported

For each strategy, state why its sub-claims together are enough to support the parent claim (completeness), and record any gap as a challenge in section 8 rather than leaving it implicit.

7. Evidence register

ID	Evidence	Type	What it shows	Limits	Date	Link
Sn1 (Example)	Permission and allow-list test suite, 64 cases	Test	Executor rejects non-allow-listed runbooks regardless of agent request	Covers current 12 runbooks only	YYYY-MM-DD	[link]
Sn2 (Example)	Red team report RT-2026-03, 140 injection attempts	Red team	No attempt caused an action outside policy	Attack set fixed at test time; new techniques appear	YYYY-MM-DD	AI Red Team Plan and Report (`ai-data/ai-red-team`) [ID]
Sn3 (Example)	Approval quality study, 200 proposals with 20 seeded bad ones	Human factors study	Engineers rejected 17 of 20 seeded bad proposals	Small sample; study conditions differ from a 3 a.m. page	YYYY-MM-DD	[link]
Sn4 (Example)	Kill switch drill	Operational test	Tool disabled in 38 seconds in both regions	Drill in business hours	YYYY-MM-DD	[link]
Sn5 (Example)	Rate limiter configuration and test	Configuration review and test	Limits of 5 per incident and 20 per hour enforced	Does not cover executor outage	YYYY-MM-DD	[link]
Sn6 (Example)	Offline evaluation, 300 replayed incidents	Evaluation	Correct runbook chosen in 91% of in-scope cases; no allow-list violations	Replay set from the last two quarters	YYYY-MM-DD	Agent Evaluation Plan (`ai-data/agent-evaluation-plan`) [run ID]
Sn7 (Example)	Context assembly review	Design review	Tickets and logs wrapped as untrusted data	Review, not a test	YYYY-MM-DD	Context Engineering Specification (`ai-data/context-engineering-spec`)
Sn8 (Example)	Approval interface design	Design review	Proposal shows the triggering evidence and expected effect	No measured effect yet	YYYY-MM-DD	Human Oversight Design (`ai-data/human-oversight-design`)
Sn9 (Example)	Production monitoring, 30-day pilot at L1	Monitoring	No anomalous action pattern alerts unresolved beyond 1 hour	Pilot at a lower level than argued	YYYY-MM-DD	LLM Observability Specification (`ai-data/llm-observability-spec`)
Sn10 (Example)	Secret redaction tests	Test	Planned	Not yet run	Due YYYY-MM-DD	[link]
Sn11 (Example)	Release pipeline attestation	Configuration evidence	Model snapshot and tool versions pinned and checked at deploy	Provider-side changes behind a snapshot are outside our control	YYYY-MM-DD	[link]

Example: in Sn3, 17 of 20 seeded bad proposals rejected is an 85% catch rate, so 3 bad proposals (15%) were approved. With only 20 seeded cases the Wilson 95% interval for the catch rate is about 64% to 95%, so the study cannot rule out a much weaker control. That figure feeds challenge CH-01.

8. Defeaters and challenges

ID	Challenge	Attacks	Type	Response	Status
CH-01 (Example)	Engineers approved 15% of seeded bad proposals; approval may become rubber-stamping under fatigue	G4, Sn3	Counter-evidence	Approval now requires choosing the evidence line that supports the action; re-run study with 50 seeded cases at night shift	Open, blocks acceptance of the approval path (G4)
CH-02 (Example)	Red team attack set may not include multi-step injection via the summariser sub-agent	G3, Sn2	Gap in evidence	Add sub-agent channel to red team scope; sub-agent has read-only tools so impact is bounded	Open, accepted with condition
CH-03 (Example)	A1 assumes the executor allow-list is independent; it shares a config repository with the agent	A1	Assumption challenged	Moved allow-list to a separate repository with separate approvers	Resolved
CH-04 (Example)	Offline evaluation replay set excludes incidents in regions added this quarter	Sn6	Evidence limitation	Extended replay set; re-ran evaluation	Resolved

When a challenge defeats an element, mark it as defeated, show the impact on the parent claims, and decide whether the top claim still holds with a narrower envelope (for example L1 only for a region).

9. Residual risk and confidence

Hazard	Residual risk	Main remaining weakness	Accepted by	Conditions
HZ-01 (Example)	Low	Misdiagnosis on novel failure modes	[@agent-owner]	Allow-list reviewed monthly
HZ-03 (Example)	Low	Loop guard depends on correct incident grouping	[@agent-owner]	Rate limits in Sn5 stay enforced outside the model
HZ-06 (Example)	Low	Token scope drift as tools are added	[@security-lead]	Scope review on every tool catalogue change
HZ-02 (Example)	Medium	New injection techniques (CH-02)	[@security-lead]	Quarterly red team; monitoring alert on tool calls after untrusted content
HZ-04 (Example)	Medium	Approval quality (CH-01)	Not yet accepted	Close CH-01 before the agent may propose non-allow-listed runbooks
HZ-05 (Example)	Medium	G6 undeveloped	Not yet accepted	Sn10 before go-live

Overall decision (Example): accept at L2 for allow-listed actions only, on condition that Sn10 passes before go-live so that G6 is supported; until CH-01 is closed the agent does not propose non-allow-listed runbooks, and engineers run them manually. Record the confidence assessment of key argument steps: where the GSN confidence argument extension is used, attach the confidence argument at the relevant assurance claim point.

10. Mapping to OWASP Top 10 for Agentic Applications

OWASP ID	Category	Claims	Evidence	Coverage
ASI01	Agent Goal Hijack	G3	Sn2, Sn7	Partial (CH-02)
ASI02	Tool Misuse and Exploitation	G2	Sn1, Sn5	Covered
ASI03	Identity and Privilege Abuse	G2, G7	Sn1, Sn11	Covered
ASI04	Agentic Supply Chain Vulnerabilities	G7	Sn11	Partial: third-party log parser not assessed
ASI05	Unexpected Code Execution (RCE)	G2	Sn1	Covered: agent cannot run arbitrary code; runbooks are fixed
ASI06	Memory & Context Poisoning	G3	Sn7	Covered: no long-term memory in this build
ASI07	Insecure Inter-Agent Communication	G3	Sn2	Partial (CH-02)
ASI08	Cascading Failures	G5	Sn4, Sn5, Sn9	Covered
ASI09	Human-Agent Trust Exploitation	G4	Sn3, Sn8	Open (CH-01)
ASI10	Rogue Agents	G5, G7	Sn4, Sn9	Covered

Rows above are Example content apart from the OWASP IDs and category names, which follow the published list.

11. Mapping to the Singapore agentic AI framework

IMDA's Model AI Governance Framework for Agentic AI groups its considerations into four dimensions. For a full self-assessment against the framework, use the Agentic AI Governance Assessment (ai-data/agentic-ai-governance-singapore) and keep this table as the summary. The names below follow the January 2026 factsheet and are unchanged in Version 1.5 (published 20 May 2026, updated 5 June 2026); confirm against the current version.

Dimension	What it asks for (paraphrased)	Claims and evidence in this case	Gaps
Assess and bound the risks upfront	Choose suitable use cases after a risk assessment that weighs autonomy and data access; limit autonomy, tools and data by design	Sections 2 and 3; G2; Sn1, Sn5	Example: None
Make humans meaningfully accountable	Allocate responsibility across parties; design oversight that resists automation bias and audit approvals; adapt governance over time	Approvals table; G4; Sn3, Sn8; section 12	Example: CH-01 open
Implement technical controls and processes	Controls for planning, tools and protocols in design; pre-deployment safety and security testing; gradual rollout with monitoring	G3, G5, G7; Sn2, Sn4, Sn6, Sn9, Sn11	Example: CH-02 open
Enable end-user responsibility	Tell users when and how agents are used; train users to oversee agents while keeping their own skills	Example: Engineer onboarding module; agent actions labelled in tickets	Example: Training completion not yet evidenced

12. Maintenance and review triggers

Reopen the case and re-check affected evidence when any of these occur:

Model or model snapshot change, or a provider notice of behaviour change.
New tool, new runbook on the allow-list, or a raised side-effect class (see Agent Tool and Function Catalogue, ai-data/tool-catalogue).
Request to raise the autonomy level or widen the envelope.
Incident or near miss involving the agent (see LLM and Agent Incident Postmortem, ai-data/llm-incident-postmortem).
New OWASP, MITRE ATLAS or regulatory guidance relevant to an open challenge.
Evidence expiry: red team older than [6 months]; approval study older than [12 months].

Review	Frequency	Owner
Evidence currency check	Example: Monthly	[@safety-case-owner]
Full case review	Example: Every 6 months and on triggers	[@ai-governance]
Independent review	Example: Before each autonomy increase	[@independent-reviewer]

13. Glossary and references

Term	Meaning
Safety case	A structured argument, supported by evidence, that a system is acceptably safe for a given use in a given environment
Claim (goal)	A statement about the system that the argument sets out to support
Argument (strategy)	The reasoning that links a claim to its sub-claims or evidence
Evidence (solution)	Artefacts such as test results, reviews and monitoring data that support a claim
Defeater (challenge)	A reason to doubt a claim, an argument step or evidence
Operating envelope	The bounded conditions under which the claim is made
Autonomy level	How far the agent acts without per-action human approval
Assurance claim point	A marked point in a GSN argument where a confidence argument is attached

References:

Assurance Case Working Group, Goal Structuring Notation Community Standard Version 3, SCSC-141C, Safety-Critical Systems Club, May 2021: https://scsc.uk/SCSC-141C
OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications for 2026, December 2025: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
Infocomm Media Development Authority (IMDA), Factsheet: Model AI Governance Framework for Agentic AI, January 2026: https://www.imda.gov.sg/-/media/imda/files/news-and-events/media-room/media-releases/2026/01/factsheet-model-ai-governance-framework-for-agentic-ai.pdf
IMDA, Updated Model AI Governance Framework for Agentic AI, press release, 20 May 2026: https://www.imda.gov.sg/resources/press-releases-factsheets-and-speeches/updated-model-ai-governance-framework-for-agentic-ai
Related templates: Agentic AI Threat Model (ai-data/agentic-threat-model), Human Oversight Design (ai-data/human-oversight-design), Frontier AI Safety Framework (ai-data/frontier-safety-framework), AI Risk Assessment (ai-data/ai-risk-assessment), Agentic AI Governance Assessment (ai-data/agentic-ai-governance-singapore).

Standard and licence note

Structure aligned with the Goal Structuring Notation Community Standard Version 3 (SCSC-141C, May 2021): element types, relationships and the dialectic and confidence extensions are referred to by name only, and no text or diagrams from the standard are reproduced. Based on the OWASP Top 10 for Agentic Applications for 2026 by the OWASP GenAI Security Project (https://genai.owasp.org/), licensed under Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/): category IDs and short names are used for mapping, and because the mapping table in section 10 adapts the OWASP category structure, that table is shared under CC BY-SA 4.0 and any further adaptation of it must carry the same licence. The four dimensions of IMDA's Model AI Governance Framework for Agentic AI are named and paraphrased in original words. The autonomy scale, hazards, claims and examples are original. Check the current official versions of all three sources before use. Apart from the adapted OWASP structure, which stays under CC BY-SA 4.0, the original text is under the template licence. Template licence: MIT.

Plantillas relacionadas

IA y datos