# LLM and Agent Observability Specification: [System name] [version]

> Specification of the telemetry an LLM application or agent system must
> emit and how that telemetry is processed, protected and used: traces
> for model calls, tool calls, retrieval and agent steps; metrics for
> tokens, latency, cost, errors and quality; controlled capture of prompts
> and completions; evaluation hooks; dashboards, alerts and service level
> objectives. Attribute, span and metric names follow the OpenTelemetry
> semantic conventions for generative AI, which have **Development**
> status and are now maintained in the `semantic-conventions-genai`
> repository. Replace every `[bracketed]` value and every row marked
> *Example*. Worked examples use a support agent that calls a hosted chat
> model, a retrieval service and three tools.

<!--
Guidance: How to use this template
- Start from the general service observability specification
  (ops/observability-spec) for resource attributes, log schema, pipeline
  and retention. This document adds only what is specific to models,
  agents, tools and retrieval.
- The GenAI conventions have Development status: names can change
  between versions. Pin the version you follow in section 4.1, record
  the commit or release date, and review this document when it changes.
- Decide content capture (section 7) with privacy and security before
  any prompt or completion reaches the telemetry backend.
- Every alert in section 10 needs an owner and a runbook link; every SLO
  needs a measurement query that someone has run.
-->

## Document control

<!-- Guidance: Identify the system, the convention version pinned and the people who own telemetry, privacy and on-call. -->

| Field | Value |
|---|---|
| Document ID | [OBS-LLM-001] |
| System and version | [Name] [version] |
| Semantic convention baseline | *Example:* OpenTelemetry GenAI semantic conventions, `semantic-conventions-genai` main branch at commit [sha], YYYY-MM-DD |
| Telemetry backend | [Tracing, metrics and log backends] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@ai-platform-owner] |
| Approvers / reviewers | [@sre-lead], [@privacy-officer], [@security-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related documents | [Observability specification], [Guardrails specification], [Agent evaluation plan], [Model monitoring plan], [AI-BOM] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| AI platform owner | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| SRE lead | [@name] | [Decision] | YYYY-MM-DD |
| Privacy officer | [@name] | [Decision] | YYYY-MM-DD |
| Security lead | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved with content capture limited to external store |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Moved token metrics to `gen_ai.client.inference.usage.*` after convention update |

## Contents

<!-- Guidance: Keep this list aligned with the headings. -->

1. [Objectives and scope](#1-objectives-and-scope)
2. [Telemetry architecture](#2-telemetry-architecture)
3. [Signals and span model](#3-signals-and-span-model)
4. [Attributes and semantic conventions](#4-attributes-and-semantic-conventions)
5. [Metrics](#5-metrics)
6. [Logs and events](#6-logs-and-events)
7. [Prompt and completion capture](#7-prompt-and-completion-capture)
8. [Evaluation hooks](#8-evaluation-hooks)
9. [Dashboards](#9-dashboards)
10. [Alerts and service level objectives](#10-alerts-and-service-level-objectives)
11. [Cost attribution](#11-cost-attribution)
12. [Data retention and access control](#12-data-retention-and-access-control)
13. [Rollout and verification](#13-rollout-and-verification)
14. [Glossary and references](#14-glossary-and-references)

## 1. Objectives and scope

<!-- Guidance: Name the questions operators, engineers and governance teams must be able to answer from telemetry. Each later section exists to answer one of them. Pitfall: collecting everything "just in case", which creates cost and privacy risk without better answers. -->

### 1.1 Objectives

| ID | Question telemetry must answer | Primary users |
|---|---|---|
| OBJ-001 *(Example)* | Is the assistant available and fast enough for customers right now? | On-call SRE |
| OBJ-002 *(Example)* | Which step of a slow or failed conversation caused the problem: model, retrieval, tool or guardrail? | Engineers |
| OBJ-003 *(Example)* | How many tokens and how much money does each feature, tenant and model consume? | Product, finance |
| OBJ-004 *(Example)* | Are answers grounded, safe and helpful, and is quality drifting? | AI quality team |
| OBJ-005 *(Example)* | Did an agent take an action it should not have, and can we reconstruct why? | Security, incident response |
| [OBJ-nnn] | [Question] | [Users] |

### 1.2 Scope

| In scope | Out of scope |
|---|---|
| *Example:* Assistant API, agent runtime, retrieval service, tool servers | Training pipelines (see model monitoring plan) |
| *Example:* Calls to hosted and self-hosted models | Provider-internal telemetry we cannot see |
| *Example:* Online evaluation and guardrail decisions | Offline benchmark runs |
| [Item] | [Item] |

## 2. Telemetry architecture

<!-- Guidance: Show where telemetry is produced, where sensitive content is separated, and where it is stored. The content path and the operational path should be visibly different. -->

```mermaid
flowchart LR
  app["Assistant API and agent runtime (OTel SDK, GenAI instrumentation)"] --> col["OTel Collector"]
  tools["Tool servers (MCP)"] --> col
  rag["Retrieval service"] --> col
  app -->|content hook, redacted| store[("Content store, restricted access")]
  col -->|redaction processor| traces[("Trace backend")]
  col --> metrics[("Metrics backend")]
  col --> logs[("Log backend")]
  traces --> eval["Online evaluators"]
  store --> eval
  eval -->|gen_ai.evaluation.result events| col
  metrics --> dash["Dashboards and alerts"]
```

[Describe the pipeline. Example: "Services use the OpenTelemetry SDK with
GenAI instrumentation for the model client and agent framework. Spans and
metrics go to a central Collector that applies redaction and tail-based
sampling. Prompts and completions are never stored in span attributes in
production; an in-process hook writes them, after redaction, to a
restricted content store and records a reference on the span."]

## 3. Signals and span model

<!-- Guidance: Define the span tree for one request so every team instruments the same way. Use the operation names from the conventions; add custom spans only where the conventions have no equivalent. -->

### 3.1 Span model

The tree below shows the spans for one customer turn in which the agent
plans, calls the model, retrieves documents, uses one tool and answers.

```mermaid
flowchart TB
  root["HTTP POST /chat (server span)"] --> agent["invoke_agent support-agent"]
  agent --> plan["plan support-agent"]
  agent --> chat1["chat [model]"]
  agent --> ret["retrieval help-articles"]
  ret --> emb["embeddings [embedding model]"]
  agent --> tool["execute_tool order_lookup"]
  tool --> mcp["tools/call order_lookup (MCP client span)"]
  agent --> chat2["chat [model]"]
  agent --> guard["guardrail output-check (custom span)"]
```

### 3.2 Span catalogue

| ID | Operation | Span name pattern | Kind | Emitted by | Notes |
|---|---|---|---|---|---|
| SPN-001 *(Example)* | Model inference (`chat`, `text_completion`, `generate_content`) | `{gen_ai.operation.name} {gen_ai.request.model}` | Client | Model client instrumentation | One span per logical call, including automatic retries |
| SPN-002 *(Example)* | Embeddings (`embeddings`) | `{gen_ai.operation.name} {gen_ai.request.model}` | Client | Retrieval service | |
| SPN-003 *(Example)* | Retrieval (`retrieval`) | `{gen_ai.operation.name} {gen_ai.data_source.id}` | Client | Retrieval service | Records top-k and data source |
| SPN-004 *(Example)* | Tool execution (`execute_tool`) | `execute_tool {gen_ai.tool.name}` | Internal | Agent runtime | One per tool call, linked by `gen_ai.tool.call.id` |
| SPN-005 *(Example)* | Agent invocation (`invoke_agent`) | `invoke_agent {gen_ai.agent.name}` | Internal or client | Agent runtime | Parent of the agent's steps |
| SPN-006 *(Example)* | Planning (`plan`) | `plan {gen_ai.agent.name}` | Internal | Agent runtime | Optional |
| SPN-007 *(Example)* | Workflow (`invoke_workflow`) | `invoke_workflow {gen_ai.workflow.name}` | Internal | Orchestrator | For multi-agent flows |
| SPN-008 *(Example)* | MCP request | `{mcp.method.name} {target}` | Client and server | MCP client and server | Per the MCP section of the conventions |
| SPN-009 *(Example)* | Guardrail check | `guardrail {policy}` | Internal | Guardrail service | Custom: no convention yet; see 4.3 |
| [SPN-nnn] | [Operation] | [Pattern] | [Kind] | [Emitter] | [Notes] |

### 3.3 Context propagation

Propagate trace context (W3C Trace Context) across every hop: API to
agent runtime, agent runtime to tool servers, and into asynchronous work
such as queued tool executions. For MCP over HTTP, use the standard
headers; for transports without headers, follow the MCP section of the
conventions. Record `gen_ai.conversation.id` on every span of a
conversation so that a multi-turn session can be reassembled without
relying on one trace per turn.

## 4. Attributes and semantic conventions

<!-- Guidance: List the attributes each span must carry, using the exact names from the pinned convention version. Mark the requirement level you adopt; you may be stricter than the convention. Pitfall: copying attribute names from blog posts written against older versions (for example gen_ai.system instead of gen_ai.provider.name). -->

### 4.1 Convention baseline

| Item | Value |
|---|---|
| Convention set | OpenTelemetry semantic conventions for generative AI |
| Status | Development (names and semantics may change) |
| Location | `github.com/open-telemetry/semantic-conventions-genai` (moved from the main semantic conventions repository from release v1.42.0) |
| Pinned version | [Release tag, or main branch commit and date if no release exists] |
| Core conventions used with it | *Example:* Semantic conventions v1.44.0 for `server.*`, `error.type`, HTTP and resource attributes |
| Review trigger | Any new release of the GenAI conventions; quarterly otherwise |

### 4.2 Attribute register

| Attribute | Applies to | Our requirement | Source | Notes |
|---|---|---|---|---|
| `gen_ai.operation.name` | All GenAI spans | Required | Convention | `chat`, `embeddings`, `retrieval`, `execute_tool`, `invoke_agent` and other listed values |
| `gen_ai.provider.name` | Model, embedding, retrieval spans | Required | Convention | *Example:* `anthropic`, `openai`, `aws.bedrock`; set to best knowledge when behind a proxy |
| `gen_ai.request.model` | Model spans | Required | Convention | Model name as requested |
| `gen_ai.response.model` | Model spans | Required when returned | Convention | Exact snapshot that answered; join key to the AI-BOM |
| `gen_ai.response.id` | Model spans | Recommended | Convention | Correlates evaluations and provider support cases |
| `gen_ai.response.finish_reasons` | Model spans | Required | Convention | Detects truncation, tool calls, content filter stops |
| `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.max_tokens` | Model spans | Required when set | Convention | Needed to reproduce behaviour |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Model spans | Required | Convention | Basis for cost in section 11 |
| `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.reasoning.output_tokens` | Model spans | Required when reported | Convention | Cached and reasoning tokens are priced differently |
| `gen_ai.conversation.id` | All spans of a conversation | Required | Convention | Pseudonymous ID, never an email or account number |
| `gen_ai.agent.name`, `gen_ai.agent.id` | Agent spans | Required | Convention | Join key to the agent specification |
| `gen_ai.tool.name`, `gen_ai.tool.type`, `gen_ai.tool.call.id` | Tool spans | Required | Convention | Tool call arguments and results follow section 7 |
| `gen_ai.data_source.id` | Retrieval spans | Required | Convention | Index or collection identifier |
| `gen_ai.prompt.name`, `gen_ai.prompt.version` | Model spans | Required when a named template is used | Convention | Join key to the prompt registry |
| `gen_ai.output.type` | Model spans | Recommended | Convention | Text, JSON, image or speech |
| `server.address`, `server.port` | Client spans | Recommended | Core conventions | Endpoint and region identification |
| `error.type` | All spans | Required on error | Core conventions (Stable) | Low-cardinality error class |
| `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, `gen_ai.tool.definitions` | Model spans | Off by default | Convention (opt-in) | Governed by section 7 |
| [Attribute] | [Spans] | [Level] | [Source] | [Notes] |

### 4.3 Custom attributes

Use a reserved organisation namespace for anything the conventions do
not cover, register each attribute here, and propose it upstream when it
is generally useful.

| Attribute | Type | Meaning | Cardinality | Example |
|---|---|---|---|---|
| `[org].tenant.id` *(Example)* | string | Customer tenant for cost and quota | Medium (bounded by tenant count) | `t-0042` |
| `[org].feature` *(Example)* | string | Product feature that issued the call | Low | `help-answer` |
| `[org].guardrail.policy` *(Example)* | string | Guardrail policy evaluated | Low | `output-safety-v6` |
| `[org].guardrail.decision` *(Example)* | string | Allow, block, redact or escalate | Low | `block` |
| `[org].cost.usd` *(Example)* | double | Estimated cost of the call | High (value, not label) | `0.0042` |
| [Attribute] | [Type] | [Meaning] | [Cardinality] | [Example] |

## 5. Metrics

<!-- Guidance: Use the convention metrics where they exist and derive the rest. Keep high-cardinality values (conversation IDs, response IDs) out of metric labels. -->

### 5.1 Convention metrics

| Metric | Instrument and unit | Use | Notes |
|---|---|---|---|
| `gen_ai.client.inference.duration` | Histogram, s | End-to-end latency of a model call | Same value as the inference span duration |
| `gen_ai.client.inference.time_to_first_chunk` | Histogram, s | Streaming responsiveness | Streaming calls only |
| `gen_ai.client.inference.time_per_output_chunk` | Histogram, s | Generation speed | Streaming calls only |
| `gen_ai.client.inference.usage.input_tokens`, `gen_ai.client.inference.usage.output_tokens` | Counters, tokens | Token consumption and cost proxy | Broken down by `gen_ai.token.modality` |
| `gen_ai.client.inference.usage.cache_read.input_tokens`, `gen_ai.client.inference.usage.reasoning.output_tokens` | Counters, tokens | Cache effectiveness, reasoning share | Subsets of the input and output counters |
| `gen_ai.client.inference.operation.input_tokens`, `gen_ai.client.inference.operation.output_tokens` | Histograms, tokens | Per-call size percentiles and outliers | Do not sum these for totals |
| `gen_ai.client.operation.duration` | Histogram, s | Duration of non-inference GenAI operations | |
| `gen_ai.server.request.duration`, `gen_ai.server.time_to_first_token`, `gen_ai.server.time_per_output_token` | Histograms, s | Self-hosted model servers | Only where we operate the server |
| `gen_ai.invoke_workflow.duration` | Histogram, s | Multi-agent workflow latency | |
| `mcp.client.operation.duration`, `mcp.server.operation.duration` | Histograms, s | Tool server latency | MCP section of the conventions |

<!-- Guidance: Releases of the main semantic conventions up to v1.41.0 defined a single gen_ai.client.token.usage histogram. If your instrumentation still emits it, record which name each service uses during migration and do not double count. -->

### 5.2 Derived metrics

| ID | Metric | Definition | Labels | Objective |
|---|---|---|---|---|
| MET-001 *(Example)* | Error rate | Model spans with `error.type` set / all model spans | provider, model, feature | OBJ-001 |
| MET-002 *(Example)* | Tool failure rate | `execute_tool` spans with error / all tool spans | tool | OBJ-002 |
| MET-003 *(Example)* | Truncation rate | Responses with finish reason `length` / all responses | model, feature | OBJ-004 |
| MET-004 *(Example)* | Refusal rate | Responses classified as refusal by the online evaluator / all responses | feature | OBJ-004 |
| MET-005 *(Example)* | Guardrail block rate | Guardrail decisions `block` / all guardrail checks | policy, direction | OBJ-004, OBJ-005 |
| MET-006 *(Example)* | Groundedness score | Mean `gen_ai.evaluation.score.value` for evaluation `groundedness` | feature | OBJ-004 |
| MET-007 *(Example)* | Steps per task | Count of child spans of `invoke_agent` | agent | OBJ-002 |
| MET-008 *(Example)* | Cost per conversation | Sum of `[org].cost.usd` over a conversation | feature, tenant tier | OBJ-003 |
| [MET-nnn] | [Metric] | [Definition] | [Labels] | [Objective] |

### 5.3 Cardinality budget

| Label | Allowed on metrics | Reason |
|---|---|---|
| `gen_ai.request.model`, `gen_ai.provider.name`, `gen_ai.operation.name` | Yes | Bounded |
| `gen_ai.tool.name`, `gen_ai.agent.name` | Yes | Bounded by registry |
| `[org].tenant.id` | Only on cost metrics | Bounded, but large |
| `gen_ai.conversation.id`, `gen_ai.response.id`, `gen_ai.tool.call.id` | No | Unbounded; use traces |

## 6. Logs and events

<!-- Guidance: Use events for things that happen at a point in time (an evaluation result, a guardrail decision, an exception) and correlate them to the span. Follow the log schema of the general observability specification. -->

| Event | Source | Content | Correlation |
|---|---|---|---|
| `gen_ai.evaluation.result` | Online evaluators | `gen_ai.evaluation.name`, `gen_ai.evaluation.score.value`, `gen_ai.evaluation.score.label`, `gen_ai.evaluation.explanation` | Parent span or `gen_ai.response.id` |
| `gen_ai.client.inference.operation.details` | Model client (opt-in) | Request and response details, including content when enabled | Inference span |
| `gen_ai.client.operation.exception` | Model client | Exception details | Failing span |
| `[org].guardrail.decision` *(Example)* | Guardrail service | Policy, decision, category, no raw content | Guardrail span |
| `[org].agent.action` *(Example)* | Agent runtime | Side-effecting action, target, confirmation status | Tool span |
| [Event] | [Source] | [Content] | [Correlation] |

Agent actions with side effects (refunds, emails, data changes) are also
written to the audit log with the trace ID, so that security can
reconstruct an action even when trace sampling dropped the trace.

## 7. Prompt and completion capture

<!-- Guidance: Prompts, system instructions, retrieved documents, tool arguments and completions can contain personal data, secrets and confidential business content. The conventions say instrumentations should not capture content by default. Decide here, per environment, what is captured, where, for how long and who can read it. Pitfall: enabling content capture in production to debug one incident and never turning it off. -->

### 7.1 Capture modes

| Mode | Description | Allowed in |
|---|---|---|
| M0 Off | No content recorded; metadata only | All environments (default) |
| M1 On span attributes | `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions` on spans | *Example:* Development and test with synthetic data only |
| M2 External store with reference | Content written by an in-process hook to a restricted store; span holds a reference | *Example:* Production, sampled, after redaction |
| M3 Full for an incident | Temporary M2 at 100% sampling for one tenant or feature | *Example:* Production, with approval, maximum [7] days |

### 7.2 Controls

| Control | Rule |
|---|---|
| Consent and notice | *Example:* Privacy notice states that conversations may be reviewed for quality; enterprise tenants can opt out contractually, and opted-out tenants are M0 |
| Redaction | *Example:* Hook removes payment card numbers, government IDs, secrets and access tokens by pattern and classifier before storage; redaction counts recorded as metrics |
| Sampling | *Example:* M2 at [5]% of conversations, stratified by feature; 100% of conversations with a guardrail block or negative feedback |
| Tool arguments and results | *Example:* `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` follow the same mode and redaction as messages |
| Retrieved documents | *Example:* Store document IDs and scores, not document text |
| Truncation | *Example:* Message content truncated at [n] characters while preserving JSON structure |
| Configuration | *Example:* Capture enabled only through deployment configuration reviewed by privacy; instrumentation flags such as `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` are off in production images |
| Data subject requests | *Example:* Content store indexed by pseudonymous conversation ID so that erasure requests can be honoured |
| [Control] | [Rule] |

## 8. Evaluation hooks

<!-- Guidance: Connect telemetry to evaluation so that quality is measured on real traffic, not only on test sets. Define which evaluators run, on what sample, and how results come back as telemetry. Link the agent evaluation plan for offline evaluation. -->

| ID | Evaluator | Type | Input | Sample | Output | Owner |
|---|---|---|---|---|---|---|
| EVH-001 *(Example)* | Groundedness | LLM-as-judge with rubric | Answer and retrieved document IDs resolved from the content store | [5]% of M2 conversations | `gen_ai.evaluation.result`, name `groundedness`, score 0 to 1 | [@quality-owner] |
| EVH-002 *(Example)* | Refusal detector | Classifier | Answer text | 100% (runs in-process, stores label only) | Event with label `refusal` or `answer` | [@quality-owner] |
| EVH-003 *(Example)* | Task success | Rule on outcome | Ticket closed without escalation within [24] hours | 100% | Event joined by conversation ID | [@product-owner] |
| EVH-004 *(Example)* | User feedback | Explicit thumbs up or down | UI event | 100% | Event with label | [@product-owner] |
| [EVH-nnn] | [Evaluator] | [Type] | [Input] | [Sample] | [Output] | [@owner] |

Evaluator models are themselves recorded in the AI-BOM and their spans
are tagged with `[org].feature` set to `evaluation` so that their cost and
latency do not pollute production figures.

## 9. Dashboards

<!-- Guidance: One dashboard per audience. Each panel should map to an objective in section 1. Remove panels nobody uses. -->

| Dashboard | Audience | Panels | Objective |
|---|---|---|---|
| DSH-001 *(Example)* Service health | On-call | Request rate, error rate (MET-001), p50/p95/p99 `gen_ai.client.inference.duration`, time to first chunk, provider status | OBJ-001 |
| DSH-002 *(Example)* Agent behaviour | Engineers | Steps per task (MET-007), tool failure rate (MET-002), top slow spans, trace exemplars | OBJ-002 |
| DSH-003 *(Example)* Tokens and cost | Product, finance | Tokens by model and feature, cache read share, cost per conversation (MET-008), budget burn | OBJ-003 |
| DSH-004 *(Example)* Quality and safety | AI quality team | Groundedness (MET-006), refusal rate (MET-004), guardrail block rate (MET-005), feedback ratio, truncation rate | OBJ-004 |
| DSH-005 *(Example)* Agent actions | Security | Side-effecting actions by tool, confirmations, blocked actions, anomalies | OBJ-005 |
| [DSH-nnn] | [Audience] | [Panels] | [Objective] |

## 10. Alerts and service level objectives

<!-- Guidance: Define SLOs from the user's point of view, then alerts on burn rate. Quality alerts are slower and noisier than availability alerts; route them to working hours unless they indicate harm. Thresholds below are examples, not recommendations. -->

### 10.1 Service level objectives

| ID | SLI | Target | Window | Measurement |
|---|---|---|---|---|
| SLO-001 *(Example)* | Share of chat requests that complete without error | [99.5]% | 28 days | Server spans without `error.type` / all server spans |
| SLO-002 *(Example)* | Share of streaming answers with first chunk under [2] s | [95]% | 28 days | `gen_ai.client.inference.time_to_first_chunk` |
| SLO-003 *(Example)* | Share of evaluated answers with groundedness at least [0.8] | [90]% | 7 days | EVH-001 events |
| [SLO-nnn] | [SLI] | [Target] | [Window] | [Query] |

### 10.2 Alerts

| ID | Condition | Severity | Route | Runbook |
|---|---|---|---|---|
| ALR-001 *(Example)* | SLO-001 burn rate above [14] over 1 hour | Page | On-call SRE | RB-LLM-01 provider failover |
| ALR-002 *(Example)* | Tool failure rate above [5]% for 15 minutes for any tool | Page | Agent platform on-call | RB-LLM-02 |
| ALR-003 *(Example)* | Guardrail block rate doubles against 7-day baseline | Ticket, page if above [2]% | AI quality and security | RB-LLM-03 possible attack or regression |
| ALR-004 *(Example)* | Daily cost above [120]% of budget | Ticket | Platform owner | RB-LLM-04 |
| ALR-005 *(Example)* | Side-effecting tool calls without confirmation event | Page | Security | RB-LLM-05 agent containment |
| ALR-006 *(Example)* | `gen_ai.response.model` changes without a release | Ticket | AI platform | RB-LLM-06 unannounced model change |
| [ALR-nnn] | [Condition] | [Severity] | [Route] | [Runbook] |

## 11. Cost attribution

<!-- Guidance: Turn token counts into money per feature, tenant and model, using a versioned price table. Pitfall: using list prices when the contract price differs, or ignoring cached and reasoning tokens. -->

| Item | Rule |
|---|---|
| Price source | *Example:* Price table `[repo]/pricing.yaml`, versioned, owned by finance; one row per provider, model and token category |
| Calculation | Input, cached input, output and reasoning tokens multiplied by their unit prices; result stored as `[org].cost.usd` on the span |
| Attribution keys | `[org].feature`, `[org].tenant.id`, `gen_ai.agent.name`, `gen_ai.request.model` |
| Shared costs | *Example:* Embedding and evaluator calls allocated to the feature that triggered them |
| Reconciliation | *Example:* Monthly comparison with provider invoices; difference above [3]% investigated |
| Budgets | *Example:* Monthly budget per feature with alert ALR-004 |

## 12. Data retention and access control

<!-- Guidance: Set retention by data class, not by backend. Content is the most sensitive class and should have the shortest retention and the narrowest access. Align with the general observability specification and the privacy notice. -->

| Data class | Examples | Retention | Access | Storage |
|---|---|---|---|---|
| Metrics | Convention and derived metrics | *Example:* 13 months | Engineering, product, finance | Metrics backend |
| Traces without content | Span metadata | *Example:* 30 days | Engineering | Trace backend |
| Captured content (M2, M3) | Redacted prompts, completions, tool arguments | *Example:* 30 days, then deleted | Named reviewers in the AI quality team; access logged | Content store, encrypted, separate keys |
| Evaluation events | Scores and labels | *Example:* 13 months | AI quality team | Log backend |
| Agent action audit log | Side-effecting actions | *Example:* [n] years per records policy | Security, audit | Audit store, append-only |
| [Class] | [Examples] | [Retention] | [Access] | [Storage] |

Access reviews run every [quarter]. Exports of content outside the content
store require approval from the privacy officer and are logged.

## 13. Rollout and verification

<!-- Guidance: Verify that telemetry is correct before relying on it. Test that redaction works with seeded data, that names match the pinned convention, and that dashboards show what they claim. -->

| ID | Check | Method | Pass criterion |
|---|---|---|---|
| VER-001 *(Example)* | Span names and attributes match section 4 | Automated test against recorded traces | 100% of required attributes present |
| VER-002 *(Example)* | Redaction removes seeded secrets and identifiers | Inject synthetic records in staging | Zero seeded values in the content store |
| VER-003 *(Example)* | Content capture off in production images | Configuration scan in CI | Flag absent or false |
| VER-004 *(Example)* | Token counts match provider usage | Compare one day of metrics with provider console | Within [1]% |
| VER-005 *(Example)* | Every alert fires in a game day | Fault injection | All alerts reach the right route |
| [VER-nnn] | [Check] | [Method] | [Criterion] |

## 14. Glossary and references

<!-- Guidance: Define terms readers may confuse, and list the exact convention versions used. -->

| Term | Meaning in this document |
|---|---|
| Span | A timed operation in a trace, such as one model call or one tool execution |
| Semantic conventions | Agreed names and meanings for telemetry attributes, spans, metrics and events |
| Development status | OpenTelemetry stability level meaning a convention may still change without a deprecation period |
| Content capture | Recording prompts, instructions, retrieved text, tool arguments or completions in telemetry |
| Groundedness | Degree to which an answer is supported by the retrieved sources |
| Burn rate | Speed at which an SLO error budget is being consumed |

References:

- OpenTelemetry GenAI semantic conventions, https://github.com/open-telemetry/semantic-conventions-genai
- OpenTelemetry semantic conventions (core), https://opentelemetry.io/docs/specs/semconv/
- OpenTelemetry document status definitions, https://opentelemetry.io/docs/specs/otel/document-status/
- Related templates: `ops/observability-spec`, `ai-data/model-monitoring-plan`, `ai-data/guardrails-spec`, `ai-data/agent-evaluation-plan`, `ai-data/ai-bom`, `ai-data/mcp-server-spec`

## Standard and licence note

Based on the OpenTelemetry semantic conventions for generative AI
(Development status, `semantic-conventions-genai` repository, checked
2026-10-22) and the OpenTelemetry core semantic conventions v1.44.0,
published by the OpenTelemetry project under the Apache License 2.0.
Attribute, span and metric names are cited for interoperability; all
explanatory text is original. Because these conventions are still in
Development, check the current version in the repository before
instrumenting and update section 4.1. Template licence: MIT.