LLM and Agent Observability Specification: [System name] [version]
Specification of the telemetry an LLM application or agent system must
emit and how that telemetry is processed, protected and used: traces
for model calls, tool calls, retrieval and agent steps; metrics for
tokens, latency, cost, errors and quality; controlled capture of prompts
and completions; evaluation hooks; dashboards, alerts and service level
objectives. Attribute, span and metric names follow the OpenTelemetry
semantic conventions for generative AI, which have Development
status and are now maintained in the semantic-conventions-genai
repository. Replace every [bracketed] value and every row marked
Example. Worked examples use a support agent that calls a hosted chat
model, a retrieval service and three tools.
Document control
| Field | Value |
|---|
| Document ID | [OBS-LLM-001] |
| System and version | [Name] [version] |
| Semantic convention baseline | Example: OpenTelemetry GenAI semantic conventions, semantic-conventions-genai main branch at commit [sha], YYYY-MM-DD |
| Telemetry backend | [Tracing, metrics and log backends] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@ai-platform-owner] |
| Approvers / reviewers | [@sre-lead], [@privacy-officer], [@security-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Related documents | [Observability specification], [Guardrails specification], [Agent evaluation plan], [Model monitoring plan], [AI-BOM] |
Approvals
| Role | Name | Decision | Date |
|---|
| AI platform owner | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| SRE lead | [@name] | [Decision] | YYYY-MM-DD |
| Privacy officer | [@name] | [Decision] | YYYY-MM-DD |
| Security lead | [@name] | [Decision] | YYYY-MM-DD |
Revision history
| Version | Date | Author | Change summary |
|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | Example: Approved with content capture limited to external store |
| 1.1 | YYYY-MM-DD | [@name] | Example: Moved token metrics to gen_ai.client.inference.usage.* after convention update |
Contents
1. Objectives and scope
1.1 Objectives
| ID | Question telemetry must answer | Primary users |
|---|
| OBJ-001 (Example) | Is the assistant available and fast enough for customers right now? | On-call SRE |
| OBJ-002 (Example) | Which step of a slow or failed conversation caused the problem: model, retrieval, tool or guardrail? | Engineers |
| OBJ-003 (Example) | How many tokens and how much money does each feature, tenant and model consume? | Product, finance |
| OBJ-004 (Example) | Are answers grounded, safe and helpful, and is quality drifting? | AI quality team |
| OBJ-005 (Example) | Did an agent take an action it should not have, and can we reconstruct why? | Security, incident response |
| [OBJ-nnn] | [Question] | [Users] |
1.2 Scope
| In scope | Out of scope |
|---|
| Example: Assistant API, agent runtime, retrieval service, tool servers | Training pipelines (see model monitoring plan) |
| Example: Calls to hosted and self-hosted models | Provider-internal telemetry we cannot see |
| Example: Online evaluation and guardrail decisions | Offline benchmark runs |
| [Item] | [Item] |
2. Telemetry architecture
[Describe the pipeline. Example: "Services use the OpenTelemetry SDK with
GenAI instrumentation for the model client and agent framework. Spans and
metrics go to a central Collector that applies redaction and tail-based
sampling. Prompts and completions are never stored in span attributes in
production; an in-process hook writes them, after redaction, to a
restricted content store and records a reference on the span."]
3. Signals and span model
3.1 Span model
The tree below shows the spans for one customer turn in which the agent
plans, calls the model, retrieves documents, uses one tool and answers.
3.2 Span catalogue
| ID | Operation | Span name pattern | Kind | Emitted by | Notes |
|---|
| SPN-001 (Example) | Model inference (chat, text_completion, generate_content) | {gen_ai.operation.name} {gen_ai.request.model} | Client | Model client instrumentation | One span per logical call, including automatic retries |
| SPN-002 (Example) | Embeddings (embeddings) | {gen_ai.operation.name} {gen_ai.request.model} | Client | Retrieval service | |
| SPN-003 (Example) | Retrieval (retrieval) | {gen_ai.operation.name} {gen_ai.data_source.id} | Client | Retrieval service | Records top-k and data source |
| SPN-004 (Example) | Tool execution (execute_tool) | execute_tool {gen_ai.tool.name} | Internal | Agent runtime | One per tool call, linked by gen_ai.tool.call.id |
| SPN-005 (Example) | Agent invocation (invoke_agent) | invoke_agent {gen_ai.agent.name} | Internal or client | Agent runtime | Parent of the agent's steps |
| SPN-006 (Example) | Planning (plan) | plan {gen_ai.agent.name} | Internal | Agent runtime | Optional |
| SPN-007 (Example) | Workflow (invoke_workflow) | invoke_workflow {gen_ai.workflow.name} | Internal | Orchestrator | For multi-agent flows |
| SPN-008 (Example) | MCP request | {mcp.method.name} {target} | Client and server | MCP client and server | Per the MCP section of the conventions |
| SPN-009 (Example) | Guardrail check | guardrail {policy} | Internal | Guardrail service | Custom: no convention yet; see 4.3 |
| [SPN-nnn] | [Operation] | [Pattern] | [Kind] | [Emitter] | [Notes] |
3.3 Context propagation
Propagate trace context (W3C Trace Context) across every hop: API to
agent runtime, agent runtime to tool servers, and into asynchronous work
such as queued tool executions. For MCP over HTTP, use the standard
headers; for transports without headers, follow the MCP section of the
conventions. Record gen_ai.conversation.id on every span of a
conversation so that a multi-turn session can be reassembled without
relying on one trace per turn.
4. Attributes and semantic conventions
4.1 Convention baseline
| Item | Value |
|---|
| Convention set | OpenTelemetry semantic conventions for generative AI |
| Status | Development (names and semantics may change) |
| Location | github.com/open-telemetry/semantic-conventions-genai (moved from the main semantic conventions repository from release v1.42.0) |
| Pinned version | [Release tag, or main branch commit and date if no release exists] |
| Core conventions used with it | Example: Semantic conventions v1.44.0 for server.*, error.type, HTTP and resource attributes |
| Review trigger | Any new release of the GenAI conventions; quarterly otherwise |
4.2 Attribute register
| Attribute | Applies to | Our requirement | Source | Notes |
|---|
gen_ai.operation.name | All GenAI spans | Required | Convention | chat, embeddings, retrieval, execute_tool, invoke_agent and other listed values |
gen_ai.provider.name | Model, embedding, retrieval spans | Required | Convention | Example: anthropic, openai, aws.bedrock; set to best knowledge when behind a proxy |
gen_ai.request.model | Model spans | Required | Convention | Model name as requested |
gen_ai.response.model | Model spans | Required when returned | Convention | Exact snapshot that answered; join key to the AI-BOM |
gen_ai.response.id | Model spans | Recommended | Convention | Correlates evaluations and provider support cases |
gen_ai.response.finish_reasons | Model spans | Required | Convention | Detects truncation, tool calls, content filter stops |
gen_ai.request.temperature, gen_ai.request.top_p, gen_ai.request.max_tokens | Model spans | Required when set | Convention | Needed to reproduce behaviour |
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | Model spans | Required | Convention | Basis for cost in section 11 |
gen_ai.usage.cache_read.input_tokens, gen_ai.usage.reasoning.output_tokens | Model spans | Required when reported | Convention | Cached and reasoning tokens are priced differently |
gen_ai.conversation.id | All spans of a conversation | Required | Convention | Pseudonymous ID, never an email or account number |
gen_ai.agent.name, gen_ai.agent.id | Agent spans | Required | Convention | Join key to the agent specification |
gen_ai.tool.name, gen_ai.tool.type, gen_ai.tool.call.id | Tool spans | Required | Convention | Tool call arguments and results follow section 7 |
gen_ai.data_source.id | Retrieval spans | Required | Convention | Index or collection identifier |
gen_ai.prompt.name, gen_ai.prompt.version | Model spans | Required when a named template is used | Convention | Join key to the prompt registry |
gen_ai.output.type | Model spans | Recommended | Convention | Text, JSON, image or speech |
server.address, server.port | Client spans | Recommended | Core conventions | Endpoint and region identification |
error.type | All spans | Required on error | Core conventions (Stable) | Low-cardinality error class |
gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions | Model spans | Off by default | Convention (opt-in) | Governed by section 7 |
| [Attribute] | [Spans] | [Level] | [Source] | [Notes] |
4.3 Custom attributes
Use a reserved organisation namespace for anything the conventions do
not cover, register each attribute here, and propose it upstream when it
is generally useful.
| Attribute | Type | Meaning | Cardinality | Example |
|---|
[org].tenant.id (Example) | string | Customer tenant for cost and quota | Medium (bounded by tenant count) | t-0042 |
[org].feature (Example) | string | Product feature that issued the call | Low | help-answer |
[org].guardrail.policy (Example) | string | Guardrail policy evaluated | Low | output-safety-v6 |
[org].guardrail.decision (Example) | string | Allow, block, redact or escalate | Low | block |
[org].cost.usd (Example) | double | Estimated cost of the call | High (value, not label) | 0.0042 |
| [Attribute] | [Type] | [Meaning] | [Cardinality] | [Example] |
5. Metrics
5.1 Convention metrics
| Metric | Instrument and unit | Use | Notes |
|---|
gen_ai.client.inference.duration | Histogram, s | End-to-end latency of a model call | Same value as the inference span duration |
gen_ai.client.inference.time_to_first_chunk | Histogram, s | Streaming responsiveness | Streaming calls only |
gen_ai.client.inference.time_per_output_chunk | Histogram, s | Generation speed | Streaming calls only |
gen_ai.client.inference.usage.input_tokens, gen_ai.client.inference.usage.output_tokens | Counters, tokens | Token consumption and cost proxy | Broken down by gen_ai.token.modality |
gen_ai.client.inference.usage.cache_read.input_tokens, gen_ai.client.inference.usage.reasoning.output_tokens | Counters, tokens | Cache effectiveness, reasoning share | Subsets of the input and output counters |
gen_ai.client.inference.operation.input_tokens, gen_ai.client.inference.operation.output_tokens | Histograms, tokens | Per-call size percentiles and outliers | Do not sum these for totals |
gen_ai.client.operation.duration | Histogram, s | Duration of non-inference GenAI operations | |
gen_ai.server.request.duration, gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token | Histograms, s | Self-hosted model servers | Only where we operate the server |
gen_ai.invoke_workflow.duration | Histogram, s | Multi-agent workflow latency | |
mcp.client.operation.duration, mcp.server.operation.duration | Histograms, s | Tool server latency | MCP section of the conventions |
5.2 Derived metrics
| ID | Metric | Definition | Labels | Objective |
|---|
| MET-001 (Example) | Error rate | Model spans with error.type set / all model spans | provider, model, feature | OBJ-001 |
| MET-002 (Example) | Tool failure rate | execute_tool spans with error / all tool spans | tool | OBJ-002 |
| MET-003 (Example) | Truncation rate | Responses with finish reason length / all responses | model, feature | OBJ-004 |
| MET-004 (Example) | Refusal rate | Responses classified as refusal by the online evaluator / all responses | feature | OBJ-004 |
| MET-005 (Example) | Guardrail block rate | Guardrail decisions block / all guardrail checks | policy, direction | OBJ-004, OBJ-005 |
| MET-006 (Example) | Groundedness score | Mean gen_ai.evaluation.score.value for evaluation groundedness | feature | OBJ-004 |
| MET-007 (Example) | Steps per task | Count of child spans of invoke_agent | agent | OBJ-002 |
| MET-008 (Example) | Cost per conversation | Sum of [org].cost.usd over a conversation | feature, tenant tier | OBJ-003 |
| [MET-nnn] | [Metric] | [Definition] | [Labels] | [Objective] |
5.3 Cardinality budget
| Label | Allowed on metrics | Reason |
|---|
gen_ai.request.model, gen_ai.provider.name, gen_ai.operation.name | Yes | Bounded |
gen_ai.tool.name, gen_ai.agent.name | Yes | Bounded by registry |
[org].tenant.id | Only on cost metrics | Bounded, but large |
gen_ai.conversation.id, gen_ai.response.id, gen_ai.tool.call.id | No | Unbounded; use traces |
6. Logs and events
| Event | Source | Content | Correlation |
|---|
gen_ai.evaluation.result | Online evaluators | gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, gen_ai.evaluation.explanation | Parent span or gen_ai.response.id |
gen_ai.client.inference.operation.details | Model client (opt-in) | Request and response details, including content when enabled | Inference span |
gen_ai.client.operation.exception | Model client | Exception details | Failing span |
[org].guardrail.decision (Example) | Guardrail service | Policy, decision, category, no raw content | Guardrail span |
[org].agent.action (Example) | Agent runtime | Side-effecting action, target, confirmation status | Tool span |
| [Event] | [Source] | [Content] | [Correlation] |
Agent actions with side effects (refunds, emails, data changes) are also
written to the audit log with the trace ID, so that security can
reconstruct an action even when trace sampling dropped the trace.
7. Prompt and completion capture
7.1 Capture modes
| Mode | Description | Allowed in |
|---|
| M0 Off | No content recorded; metadata only | All environments (default) |
| M1 On span attributes | gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions on spans | Example: Development and test with synthetic data only |
| M2 External store with reference | Content written by an in-process hook to a restricted store; span holds a reference | Example: Production, sampled, after redaction |
| M3 Full for an incident | Temporary M2 at 100% sampling for one tenant or feature | Example: Production, with approval, maximum [7] days |
7.2 Controls
| Control | Rule |
|---|
| Consent and notice | Example: Privacy notice states that conversations may be reviewed for quality; enterprise tenants can opt out contractually, and opted-out tenants are M0 |
| Redaction | Example: Hook removes payment card numbers, government IDs, secrets and access tokens by pattern and classifier before storage; redaction counts recorded as metrics |
| Sampling | Example: M2 at [5]% of conversations, stratified by feature; 100% of conversations with a guardrail block or negative feedback |
| Tool arguments and results | Example: gen_ai.tool.call.arguments and gen_ai.tool.call.result follow the same mode and redaction as messages |
| Retrieved documents | Example: Store document IDs and scores, not document text |
| Truncation | Example: Message content truncated at [n] characters while preserving JSON structure |
| Configuration | Example: Capture enabled only through deployment configuration reviewed by privacy; instrumentation flags such as OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT are off in production images |
| Data subject requests | Example: Content store indexed by pseudonymous conversation ID so that erasure requests can be honoured |
| [Control] | [Rule] |
8. Evaluation hooks
| ID | Evaluator | Type | Input | Sample | Output | Owner |
|---|
| EVH-001 (Example) | Groundedness | LLM-as-judge with rubric | Answer and retrieved document IDs resolved from the content store | [5]% of M2 conversations | gen_ai.evaluation.result, name groundedness, score 0 to 1 | [@quality-owner] |
| EVH-002 (Example) | Refusal detector | Classifier | Answer text | 100% (runs in-process, stores label only) | Event with label refusal or answer | [@quality-owner] |
| EVH-003 (Example) | Task success | Rule on outcome | Ticket closed without escalation within [24] hours | 100% | Event joined by conversation ID | [@product-owner] |
| EVH-004 (Example) | User feedback | Explicit thumbs up or down | UI event | 100% | Event with label | [@product-owner] |
| [EVH-nnn] | [Evaluator] | [Type] | [Input] | [Sample] | [Output] | [@owner] |
Evaluator models are themselves recorded in the AI-BOM and their spans
are tagged with [org].feature set to evaluation so that their cost and
latency do not pollute production figures.
9. Dashboards
| Dashboard | Audience | Panels | Objective |
|---|
| DSH-001 (Example) Service health | On-call | Request rate, error rate (MET-001), p50/p95/p99 gen_ai.client.inference.duration, time to first chunk, provider status | OBJ-001 |
| DSH-002 (Example) Agent behaviour | Engineers | Steps per task (MET-007), tool failure rate (MET-002), top slow spans, trace exemplars | OBJ-002 |
| DSH-003 (Example) Tokens and cost | Product, finance | Tokens by model and feature, cache read share, cost per conversation (MET-008), budget burn | OBJ-003 |
| DSH-004 (Example) Quality and safety | AI quality team | Groundedness (MET-006), refusal rate (MET-004), guardrail block rate (MET-005), feedback ratio, truncation rate | OBJ-004 |
| DSH-005 (Example) Agent actions | Security | Side-effecting actions by tool, confirmations, blocked actions, anomalies | OBJ-005 |
| [DSH-nnn] | [Audience] | [Panels] | [Objective] |
10. Alerts and service level objectives
10.1 Service level objectives
| ID | SLI | Target | Window | Measurement |
|---|
| SLO-001 (Example) | Share of chat requests that complete without error | [99.5]% | 28 days | Server spans without error.type / all server spans |
| SLO-002 (Example) | Share of streaming answers with first chunk under [2] s | [95]% | 28 days | gen_ai.client.inference.time_to_first_chunk |
| SLO-003 (Example) | Share of evaluated answers with groundedness at least [0.8] | [90]% | 7 days | EVH-001 events |
| [SLO-nnn] | [SLI] | [Target] | [Window] | [Query] |
10.2 Alerts
| ID | Condition | Severity | Route | Runbook |
|---|
| ALR-001 (Example) | SLO-001 burn rate above [14] over 1 hour | Page | On-call SRE | RB-LLM-01 provider failover |
| ALR-002 (Example) | Tool failure rate above [5]% for 15 minutes for any tool | Page | Agent platform on-call | RB-LLM-02 |
| ALR-003 (Example) | Guardrail block rate doubles against 7-day baseline | Ticket, page if above [2]% | AI quality and security | RB-LLM-03 possible attack or regression |
| ALR-004 (Example) | Daily cost above [120]% of budget | Ticket | Platform owner | RB-LLM-04 |
| ALR-005 (Example) | Side-effecting tool calls without confirmation event | Page | Security | RB-LLM-05 agent containment |
| ALR-006 (Example) | gen_ai.response.model changes without a release | Ticket | AI platform | RB-LLM-06 unannounced model change |
| [ALR-nnn] | [Condition] | [Severity] | [Route] | [Runbook] |
11. Cost attribution
| Item | Rule |
|---|
| Price source | Example: Price table [repo]/pricing.yaml, versioned, owned by finance; one row per provider, model and token category |
| Calculation | Input, cached input, output and reasoning tokens multiplied by their unit prices; result stored as [org].cost.usd on the span |
| Attribution keys | [org].feature, [org].tenant.id, gen_ai.agent.name, gen_ai.request.model |
| Shared costs | Example: Embedding and evaluator calls allocated to the feature that triggered them |
| Reconciliation | Example: Monthly comparison with provider invoices; difference above [3]% investigated |
| Budgets | Example: Monthly budget per feature with alert ALR-004 |
12. Data retention and access control
| Data class | Examples | Retention | Access | Storage |
|---|
| Metrics | Convention and derived metrics | Example: 13 months | Engineering, product, finance | Metrics backend |
| Traces without content | Span metadata | Example: 30 days | Engineering | Trace backend |
| Captured content (M2, M3) | Redacted prompts, completions, tool arguments | Example: 30 days, then deleted | Named reviewers in the AI quality team; access logged | Content store, encrypted, separate keys |
| Evaluation events | Scores and labels | Example: 13 months | AI quality team | Log backend |
| Agent action audit log | Side-effecting actions | Example: [n] years per records policy | Security, audit | Audit store, append-only |
| [Class] | [Examples] | [Retention] | [Access] | [Storage] |
Access reviews run every [quarter]. Exports of content outside the content
store require approval from the privacy officer and are logged.
13. Rollout and verification
| ID | Check | Method | Pass criterion |
|---|
| VER-001 (Example) | Span names and attributes match section 4 | Automated test against recorded traces | 100% of required attributes present |
| VER-002 (Example) | Redaction removes seeded secrets and identifiers | Inject synthetic records in staging | Zero seeded values in the content store |
| VER-003 (Example) | Content capture off in production images | Configuration scan in CI | Flag absent or false |
| VER-004 (Example) | Token counts match provider usage | Compare one day of metrics with provider console | Within [1]% |
| VER-005 (Example) | Every alert fires in a game day | Fault injection | All alerts reach the right route |
| [VER-nnn] | [Check] | [Method] | [Criterion] |
14. Glossary and references
| Term | Meaning in this document |
|---|
| Span | A timed operation in a trace, such as one model call or one tool execution |
| Semantic conventions | Agreed names and meanings for telemetry attributes, spans, metrics and events |
| Development status | OpenTelemetry stability level meaning a convention may still change without a deprecation period |
| Content capture | Recording prompts, instructions, retrieved text, tool arguments or completions in telemetry |
| Groundedness | Degree to which an answer is supported by the retrieved sources |
| Burn rate | Speed at which an SLO error budget is being consumed |
Standard and licence note
Based on the OpenTelemetry semantic conventions for generative AI
(Development status, semantic-conventions-genai repository, checked
2026-10-22) and the OpenTelemetry core semantic conventions v1.44.0,
published by the OpenTelemetry project under the Apache License 2.0.
Attribute, span and metric names are cited for interoperability; all
explanatory text is original. Because these conventions are still in
Development, check the current version in the repository before
instrumenting and update section 4.1. Template licence: MIT.