Observability Specification Template (OpenTelemetry)

Observability spec: logs, metrics and traces standards, semantic conventions, log schema, RED/USE catalogue, sampling, alerts, privacy, cost

OperationsFullOpenTelemetry conceptsobservability spec mathUpdated Oct 20, 2026

Open in editorDownload .md

How to use: Use it to set one telemetry standard for a service or platform before teams instrument independently. Start from the operator questions in section 1.2 and the naming conventions in section 4. The main pitfall is alerts without runbooks or owners, and personal data leaking into logs.

Preview

Observability Specification: [Service or platform name]

Complete observability specification for a service or platform: goals, telemetry signals, naming and semantic conventions, log schema, metrics catalogue, tracing, dashboards, SLIs and SLOs, alert rules, retention, cost, privacy and ownership. Concepts follow OpenTelemetry (signals, resources, attributes, context propagation, sampling). Replace every [bracketed] value and every row marked Example.

Document control

Field	Value
Document ID	[OBS-PLATFORM-001]
Services covered	[Service names or "all services on platform X"]
Version	[0.1]
Status	[Draft / In review / Approved / Superseded]
Owner	[@observability-owner]
Classification	[Internal]
Last updated	YYYY-MM-DD
Related documents	[SLO document], [On-call handbook], [Runbooks], [Data protection policy]

Approvals

Role	Name	Decision	Date
SRE lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Engineering lead	[@name]	[Decision]	YYYY-MM-DD
Security / privacy	[@name]	[Decision]	YYYY-MM-DD
FinOps	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	Initial draft
0.2	YYYY-MM-DD	[@name]	Example: Renamed `app.user_id` to `enduser.pseudo_id`; added PII scrubbing rule PR-04
1.0	YYYY-MM-DD	[@name]	Example: Approved

Goals and scope
Architecture and telemetry pipeline
Signal standards
Naming and semantic conventions
Logs
Metrics
Traces
Dashboards
SLIs and SLOs
Alerting
Privacy and security
Retention and cost
Ownership and governance
Appendices

1. Goals and scope

1.1 Goals

ID	Goal	Measure
OG-01	Example: Detect user-facing failures before customers report them	90% of incidents detected by alert first
OG-02	Example: Locate the failing component quickly	Median time to identify root service < 10 min
OG-03	Example: Keep telemetry cost proportional to traffic	Telemetry spend < 8% of infrastructure spend
[OG-nn]	[Goal]	[Measure]

1.2 Operator questions

The signals in this specification must answer these questions without deploying new code:

Is the service healthy for users right now, and since when has it not been?
Which endpoint, region, version or customer segment is affected?
Which downstream dependency is slow or failing?
What changed just before the problem started (deploys, config, traffic)?
[Question]

1.3 Scope

In scope	Out of scope
Example: All production and staging services on [platform]	Developer laptops
Example: Managed databases and queues via provider metrics	Business analytics events (see analytics spec)
[Item]	[Item]

2. Architecture and telemetry pipeline

Services emit telemetry with the OpenTelemetry SDK through the OTLP protocol to a local collector, which processes, scrubs and exports it to the storage backends.

Component	Technology	Version	Owner	Notes
SDK	Example: OpenTelemetry SDK for [language]	[x.y]	Each service team	Auto plus manual instrumentation
Collector	Example: OpenTelemetry Collector	[x.y]	Platform team	Agent per node, gateway per region
Log backend	[Product]	[x.y]	Platform team	[Notes]
Metrics backend	[Product]	[x.y]	Platform team	[Notes]
Trace backend	[Product]	[x.y]	Platform team	[Notes]

3. Signal standards

Signal	Use it for	Do not use it for	Required for every service
Metrics	Aggregated rates, ratios, latency distributions, saturation; alerting	Per-request detail, high-cardinality IDs	RED metrics for every entry point; USE metrics for every resource
Traces	Following one request across services; latency breakdown	Long-term trend analysis	Server and client spans for every network call
Logs	Discrete events with context: errors, state changes, audit	Counting things that should be metrics	Structured JSON per 5.1, correlated by trace ID
[Profiles / events]	[Use]	[Not for]	[Requirement]

All three signals carry the same resource attributes (4.2) so that a metric spike can be followed to traces and then to logs.

4. Naming and semantic conventions

4.1 General rules

Attribute and metric names are lowercase, dot-separated namespaces: http.request.method, db.system.
Custom attributes use the organisation namespace: [org].[domain].[name], for example acme.order.channel.
Units follow UCUM-style abbreviations in metric metadata (s, By, {request}), not in the name.
Durations are recorded in seconds as floating point.
Never put unbounded values (user IDs, URLs with IDs, error messages) into metric attributes.

4.2 Required resource attributes

Attribute	Example	Purpose
`service.name`	`checkout-api`	Identifies the service
`service.version`	`2026.10.3`	Correlates with deploys
`service.namespace`	`commerce`	Groups services by domain
`deployment.environment.name`	`production`	Separates environments
`cloud.region`	`eu-west-1`	Locates failures by region
`k8s.pod.name` or `host.name`	`checkout-api-7d9f`	Finds the instance
[Attribute]	[Example]	[Purpose]

4.3 Custom attributes register

Attribute	Type	Allowed values	Signals	Cardinality	Owner
`acme.order.channel`	string	`web`, `ios`, `android`, `partner`	metrics, traces, logs	4	[@team]
`acme.tenant.tier`	string	`free`, `pro`, `enterprise`	metrics, traces	3	[@team]
[name]	[type]	[values]	[signals]	[n]	[@team]

5. Logs

5.1 Log schema

Field	Type	Required	Description	Example
`timestamp`	RFC 3339 string, UTC	Yes	Time the event occurred	`2026-10-21T09:14:03.512Z`
`severity_text`	string	Yes	One of DEBUG, INFO, WARN, ERROR, FATAL	`ERROR`
`body`	string	Yes	Human-readable message without variable data where possible	`payment authorisation failed`
`trace_id`	hex string	When in a request	Correlates with traces	`4bf92f3577b34da6a3ce929d0e0e4736`
`span_id`	hex string	When in a request	Correlates with the span	`00f067aa0ba902b7`
`service.name`	string	Yes	From resource attributes	`checkout-api`
`event.name`	string	For domain events	Stable event identifier	`payment.authorisation.failed`
`error.type`	string	On errors	Error class or code	`card_declined`
`attributes`	object	No	Additional context per 4.3	`{"acme.order.channel":"web"}`
[field]	[type]	[Yes / No]	[Description]	[Example]

5.2 Severity guidelines

Level	Use when	Pages anyone?
DEBUG	Detail useful only during development; disabled in production by default	No
INFO	Normal state changes: started, configuration loaded, job completed	No
WARN	Unexpected but handled: retry succeeded, fallback used	No
ERROR	A request or job failed and the caller saw a failure	Only via metric-based alerts
FATAL	The process cannot continue	Via health checks

5.3 Logging rules

Log once per failure at the boundary that handles it, not at every layer.
Do not log request or response bodies by default.
Sample high-volume INFO logs at the collector when volume exceeds [n] lines per second per service.

6. Metrics

6.1 Method

RED for request-driven components: Rate, Errors, Duration.
USE for resources: Utilisation, Saturation, Errors.

6.2 Metrics catalogue

Metric name	Instrument	Unit	Attributes	Method	Description	Owner
`http.server.request.duration`	Histogram	s	`http.request.method`, `http.route`, `http.response.status_code`	RED (rate, errors, duration)	Server request latency; count gives rate	Service team
`http.client.request.duration`	Histogram	s	`server.address`, `http.response.status_code`	RED	Outbound call latency	Service team
`db.client.operation.duration`	Histogram	s	`db.system`, `db.operation.name`	RED	Database call latency	Service team
`process.cpu.utilization`	Gauge	1	`process.cpu.state`	USE	CPU utilisation per process	Platform
`acme.queue.depth`	Gauge	{message}	`messaging.destination.name`	USE (saturation)	Messages waiting	[@team]
`acme.orders.placed`	Counter	{order}	`acme.order.channel`	Business	Orders successfully placed	[@team]
[name]	[Counter / Histogram / Gauge / UpDownCounter]	[unit]	[attributes]	[RED / USE / Business]	[Description]	[@team]

6.3 Histogram buckets

[Define standard bucket boundaries or use exponential histograms. Example: latency buckets 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 seconds.]

6.4 Cardinality budget

The number of time series per metric is the product of distinct values of its attributes:

N_{\text{series}} = \prod_{i=1}^{k} |V_i|

Example: 5 methods x 40 routes x 12 status codes x 11 buckets x 30 pods gives about 792,000 series. Remove the pod attribute from histograms and aggregate at the collector to bring it under the budget of [n] series per service.

7. Traces

7.1 Instrumentation

Boundary	Span kind	Instrumentation	Required attributes
Incoming HTTP / gRPC	SERVER	Auto	`http.route`, `http.response.status_code`
Outgoing HTTP / gRPC	CLIENT	Auto	`server.address`, `http.response.status_code`
Database calls	CLIENT	Auto	`db.system`, `db.operation.name` (no full statements with values)
Message publish / consume	PRODUCER / CONSUMER	Manual or library	`messaging.system`, `messaging.destination.name`
Significant internal work	INTERNAL	Manual	Business attributes from 4.3

7.2 Context propagation

[State the propagation format and where it is applied. Example: "W3C Trace Context (traceparent, tracestate) on all HTTP and gRPC calls; W3C Baggage only for acme.tenant.tier; trace context copied into message headers for queues; batch jobs start a new trace and link to the triggering trace."]

The sequence below shows propagation across a synchronous call and an asynchronous queue.

7.3 Sampling

Rule	Where	Rate	Rationale
Head sampling, parent-based	SDK	10% of root traces	Bounded overhead
Keep all errors	Collector tail sampling	100%	Errors are rare and valuable
Keep slow traces	Collector tail sampling	100% above p99 latency threshold	Latency investigation
[Rule]	[Where]	[Rate]	[Rationale]

Effective sampled volume:

V_{\text{stored}} = V \times \left( p_{\text{head}} + (1 - p_{\text{head}}) \times p_{\text{tail}} \right)

where $p_{\text{tail}}$ is the share of unsampled traces kept by tail rules.

8. Dashboards

Dashboard	Audience	Required panels	Owner	Link
Service overview (one per service)	On-call	Rate, error ratio, p50/p95/p99 latency, saturation, SLO burn, deploy markers	Service team	[Link]
Dependency view	On-call	Outbound call latency and errors per dependency	Service team	[Link]
Platform health	SRE	Cluster utilisation, collector health, queue depth	Platform	[Link]
Business KPIs	Product	Orders, sign-ups, conversion	Product analytics	[Link]
[Dashboard]	[Audience]	[Panels]	[@team]	[Link]

Dashboards are stored as code in [repository path] and reviewed like any other change.

9. SLIs and SLOs

SLO ID	SLI	Metric query (summary)	Objective	Window	Error budget
SLO-CHK-01	Availability of `POST /orders`	Non-5xx count divided by total count from `http.server.request.duration`	99.9%	28 days	40.3 min
SLO-CHK-02	Latency of `POST /orders`	Share of requests under 500 ms	99%	28 days	1% of requests
[SLO-ID]	[SLI]	[Query]	[%]	[Window]	[Budget]

Burn rate, used by alerts in section 10:

\text{burn rate} = \frac{\text{observed error ratio}}{1 - \text{SLO}}

A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget.

10. Alerting

10.1 Severity levels

Severity	Meaning	Notification	Response
SEV1 / page	Users affected now or budget burning fast	Page primary on-call	Acknowledge within 5 min
SEV2 / ticket	Degradation or slow burn	Ticket and chat	Next working hours
Info	Awareness only	Chat channel	None required

10.2 Alert rules

ID	Alert	Condition	For	Severity	Routing	Runbook
AL-01	Example: Checkout fast burn	Burn rate > 14.4 over 1 h and > 14.4 over 5 min	2 min	Page	checkout on-call	[Runbook link]
AL-02	Example: Checkout slow burn	Burn rate > 1 over 3 days and > 1 over 6 h	1 h	Ticket	checkout team queue	[Runbook link]
AL-03	Example: Queue saturation	`acme.queue.depth` > 50,000	10 min	Page	fulfilment on-call	[Runbook link]
AL-04	Example: Collector dropping data	Exporter failures > 1%	15 min	Ticket	platform team	[Runbook link]
[AL-nn]	[Alert]	[Condition]	[Duration]	[Severity]	[Route]	[Link]

10.3 Alert hygiene

Review alert volume monthly; any alert that fired without action is tuned or removed.
Silence windows require an owner and expiry.
Target: fewer than [2] pages per on-call shift on average.

11. Privacy and security

ID	Rule	Enforcement point	Verification
PR-01	Never record passwords, tokens, keys or session cookies	SDK and collector redaction processor	Automated scan of sampled logs weekly
PR-02	Email addresses, phone numbers and card numbers are masked	Collector pattern-based scrubber	Unit tests on scrubber rules
PR-03	User identifiers are pseudonymised (`enduser.pseudo_id`)	SDK helper	Code review checklist
PR-04	Query strings are dropped from `url.full`	Collector attribute processor	Sample inspection
PR-05	Telemetry access is role-based and logged	Backend access control	Quarterly access review
[PR-nn]	[Rule]	[Where]	[How checked]

Data subject requests: [describe how telemetry is handled for deletion requests, for example "retention below 30 days for logs containing pseudonymous IDs, so data expires within the response deadline"].

12. Retention and cost

Signal	Hot retention	Archive retention	Volume per day	Unit cost	Monthly cost
Logs	Example: 14 days	90 days compressed	400 GB	[Amount per GB]	[Amount]
Metrics	Example: 30 days raw, 13 months downsampled	n/a	2 M active series	[Amount per series]	[Amount]
Traces	Example: 7 days	None	150 GB	[Amount per GB]	[Amount]
[Signal]	[Days]	[Days]	[Volume]	[Cost]	[Amount]

Monthly cost estimate per signal:

C = V_{\text{day}} \times 30 \times c_{\text{ingest}} + V_{\text{stored}} \times c_{\text{storage}}

Cost controls: [list sampling, log level defaults, attribute limits and budget alerts that keep spend within OG-03].

13. Ownership and governance

Area	Owner	Responsibilities
Telemetry pipeline and backends	Platform team	Availability, scaling, upgrades, scrubbing processors
Service instrumentation	Each service team	RED metrics, spans, logs per this spec; dashboards and alerts
Semantic conventions register (4.3)	Observability guild	Review new attributes and metric names
Privacy rules (11)	Security / privacy	Approve rules, audit compliance
[Area]	[@team]	[Responsibilities]

Changes to this specification follow [process, for example "pull request to the observability repository with review from two guild members"]. A new service is production-ready only when it passes the checklist in appendix 14.1.

14. Appendices

14.1 Service onboarding checklist

Resource attributes in 4.2 set and verified in each backend.
RED metrics exported for every entry point.
Trace context propagated on all inbound and outbound calls.
Logs follow schema 5.1 and include trace_id.
Service overview dashboard created from the standard template.
At least one SLO defined and burn-rate alerts configured with runbooks.
Privacy rules in section 11 verified by a sample review.

14.2 Glossary

Term	Meaning
OTLP	OpenTelemetry Protocol for transporting telemetry
Resource	The entity producing telemetry, described by attributes
Span	One timed operation within a trace
RED	Rate, Errors, Duration
USE	Utilisation, Saturation, Errors
Burn rate	Speed at which an error budget is consumed relative to the SLO

14.3 References

Reference	Use
OpenTelemetry specification and semantic conventions (opentelemetry.io)	Signals, attribute names, propagation
W3C Trace Context Recommendation	Propagation headers
[Internal SLO document]	Objectives in section 9

Standard and licence note

Based on OpenTelemetry concepts (signals, resources, semantic conventions, context propagation and sampling) and on common SRE practice for RED and USE metrics and burn-rate alerting. Attribute names shown follow the public OpenTelemetry semantic conventions; check the current version for changes. This template is original text written for this collection and reproduces no text from the specification. Template licence: MIT.

Related templates

Operations