# Observability Specification: [Service or platform name]

> Complete observability specification for a service or platform:
> goals, telemetry signals, naming and semantic conventions, log schema,
> metrics catalogue, tracing, dashboards, SLIs and SLOs, alert rules,
> retention, cost, privacy and ownership. Concepts follow OpenTelemetry
> (signals, resources, attributes, context propagation, sampling).
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Start from the questions operators must answer during an incident
  (section 1.2), then decide which signal answers each one.
- Reuse OpenTelemetry semantic conventions for attribute names before
  inventing your own. Custom names go under your own namespace.
- Every alert must link to a runbook and route to an owner. An alert
  without both is noise.
- Write the privacy rules (section 11) before shipping any log line that
  may contain user data.
-->

## Document control

<!-- Guidance: Observability specs are referenced by every team that emits telemetry; version them like an API. -->

| Field | Value |
|---|---|
| Document ID | [OBS-PLATFORM-001] |
| Services covered | [Service names or "all services on platform X"] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@observability-owner] |
| Classification | [Internal] |
| Last updated | YYYY-MM-DD |
| Related documents | [SLO document], [On-call handbook], [Runbooks], [Data protection policy] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| SRE lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Engineering lead | [@name] | [Decision] | YYYY-MM-DD |
| Security / privacy | [@name] | [Decision] | YYYY-MM-DD |
| FinOps | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Breaking changes to names or schemas need a migration note; dashboards and alerts depend on them. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Initial draft |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Renamed `app.user_id` to `enduser.pseudo_id`; added PII scrubbing rule PR-04 |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings below. -->

1. [Goals and scope](#1-goals-and-scope)
2. [Architecture and telemetry pipeline](#2-architecture-and-telemetry-pipeline)
3. [Signal standards](#3-signal-standards)
4. [Naming and semantic conventions](#4-naming-and-semantic-conventions)
5. [Logs](#5-logs)
6. [Metrics](#6-metrics)
7. [Traces](#7-traces)
8. [Dashboards](#8-dashboards)
9. [SLIs and SLOs](#9-slis-and-slos)
10. [Alerting](#10-alerting)
11. [Privacy and security](#11-privacy-and-security)
12. [Retention and cost](#12-retention-and-cost)
13. [Ownership and governance](#13-ownership-and-governance)
14. [Appendices](#14-appendices)

## 1. Goals and scope

<!-- Guidance: State what observability must achieve for this system in measurable terms, and which services and environments are covered. -->

### 1.1 Goals

| ID | Goal | Measure |
|---|---|---|
| OG-01 | *Example:* Detect user-facing failures before customers report them | 90% of incidents detected by alert first |
| OG-02 | *Example:* Locate the failing component quickly | Median time to identify root service < 10 min |
| OG-03 | *Example:* Keep telemetry cost proportional to traffic | Telemetry spend < 8% of infrastructure spend |
| [OG-nn] | [Goal] | [Measure] |

### 1.2 Operator questions

The signals in this specification must answer these questions without
deploying new code:

- Is the service healthy for users right now, and since when has it not been?
- Which endpoint, region, version or customer segment is affected?
- Which downstream dependency is slow or failing?
- What changed just before the problem started (deploys, config, traffic)?
- [Question]

### 1.3 Scope

| In scope | Out of scope |
|---|---|
| *Example:* All production and staging services on [platform] | Developer laptops |
| *Example:* Managed databases and queues via provider metrics | Business analytics events (see analytics spec) |
| [Item] | [Item] |

## 2. Architecture and telemetry pipeline

<!-- Guidance: Show how telemetry flows from code to storage. Name the SDKs, collectors, processors and backends. -->

Services emit telemetry with the OpenTelemetry SDK through the OTLP
protocol to a local collector, which processes, scrubs and exports it to
the storage backends.

```mermaid
flowchart LR
  S1["Service A (OTel SDK)"] --> C["Collector agent"]
  S2["Service B (OTel SDK)"] --> C
  C --> G["Collector gateway: batch, scrub, sample"]
  G --> L["Log store"]
  G --> M["Metrics store"]
  G --> T["Trace store"]
  M --> AL["Alert manager"]
  AL --> P["Paging and chat"]
```

| Component | Technology | Version | Owner | Notes |
|---|---|---|---|---|
| SDK | *Example:* OpenTelemetry SDK for [language] | [x.y] | Each service team | Auto plus manual instrumentation |
| Collector | *Example:* OpenTelemetry Collector | [x.y] | Platform team | Agent per node, gateway per region |
| Log backend | [Product] | [x.y] | Platform team | [Notes] |
| Metrics backend | [Product] | [x.y] | Platform team | [Notes] |
| Trace backend | [Product] | [x.y] | Platform team | [Notes] |

## 3. Signal standards

<!-- Guidance: Define what each signal is for so teams do not log what should be a metric, or count what should be traced. -->

| Signal | Use it for | Do not use it for | Required for every service |
|---|---|---|---|
| Metrics | Aggregated rates, ratios, latency distributions, saturation; alerting | Per-request detail, high-cardinality IDs | RED metrics for every entry point; USE metrics for every resource |
| Traces | Following one request across services; latency breakdown | Long-term trend analysis | Server and client spans for every network call |
| Logs | Discrete events with context: errors, state changes, audit | Counting things that should be metrics | Structured JSON per 5.1, correlated by trace ID |
| [Profiles / events] | [Use] | [Not for] | [Requirement] |

All three signals carry the same resource attributes (4.2) so that a
metric spike can be followed to traces and then to logs.

## 4. Naming and semantic conventions

<!-- Guidance: Consistent names are what make cross-service queries possible. Adopt OpenTelemetry semantic conventions first; document every custom attribute here. -->

### 4.1 General rules

- Attribute and metric names are lowercase, dot-separated namespaces: `http.request.method`, `db.system`.
- Custom attributes use the organisation namespace: `[org].[domain].[name]`, for example `acme.order.channel`.
- Units follow UCUM-style abbreviations in metric metadata (`s`, `By`, `{request}`), not in the name.
- Durations are recorded in seconds as floating point.
- Never put unbounded values (user IDs, URLs with IDs, error messages) into metric attributes.

### 4.2 Required resource attributes

| Attribute | Example | Purpose |
|---|---|---|
| `service.name` | `checkout-api` | Identifies the service |
| `service.version` | `2026.10.3` | Correlates with deploys |
| `service.namespace` | `commerce` | Groups services by domain |
| `deployment.environment.name` | `production` | Separates environments |
| `cloud.region` | `eu-west-1` | Locates failures by region |
| `k8s.pod.name` or `host.name` | `checkout-api-7d9f` | Finds the instance |
| [Attribute] | [Example] | [Purpose] |

### 4.3 Custom attributes register

| Attribute | Type | Allowed values | Signals | Cardinality | Owner |
|---|---|---|---|---|---|
| `acme.order.channel` | string | `web`, `ios`, `android`, `partner` | metrics, traces, logs | 4 | [@team] |
| `acme.tenant.tier` | string | `free`, `pro`, `enterprise` | metrics, traces | 3 | [@team] |
| [name] | [type] | [values] | [signals] | [n] | [@team] |

## 5. Logs

<!-- Guidance: Define one schema for all services. Structured logs with consistent fields are cheaper to store and much faster to search. -->

### 5.1 Log schema

| Field | Type | Required | Description | Example |
|---|---|---|---|---|
| `timestamp` | RFC 3339 string, UTC | Yes | Time the event occurred | `2026-10-21T09:14:03.512Z` |
| `severity_text` | string | Yes | One of DEBUG, INFO, WARN, ERROR, FATAL | `ERROR` |
| `body` | string | Yes | Human-readable message without variable data where possible | `payment authorisation failed` |
| `trace_id` | hex string | When in a request | Correlates with traces | `4bf92f3577b34da6a3ce929d0e0e4736` |
| `span_id` | hex string | When in a request | Correlates with the span | `00f067aa0ba902b7` |
| `service.name` | string | Yes | From resource attributes | `checkout-api` |
| `event.name` | string | For domain events | Stable event identifier | `payment.authorisation.failed` |
| `error.type` | string | On errors | Error class or code | `card_declined` |
| `attributes` | object | No | Additional context per 4.3 | `{"acme.order.channel":"web"}` |
| [field] | [type] | [Yes / No] | [Description] | [Example] |

### 5.2 Severity guidelines

| Level | Use when | Pages anyone? |
|---|---|---|
| DEBUG | Detail useful only during development; disabled in production by default | No |
| INFO | Normal state changes: started, configuration loaded, job completed | No |
| WARN | Unexpected but handled: retry succeeded, fallback used | No |
| ERROR | A request or job failed and the caller saw a failure | Only via metric-based alerts |
| FATAL | The process cannot continue | Via health checks |

### 5.3 Logging rules

- Log once per failure at the boundary that handles it, not at every layer.
- Do not log request or response bodies by default.
- Sample high-volume INFO logs at the collector when volume exceeds [n] lines per second per service.

## 6. Metrics

<!-- Guidance: Every service exposes RED metrics for its entry points and USE metrics for its resources. List them in a catalogue so dashboards and alerts can rely on them. -->

### 6.1 Method

- **RED** for request-driven components: Rate, Errors, Duration.
- **USE** for resources: Utilisation, Saturation, Errors.

### 6.2 Metrics catalogue

| Metric name | Instrument | Unit | Attributes | Method | Description | Owner |
|---|---|---|---|---|---|---|
| `http.server.request.duration` | Histogram | s | `http.request.method`, `http.route`, `http.response.status_code` | RED (rate, errors, duration) | Server request latency; count gives rate | Service team |
| `http.client.request.duration` | Histogram | s | `server.address`, `http.response.status_code` | RED | Outbound call latency | Service team |
| `db.client.operation.duration` | Histogram | s | `db.system`, `db.operation.name` | RED | Database call latency | Service team |
| `process.cpu.utilization` | Gauge | 1 | `process.cpu.state` | USE | CPU utilisation per process | Platform |
| `acme.queue.depth` | Gauge | {message} | `messaging.destination.name` | USE (saturation) | Messages waiting | [@team] |
| `acme.orders.placed` | Counter | {order} | `acme.order.channel` | Business | Orders successfully placed | [@team] |
| [name] | [Counter / Histogram / Gauge / UpDownCounter] | [unit] | [attributes] | [RED / USE / Business] | [Description] | [@team] |

### 6.3 Histogram buckets

[Define standard bucket boundaries or use exponential histograms.
Example: latency buckets 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5,
5, 10 seconds.]

### 6.4 Cardinality budget

The number of time series per metric is the product of distinct values
of its attributes:

$$
N_{\text{series}} = \prod_{i=1}^{k} |V_i|
$$

*Example:* 5 methods x 40 routes x 12 status codes x 11 buckets x 30 pods
gives about 792,000 series. Remove the pod attribute from histograms and
aggregate at the collector to bring it under the budget of [n] series per
service.

## 7. Traces

<!-- Guidance: Decide how context crosses every boundary (HTTP, queues, batch jobs) and how much you keep. Broken propagation makes traces useless. -->

### 7.1 Instrumentation

| Boundary | Span kind | Instrumentation | Required attributes |
|---|---|---|---|
| Incoming HTTP / gRPC | SERVER | Auto | `http.route`, `http.response.status_code` |
| Outgoing HTTP / gRPC | CLIENT | Auto | `server.address`, `http.response.status_code` |
| Database calls | CLIENT | Auto | `db.system`, `db.operation.name` (no full statements with values) |
| Message publish / consume | PRODUCER / CONSUMER | Manual or library | `messaging.system`, `messaging.destination.name` |
| Significant internal work | INTERNAL | Manual | Business attributes from 4.3 |

### 7.2 Context propagation

[State the propagation format and where it is applied. Example: "W3C
Trace Context (`traceparent`, `tracestate`) on all HTTP and gRPC calls;
W3C Baggage only for `acme.tenant.tier`; trace context copied into
message headers for queues; batch jobs start a new trace and link to the
triggering trace."]

The sequence below shows propagation across a synchronous call and an
asynchronous queue.

```mermaid
sequenceDiagram
  participant B as Browser
  participant A as Checkout API
  participant P as Payment service
  participant Q as Order queue
  participant W as Fulfilment worker
  B->>A: POST /orders (traceparent)
  A->>P: Authorise (traceparent, child span)
  P-->>A: Approved
  A->>Q: Publish order (traceparent in headers)
  A-->>B: 201 Created
  Q->>W: Consume (span links to producer)
```

### 7.3 Sampling

| Rule | Where | Rate | Rationale |
|---|---|---|---|
| Head sampling, parent-based | SDK | 10% of root traces | Bounded overhead |
| Keep all errors | Collector tail sampling | 100% | Errors are rare and valuable |
| Keep slow traces | Collector tail sampling | 100% above p99 latency threshold | Latency investigation |
| [Rule] | [Where] | [Rate] | [Rationale] |

Effective sampled volume:

$$
V_{\text{stored}} = V \times \left( p_{\text{head}} + (1 - p_{\text{head}}) \times p_{\text{tail}} \right)
$$

where $p_{\text{tail}}$ is the share of unsampled traces kept by tail
rules.

## 8. Dashboards

<!-- Guidance: Few, standard dashboards beat many personal ones. Define the hierarchy and the panels every service must have. -->

| Dashboard | Audience | Required panels | Owner | Link |
|---|---|---|---|---|
| Service overview (one per service) | On-call | Rate, error ratio, p50/p95/p99 latency, saturation, SLO burn, deploy markers | Service team | [Link] |
| Dependency view | On-call | Outbound call latency and errors per dependency | Service team | [Link] |
| Platform health | SRE | Cluster utilisation, collector health, queue depth | Platform | [Link] |
| Business KPIs | Product | Orders, sign-ups, conversion | Product analytics | [Link] |
| [Dashboard] | [Audience] | [Panels] | [@team] | [Link] |

Dashboards are stored as code in [repository path] and reviewed like
any other change.

## 9. SLIs and SLOs

<!-- Guidance: Link each SLO to the exact metric query that measures it. Detail lives in the SLO document; this section proves the telemetry exists. -->

| SLO ID | SLI | Metric query (summary) | Objective | Window | Error budget |
|---|---|---|---|---|---|
| SLO-CHK-01 | Availability of `POST /orders` | Non-5xx count divided by total count from `http.server.request.duration` | 99.9% | 28 days | 40.3 min |
| SLO-CHK-02 | Latency of `POST /orders` | Share of requests under 500 ms | 99% | 28 days | 1% of requests |
| [SLO-ID] | [SLI] | [Query] | [%] | [Window] | [Budget] |

Burn rate, used by alerts in section 10:

$$
\text{burn rate} = \frac{\text{observed error ratio}}{1 - \text{SLO}}
$$

A burn rate of 14.4 sustained for one hour consumes 2% of a 30-day budget.

## 10. Alerting

<!-- Guidance: Alert on symptoms users feel (SLO burn) first, causes second. Each rule needs a severity, a route and a runbook. -->

### 10.1 Severity levels

| Severity | Meaning | Notification | Response |
|---|---|---|---|
| SEV1 / page | Users affected now or budget burning fast | Page primary on-call | Acknowledge within 5 min |
| SEV2 / ticket | Degradation or slow burn | Ticket and chat | Next working hours |
| Info | Awareness only | Chat channel | None required |

### 10.2 Alert rules

| ID | Alert | Condition | For | Severity | Routing | Runbook |
|---|---|---|---|---|---|---|
| AL-01 | *Example:* Checkout fast burn | Burn rate > 14.4 over 1 h and > 14.4 over 5 min | 2 min | Page | checkout on-call | [Runbook link] |
| AL-02 | *Example:* Checkout slow burn | Burn rate > 1 over 3 days and > 1 over 6 h | 1 h | Ticket | checkout team queue | [Runbook link] |
| AL-03 | *Example:* Queue saturation | `acme.queue.depth` > 50,000 | 10 min | Page | fulfilment on-call | [Runbook link] |
| AL-04 | *Example:* Collector dropping data | Exporter failures > 1% | 15 min | Ticket | platform team | [Runbook link] |
| [AL-nn] | [Alert] | [Condition] | [Duration] | [Severity] | [Route] | [Link] |

### 10.3 Alert hygiene

- Review alert volume monthly; any alert that fired without action is tuned or removed.
- Silence windows require an owner and expiry.
- Target: fewer than [2] pages per on-call shift on average.

## 11. Privacy and security

<!-- Guidance: Telemetry often leaks personal data and secrets. Define what must never be recorded and how it is removed before storage. -->

| ID | Rule | Enforcement point | Verification |
|---|---|---|---|
| PR-01 | Never record passwords, tokens, keys or session cookies | SDK and collector redaction processor | Automated scan of sampled logs weekly |
| PR-02 | Email addresses, phone numbers and card numbers are masked | Collector pattern-based scrubber | Unit tests on scrubber rules |
| PR-03 | User identifiers are pseudonymised (`enduser.pseudo_id`) | SDK helper | Code review checklist |
| PR-04 | Query strings are dropped from `url.full` | Collector attribute processor | Sample inspection |
| PR-05 | Telemetry access is role-based and logged | Backend access control | Quarterly access review |
| [PR-nn] | [Rule] | [Where] | [How checked] |

Data subject requests: [describe how telemetry is handled for deletion
requests, for example "retention below 30 days for logs containing
pseudonymous IDs, so data expires within the response deadline"].

## 12. Retention and cost

<!-- Guidance: Set retention by signal and value. Estimate cost from volume so growth does not surprise anyone. -->

| Signal | Hot retention | Archive retention | Volume per day | Unit cost | Monthly cost |
|---|---|---|---|---|---|
| Logs | *Example:* 14 days | 90 days compressed | 400 GB | [Amount per GB] | [Amount] |
| Metrics | *Example:* 30 days raw, 13 months downsampled | n/a | 2 M active series | [Amount per series] | [Amount] |
| Traces | *Example:* 7 days | None | 150 GB | [Amount per GB] | [Amount] |
| [Signal] | [Days] | [Days] | [Volume] | [Cost] | [Amount] |

Monthly cost estimate per signal:

$$
C = V_{\text{day}} \times 30 \times c_{\text{ingest}} + V_{\text{stored}} \times c_{\text{storage}}
$$

Cost controls: [list sampling, log level defaults, attribute limits and
budget alerts that keep spend within OG-03].

## 13. Ownership and governance

<!-- Guidance: Say who owns the pipeline, who owns each service's telemetry and how changes to this spec are made. -->

| Area | Owner | Responsibilities |
|---|---|---|
| Telemetry pipeline and backends | Platform team | Availability, scaling, upgrades, scrubbing processors |
| Service instrumentation | Each service team | RED metrics, spans, logs per this spec; dashboards and alerts |
| Semantic conventions register (4.3) | Observability guild | Review new attributes and metric names |
| Privacy rules (11) | Security / privacy | Approve rules, audit compliance |
| [Area] | [@team] | [Responsibilities] |

Changes to this specification follow [process, for example "pull
request to the observability repository with review from two guild
members"]. A new service is production-ready only when it passes the
checklist in appendix 14.1.

## 14. Appendices

<!-- Guidance: Checklists, glossary and references that support the spec. -->

### 14.1 Service onboarding checklist

- [ ] Resource attributes in 4.2 set and verified in each backend.
- [ ] RED metrics exported for every entry point.
- [ ] Trace context propagated on all inbound and outbound calls.
- [ ] Logs follow schema 5.1 and include `trace_id`.
- [ ] Service overview dashboard created from the standard template.
- [ ] At least one SLO defined and burn-rate alerts configured with runbooks.
- [ ] Privacy rules in section 11 verified by a sample review.

### 14.2 Glossary

| Term | Meaning |
|---|---|
| OTLP | OpenTelemetry Protocol for transporting telemetry |
| Resource | The entity producing telemetry, described by attributes |
| Span | One timed operation within a trace |
| RED | Rate, Errors, Duration |
| USE | Utilisation, Saturation, Errors |
| Burn rate | Speed at which an error budget is consumed relative to the SLO |

### 14.3 References

| Reference | Use |
|---|---|
| OpenTelemetry specification and semantic conventions (opentelemetry.io) | Signals, attribute names, propagation |
| W3C Trace Context Recommendation | Propagation headers |
| [Internal SLO document] | Objectives in section 9 |

## Standard and licence note

Based on OpenTelemetry concepts (signals, resources, semantic
conventions, context propagation and sampling) and on common SRE
practice for RED and USE metrics and burn-rate alerting. Attribute names
shown follow the public OpenTelemetry semantic conventions; check the
current version for changes. This template is original text written for
this collection and reproduces no text from the specification. Template
licence: MIT.