# System Design: [Metrics and monitoring platform name]

> Reference design for a multi-tenant time-series metrics and alerting
> platform: ingestion over OTLP and Prometheus remote write, sharded in-memory
> ingesters with a write-ahead log, compressed blocks in object storage,
> downsampling and retention tiers, cardinality control, rule evaluation and
> alert routing. It follows the OpenTelemetry metrics data model (stable
> sections as of OpenTelemetry Specification v1.62.0, October 2026) and the
> compression ideas published in the Gorilla paper (PVLDB 8(12), 2015). It
> follows the structure of the System Design Document template, adds a
> 45-minute interview path, access-pattern and failure-mode tables and a cited
> "How real systems do it" box, and is pre-filled with a worked design to
> adapt. Replace every `[bracketed]` value and every row marked *Example*;
> values marked *Example* are assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Count active series before anything else. Series count, not request rate,
  sizes memory, index and cost (section 3).
- Write the retention table (7.4) with the people who read old dashboards;
  most teams keep far more raw data than anyone queries.
- Decide early what happens when a tenant exceeds its series limit (7.3):
  drop new series, drop the whole write, or aggregate away a label.
- The monitoring system must be monitored by something that does not share
  its failure modes (7.5, "watching the watcher").
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-MON-001] |
| System / feature | [Metrics storage, query and alerting platform] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Difficulty | Advanced |
| Data model version implemented | [OpenTelemetry Specification v1.62.0, metrics data model] |
| Related | [Observability specification], [Alerting policy], [ADR: series limits] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| FinOps (storage cost) | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added per-tenant series limits after a label explosion incident |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Moved 5-minute rollups to object storage, raw retention cut to 15 days |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Each product team runs its own metrics server. Retention differs
from 3 to 90 days, there is no global view across clusters, and two outages
last quarter went unnoticed for 20 minutes because the metrics server of the
affected cluster failed together with the cluster. One team added a request
ID as a label, created 14 million series in an hour and took down the shared
server for everyone.]

### 1.2 Proposed solution

[*Example:* A central, multi-tenant platform. Agents and SDKs push metrics
over OTLP or Prometheus remote write to a stateless distributor, which
validates them, enforces per-tenant limits and hashes each series to three
ingesters. Ingesters keep the last two hours in compressed memory blocks with
a write-ahead log, then flush immutable blocks to object storage. A compactor
merges blocks and writes 5-minute and 1-hour rollups. A query frontend splits,
caches and routes queries to ingesters (recent data) and store gateways (old
data). A rule evaluator runs recording and alerting rules and sends alerts to
an alert router that deduplicates, groups and notifies. A separate, small
meta-monitoring stack watches the platform itself.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Push or pull ingestion? | Push (OTLP, remote write) into the platform; pull scraping stays inside each cluster agent | 7.1, 10 |
| Q-02 | *Example:* What happens over the series limit? | Reject new series for that tenant with a clear error; existing series keep flowing | 7.3 |
| Q-03 | *Example:* Raw retention? | 15 days raw, 13 months at 5 minutes, 5 years at 1 hour | 7.4 |
| Q-04 | *Example:* Delta or cumulative temporality at storage? | Store cumulative; convert delta streams at the distributor | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: which signals (metrics only, not logs or traces), push or pull, who queries, alerting in or out; below-the-line items (2.3) | Three to five requirements |
| 5 to 10 | Estimates: active series, samples per second, bytes per sample, memory for the recent window (section 3) | Four numbers and "series count drives memory" |
| 10 to 15 | Interface: write call, query call, alert rule (section 4) | Two calls and one rule |
| 15 to 25 | High-level design: distributor, ingesters, object storage, querier, rule evaluator (section 6) | One diagram |
| 25 to 40 | One deep dive: compression and the in-memory window (default, 7.2), or cardinality limits (7.3), or downsampling (7.4), or alert evaluation (7.5) | Options, choice, reason |
| 40 to 45 | Failures (ingester crash, cardinality explosion, monitoring down during an outage), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer: store each sample as a
row of (series ID, timestamp, value) in a key-value store, 16 bytes per point
before indexes. Show that at 6.7 million samples per second this is about
9.2 TB a day, then introduce delta-of-delta timestamps and XOR values and the
two-hour in-memory block.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Accept metrics over OTLP (HTTP and gRPC) and Prometheus remote write, per tenant | Must |
| FR-02 | *Example:* Support counters (sums), gauges, histograms (explicit and exponential buckets) | Must |
| FR-03 | *Example:* Query with a PromQL-compatible language over any time range within retention | Must |
| FR-04 | *Example:* Evaluate recording and alerting rules every 30 s and route alerts to on-call tools | Must |
| FR-05 | *Example:* Enforce per-tenant limits on active series, samples per second and labels per series | Must |
| FR-06 | *Example:* Downsample to 5-minute and 1-hour resolution and apply retention per tier | Must |
| FR-07 | *Example:* Show per-tenant usage (series, samples, top metrics by series count) | Should |
| FR-08 | *Example:* Silence and inhibit alerts with an audit trail | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Ingestion availability | *Example:* 99.9% of write requests accepted per month | Distributor success ratio |
| NFR-02 | Freshness | *Example:* A sample is queryable within 30 s of receipt at p99 | Synthetic canary series |
| NFR-03 | Query latency | *Example:* p95 at or below 2 s for a 24-hour dashboard panel over 1 000 series | Query frontend timing |
| NFR-04 | Alert latency | *Example:* Alert notification within 90 s of the condition becoming true, plus the rule `for` duration | Canary alert end to end |
| NFR-05 | Durability | *Example:* No loss of acknowledged samples on a single ingester crash | Chaos test |
| NFR-06 | Isolation | *Example:* One tenant over its limits cannot raise write latency for others | Noisy-tenant load test |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* logs and traces (separate pipelines; exemplars link to trace IDs only)]
- [*Example:* anomaly detection and forecasting on metrics]
- [*Example:* per-user product analytics (event data, not metrics)]
- [*Example:* billing-grade usage metering (needs exactly-once accounting)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. Every number should end in a consequence. -->

### 3.1 Assumptions

| Input | Symbol | Value | Source |
|---|---|---|---|
| Active series (all tenants) | $S$ | *Example:* 100 million | Sum of current servers' head series x growth |
| Collection interval | $I$ | *Example:* 15 s | Agent default |
| Uncompressed sample (8-byte timestamp, 8-byte float) | $b_u$ | 16 bytes | Data type sizes |
| Compressed sample in memory | $b_c$ | *Example:* 1.37 bytes | Gorilla paper average, check on your data |
| Compressed sample on disk including index | $b_d$ | *Example:* 2 bytes | Measured on a sample block |
| Head memory per active series (labels, index, chunks) | $m$ | *Example:* 4 KiB | Measured on target ingester build |
| Usable memory per ingester | $M_i$ | *Example:* 48 GiB | 64 GiB instance minus runtime overhead |
| Replication factor for ingesters | $RF$ | 3 | Survive one ingester loss with quorum writes |
| Alerting and recording rules | $Q$ | *Example:* 50 000 | Current rule files x growth |

### 3.2 Ingestion rate

$$
r = \frac{S}{I} = \frac{10^{8}}{15} \approx 6.67 \times 10^{6} \ \text{samples/s}, \qquad
r_{day} = r \times 86\,400 = 5.76 \times 10^{11} \ \text{samples/day}
$$

With replication each sample is written to three ingesters, so the ingester
tier absorbs $3r = 2 \times 10^{7}$ sample appends per second. At an assumed
10 bytes per sample on the wire (protobuf plus snappy) the distributors
receive about 66.7 MB/s. Consequence: the write path is CPU-bound on
decompression and hashing, not network-bound, so distributors scale on CPU.

### 3.3 Storage and retention

| Data | Formula | Per day | Retention | Total |
|---|---|---|---|---|
| Raw, uncompressed (for comparison) | $r_{day} \times 16$ B | 9.2 TB | none | not stored |
| Raw, compressed in memory | $r_{day} \times 1.37$ B | 789 GB | 2 h in memory | see 3.4 |
| Raw blocks on object storage | $r_{day} \times 2$ B | 1.15 TB | 15 days | 17.3 TB |
| 5-minute rollups (5 aggregates, 10 B per rollup) | $S \times 288 \times 10$ B | 288 GB | 13 months (395 days) | 113.8 TB |
| 1-hour rollups (5 aggregates, 10 B per rollup) | $S \times 24 \times 10$ B | 24 GB | 5 years (1 826 days) | 43.8 TB |
| **Total in object storage** | | | | **about 175 TB** |

Compression cuts the raw stream by $16 / 1.37 \approx 11.7$ times, which is
what makes an in-memory recent window affordable. Rollups, not raw data, are
90% of the stored bytes, so the retention of the 5-minute tier is the main
cost lever (section 9.4). Consequence: raw data goes to object storage and
the 13-month tier is reviewed with dashboard owners before anyone asks for
longer raw retention.

### 3.4 Memory for the recent window

$$
M_{head} = S \times m \times RF = 10^{8} \times 4 \ \text{KiB} \times 3 \approx 1\,144 \ \text{GiB}
$$

$$
N_{ingesters} = \frac{1\,144}{48} \times 1.25 \approx 23.8 \times 1.25 \Rightarrow 30 \ \text{ingesters}
$$

The factor 1.25 is headroom for uneven series distribution and for series
churn (pods restarting create new series while old ones are still in the
head). Each ingester appends about $2 \times 10^{7} / 30 \approx 667\,000$
samples per second. Consequence: memory per series, not samples per second,
decides the ingester count, so the series limit in 7.3 is a capacity control,
not only a safety net.

### 3.5 Queries and rules

- Rule evaluations: $Q / 30 = 50\,000 / 30 \approx 1\,667$ per second,
  mostly over the last few minutes, so they are served from ingesters.
- A 7-day panel at 15 s resolution reads 40 320 points per series; at
  5-minute resolution it reads 2 016, 20 times fewer. A 1 000-series panel at
  raw resolution reads 40.3 million points. Consequence: the query frontend
  routes ranges longer than 2 days to the 5-minute tier automatically.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Samples | about 6.7 million/s (20 million appends/s with RF 3) | Stateless distributors scaled on CPU; hash by series |
| Recent window memory | about 1.1 TiB with RF 3 | 30 ingesters with 48 GiB usable each |
| Object storage | about 175 TB, 90% rollups | Retention of the 5-minute tier is the cost lever |
| Rule evaluations | about 1 700/s | Rules read from ingesters; sharded rule evaluators |
| Long-range queries | 20x fewer points from 5-minute rollups | Automatic resolution selection in the frontend |

## 4. API design

<!-- Guidance: Define the contract first: ingestion endpoints, query endpoints, rule and alert management, errors and limits. Use RFC 9457 problem details for errors. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Ingestion | `POST /v1/metrics` (OTLP/HTTP, protobuf) and OTLP/gRPC `MetricsService/Export` | Push metrics from SDKs and collectors |
| Ingestion | `POST /api/v1/push` (Prometheus remote write) | Push from Prometheus-compatible agents |
| Query | `GET /api/v1/query?query=&time=` and `/api/v1/query_range?query=&start=&end=&step=` | Instant and range queries |
| Metadata | `GET /api/v1/series`, `/api/v1/labels`, `/api/v1/label/{name}/values` | Discovery for dashboards |
| Rules | `PUT /api/v1/rules/{namespace}` | Create or replace a rule group (versioned) |
| Alerts | `GET /api/v1/alerts`, `POST /api/v1/silences` | Active alerts and silences |
| Usage | `GET /api/v1/tenants/{id}/usage` | Series count, samples/s, top metrics by series |

Every request carries a tenant identity derived from the authenticated
credential, never from a free-text header set by the client.

### 4.2 Write responses and limits

A write that would create series beyond the tenant's limit is partly
accepted: existing series are stored, new series are dropped, and the
response says so. Because the stored part must not be sent again, this is
not a retryable error. On OTLP the response is `200 OK` with the
`partial_success` field filled in (`rejected_data_points` and
`error_message`), which the OTLP specification says clients must not retry;
on Prometheus remote write it is a `400`, which senders do not retry. Keep
`429` with `Retry-After` for the samples-per-second rate limit (7.3), where
the whole request was refused and a later retry is correct. *Example* OTLP
response body, shown in the OTLP JSON encoding:

```json
{"partialSuccess": {"rejectedDataPoints": "1240", "errorMessage": "Tenant team-checkout has 5000000 active series (limit 5000000). 1240 data points for new series were dropped; 18760 data points for existing series were stored. Retrying will not help: reduce label values or ask for a higher limit."}}
```

### 4.3 Alert rule format

Rules are versioned documents; a new rule group starts in `shadow` mode,
where it evaluates and records what it would have fired without notifying.

```json
{"namespace": "checkout", "group": "slo-burn", "interval_s": 30, "mode": "enforce", "rules": [{"alert": "CheckoutErrorBudgetFastBurn", "expr": "sum(rate(http_server_request_duration_seconds_count{service=\"checkout\",status_class=\"5xx\"}[5m])) / sum(rate(http_server_request_duration_seconds_count{service=\"checkout\"}[5m])) > 14.4 * 0.001", "for_s": 120, "labels": {"severity": "page"}, "annotations": {"runbook": "https://runbooks.example.com/checkout/error-budget"}}]}
```

The threshold `14.4 * 0.001` is a fast-burn alert for a 99.9% SLO: an error
ratio 14.4 times the budget spends 2% of a 30-day budget in one hour.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

A series is identified by its metric name plus the full set of label
(attribute) pairs, including resource attributes such as service and
cluster. Every distinct combination is a new series, which is why label
values must come from small, bounded sets.

```mermaid
erDiagram
  TENANT {
    string tenant_id PK
    int series_limit
    int samples_per_sec_limit
  }
  SERIES {
    bigint series_ref PK
    string tenant_id FK
    string metric_name
    string labels_hash
    string temporality
  }
  CHUNK {
    bigint series_ref FK
    bigint min_time
    bigint max_time
    bytes encoded
  }
  BLOCK {
    string block_id PK
    string tenant_id FK
    bigint min_time
    bigint max_time
    int resolution_s
  }
  RULE_GROUP {
    string namespace PK
    string group_name PK
    int version
    int interval_s
  }
  ALERT {
    string fingerprint PK
    string state
    datetime active_since
  }
  TENANT ||--o{ SERIES : owns
  SERIES ||--o{ CHUNK : "stored as"
  BLOCK ||--o{ CHUNK : contains
  TENANT ||--o{ RULE_GROUP : defines
  RULE_GROUP ||--o{ ALERT : produces
```

### 5.2 Storage choices

| Data | Store | Key | Main queries | Why |
|---|---|---|---|---|
| Recent samples (2 h) | *Example:* Ingester memory plus local write-ahead log | Series ref, time | Rules, live dashboards | Compressed chunks fit in memory; WAL replays after a crash |
| Older samples and rollups | Object storage, immutable blocks | Tenant, block time range, resolution | Long-range dashboards | Cheap, durable, no servers to scale for capacity |
| Block index (postings) | Inside each block plus a bucket index cached by store gateways | Label name and value to series refs | Find series matching a selector | Inverted index is what makes label queries fast |
| Rules, silences, limits | Relational database | Namespace, group, version | Load on change | Small, audited |
| Query results | Cache keyed by query, step-aligned range | Query hash, time split | Repeated dashboard loads | Dashboards refresh the same ranges every 30 s |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Append a sample | 6.7 million/s (20 million appends with RF 3) | Series ref in the ingester head | Hash of tenant plus full label set spreads series evenly across ingesters |
| Evaluate a rule over the last 5 minutes | about 1 700/s | Ingester head postings index | Recent data lives only in ingesters; queriers fan out to the replicas that own matching series |
| Dashboard panel over 24 hours | Thousands per minute | Ingesters plus the newest blocks, results cached per step-aligned split | Splitting by day makes cached splits reusable |
| Dashboard over 30 days | Hundreds per minute | 5-minute rollup blocks through store gateways | Rollups are partitioned by tenant and time, so a range touches few blocks |
| Find label values for a dropdown | Frequent | Postings index, label name to values | Served from index headers cached in store gateways |
| Usage report per tenant | Every minute | Ingester head series counters per tenant | Counted where series are created; no scan |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the write, read and alert paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  subgraph src["Sources"]
    sdk["OTel SDKs and collectors"]
    agent["Prometheus-compatible agents"]
  end
  sdk -->|OTLP| dist[Distributor]
  agent -->|remote write| dist
  dist -->|"hash ring, RF 3"| ing[(Ingesters with WAL)]
  ing -->|"2 h blocks"| obj[(Object storage)]
  comp[Compactor and downsampler] --> obj
  obj --> comp
  user([Dashboard user]) --> qf[Query frontend and cache]
  qf --> q[Querier]
  q --> ing
  q --> sg[Store gateway]
  sg --> obj
  ruler[Rule evaluator] --> q
  ruler -->|recorded series| dist
  ruler -->|alerts| ar[Alert router]
  ar --> notify[[Paging, chat, ticket]]
  meta[Meta-monitoring stack] -.->|probes| dist
  meta -.->|probes| ar
```

| Component | Responsibility | Scales by |
|---|---|---|
| Distributor | Authenticate tenant, validate, enforce limits, convert temporality, hash series to ingesters | CPU, stateless |
| Ingester | Hold the 2-hour head in memory, append to WAL, cut and upload blocks | Active series (memory) |
| Compactor and downsampler | Merge blocks, deduplicate replicas, write 5-minute and 1-hour rollups, apply retention | Block volume per tenant |
| Store gateway | Serve queries over blocks in object storage, cache index headers | Blocks queried |
| Query frontend | Split long queries by day, cache results, pick resolution, queue per tenant | Query rate |
| Querier | Execute the query language over ingesters and store gateways | Query complexity |
| Rule evaluator | Run recording and alerting rule groups on schedule | Number of rule groups |
| Alert router | Deduplicate, group, inhibit, silence and notify | Alerts per minute |
| Meta-monitoring stack | Watch the platform from outside its failure domain | Small, fixed |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Write path | Pull scraping by a central server, push to a key-value store, push to sharded in-memory ingesters | Push to ingesters with WAL and RF 3 | Recent data is read most; memory serves rules fast | 7.1 |
| Encoding | Row per sample, general-purpose compression, delta-of-delta and XOR chunks | Delta-of-delta timestamps, XOR values | About 1.37 bytes per sample with fast sequential decode | 7.2 |
| Cardinality | No limits, global limit, per-tenant and per-metric limits with drop of new series | Per-tenant and per-metric limits | Protects memory without losing existing series | 7.3 |
| Retention | Keep raw forever, fixed TTL, tiered rollups | Tiered rollups with correct aggregates | Long-range queries 20x cheaper; storage dominated by rollups | 7.4 |
| Alerting | Evaluate in each agent, central rule evaluator, streaming evaluation | Central evaluator with shadow mode and meta-monitoring | One place for rules, silences and audit | 7.5 |

### 7.1 Write path

1. The distributor authenticates the tenant and checks request size, label
   count (*Example:* at most 30 labels per series), label name syntax and
   value length (*Example:* 1 024 bytes).
2. It converts delta-temporality sums to cumulative per series (7.6) and
   rejects samples older than the head window or too far in the future.
3. It hashes tenant plus sorted label set onto a consistent-hash ring and
   sends the sample to the three ingesters that own the token. The write
   succeeds when two of three acknowledge.
4. Each ingester appends to its write-ahead log, then to the in-memory head.
   Every two hours it cuts an immutable block, uploads it and truncates the
   WAL once upload is confirmed.

```mermaid
sequenceDiagram
  participant A as Agent
  participant D as Distributor
  participant I1 as Ingester 1
  participant I2 as Ingester 2
  participant I3 as Ingester 3
  A->>D: POST /v1/metrics (tenant token, 20 000 samples)
  D->>D: validate, check series limit, hash series
  par replicate
    D->>I1: push samples for owned series
    D->>I2: push samples for owned series
    D->>I3: push samples for owned series
  end
  I1-->>D: ok (WAL appended)
  I2-->>D: ok (WAL appended)
  D-->>A: 200 after 2 of 3 acks
  I3-->>D: ok (late, still applied)
```

Because three replicas hold the same series, the compactor deduplicates
overlapping blocks from the replicas, keeping one sample per timestamp.

### 7.2 Compression and the in-memory window

The encoding follows the scheme published for Gorilla (paraphrased):

- **Timestamps** are stored as the difference of consecutive differences
  ("delta of delta"). For a series collected every 15 s the deltas are 15,
  15, 15, so the delta of delta is 0 and costs one bit. Small irregularities
  fall into short bit ranges with a short prefix; only large jumps use a long
  form.
- **Values** are XORed with the previous value. Identical values cost one
  bit; similar floats share sign, exponent and leading mantissa bits, so only
  the changed middle bits are stored with their position.

*Example:* timestamps 1 000, 1 015, 1 030, 1 045, 1 061 have deltas 15, 15,
15, 16 and deltas of delta 0, 0, 1. After the first two timestamps the next
three cost 1 + 1 + 9 bits (a 2-bit prefix and a 7-bit value for the 1), or
11 bits instead of 192 bits uncompressed.

Chunks are aligned to two-hour windows. Decoding is sequential, which suits
range scans and rules over the last minutes, and costs nothing for random
access because nobody reads one sample in the middle of a chunk. The paper
reports an average of 1.37 bytes per sample against 16 uncompressed; measure
your own ratio, since counters with many distinct values compress worse than
mostly constant gauges.

### 7.3 Cardinality control

Cardinality is the number of distinct series. It explodes multiplicatively:
*Example:* `http_server_request_duration_seconds` with 5 methods, 10 status
codes, 200 routes and 2 000 pods yields $5 \times 10 \times 200 \times 2\,000 = 2 \times 10^{7}$
series, 20% of the whole platform budget in one metric. Adding a user ID label
makes it unbounded. Prometheus naming guidance warns against labels with
many distinct values such as user IDs and email addresses; put those in logs
or traces and link them with exemplars.

| Limit | *Example* default | Behaviour when exceeded |
|---|---|---|
| Active series per tenant | 5 million | New series dropped, existing series stored; OTLP partial success or remote-write 400, not retried (4.2) |
| Active series per metric name per tenant | 500 000 | Same, plus usage report names the top labels by value count |
| Labels per series | 30 | Whole series rejected |
| Samples per second per tenant | 500 000, burst 1 million | Token bucket at the distributor, 429 with `Retry-After` |

Limits are counted in the ingesters, where series are created, and the
distributor reads them from a shared view refreshed every few seconds, so a
tenant can overshoot by the series created in one refresh interval.
Distributor-side relabelling rules let a tenant drop or aggregate a label
(for example remove `pod` and keep `deployment`) before series are created.

### 7.4 Downsampling and retention

| Tier | Resolution | Retention | Aggregates kept | Used for |
|---|---|---|---|---|
| Head | Native (15 s) | 2 h in memory | Raw samples | Rules, live dashboards |
| Raw blocks | Native | 15 days | Raw samples | Incident investigation |
| 5-minute rollups | 300 s | 13 months | min, max, sum, count, last (counters: increase over the interval) | Capacity trends, monthly reviews |
| 1-hour rollups | 3 600 s | 5 years | Same as 5-minute | Year-over-year planning |

Rollups must keep aggregates that can be combined again. An average of
averages is wrong when sample counts differ, so the store keeps sum and count
and computes the average at query time. Counters are rolled up as the
increase within the interval after handling resets, never as the raw counter
value. Histograms are rolled up by adding bucket counts, which is only valid
when bucket boundaries match; exponential histograms with different scales
are first converted to the coarser scale. Percentiles cannot be rolled up
from stored percentiles; keep the histogram.

### 7.5 Alert evaluation and watching the watcher

- Rule groups are sharded across evaluators by hash of namespace and group.
  Each group runs every `interval_s`; a missed evaluation is recorded as a
  metric and alerts on itself.
- An alert becomes pending when the expression returns a result and firing
  after it stays true for `for_s`. Alerts on absence (`absent()` or a canary
  series) catch a silent exporter.
- The alert router deduplicates by label fingerprint, groups related alerts
  into one notification, applies inhibition (a cluster-down alert suppresses
  its per-pod alerts) and silences with an owner and expiry.
- Meta-monitoring runs in a different region or provider with its own small
  storage. It probes the write path with a canary series, the query path and
  the alert path with a canary alert that must arrive at a test receiver
  every 5 minutes (a "dead man's switch"). If the canary stops arriving, the
  meta stack pages directly.

### 7.6 Temporality and the OpenTelemetry data model

The OpenTelemetry metrics data model defines Sum, Gauge, Histogram and
ExponentialHistogram point kinds, plus Summary kept for compatibility and
not recommended for new applications. Sums carry an aggregation temporality:
cumulative points repeat the same start time, delta points advance it. This
platform stores cumulative sums because rate functions and rollups expect
them. The distributor keeps the last cumulative total per delta series and
adds each delta to it; if the start time does not follow the previous end
time it records a reset. That state lives in the distributor that receives
the stream, so delta sources must stick to one distributor (by consistent
hashing on the series at the load balancer) or send cumulative data, which
is the safer default.

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). The Gorilla paper
> (Pelkonen and co-authors, PVLDB 8(12), 2015) describes the in-memory
> time-series store that sat in front of an HBase-backed monitoring system
> at a large social network. It argues that monitoring users care most about
> recent data and aggregates, so the store keeps the most recent 26 hours in
> memory, accepts losing a small amount of data on the write path in
> exchange for availability, and compresses timestamps with delta of delta
> and values with XOR. The authors report about 96% of timestamps compressed
> to one bit, roughly 51% of values to one bit, an average of 1.37 bytes per
> point against 16, and query latency 73 times lower than the previous
> system. Every write is streamed to two independent instances in separate
> regions, and queries move to the other region when one is unhealthy. Google's Monarch paper
> (Adams and co-authors, PVLDB 13(12), 2020) describes a planet-scale
> in-memory time-series system that keeps data in regional zones close to
> where it is produced, with global query and configuration planes on top,
> so a regional problem stays regional. This design keeps both lessons: the
> recent window in memory (7.2) and a monitoring path for the monitor that
> does not share its failure domain (7.5). Read the originals in 12.3.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Sample rate | Stateless distributors, hash ring | *Example:* about 667 000 appends/s per ingester at 30 ingesters |
| Active series | Memory per series, per-tenant limits | 48 GiB usable per ingester; add ingesters before 80% |
| Series churn | Short-lived pods create series that stay in the head until the next cut | Churn above 30% of head per 2 h doubles memory need; aggregate the pod label |
| Long-range queries | Day splits, result cache, rollup selection | Queries over 13 months at 5-minute resolution across 100 000 series |
| Index lookups | Postings in block index, index headers cached in store gateways | Regex selectors over high-cardinality labels |
| Rule load | Sharded evaluators, recording rules for expensive expressions | Rule groups slower than their interval |

Query latency budget for a 24-hour panel over 1 000 series (target p95 2 s):

| Step | Budget |
|---|---|
| Frontend split by day and cache lookup | 0.05 s |
| Querier fan-out to ingesters (last 2 h) | 0.3 s |
| Store gateway reads for the remaining 22 h | 0.9 s |
| Evaluation and merge | 0.35 s |
| Headroom | 0.4 s |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Ingester | Crash or node loss | Ring health, missing heartbeats | None for writes (2 of 3 quorum); queries read remaining replicas | WAL replay on restart; ring hands tokens to a new ingester | *Example:* 5 min to rejoin |
| Ingesters | Two replicas of the same token range down | Write quorum failures by token range | Writes fail for series whose three replicas include both failed ingesters, a small share of all series | Spread replicas across zones; zone-aware ring | *Example:* 10 min |
| Tenant | Cardinality explosion | Series creation rate per tenant | That tenant's new series | Per-tenant and per-metric limits (7.3) | Immediate (limit) |
| Object storage | Regional degradation | Upload error rate, block upload lag | Ingesters keep blocks locally longer; long-range queries fail | Local disk sized for 6 h of blocks; retry with backoff | *Example:* provider dependent |
| Compactor | Stuck or failing | Oldest uncompacted block age | Slower queries, higher storage cost | Alert at 24 h lag; compaction is idempotent and restartable | *Example:* 1 h |
| Rule evaluator | Missed evaluations | Missed iteration counter | Alerts in the affected rule groups delayed | Shard rebalancing; alert on missed iterations | *Example:* 2 min |
| Whole platform | Down during a production incident | Meta-monitoring dead man's switch | No dashboards or alerts for all tenants | Independent meta stack pages directly; cluster agents buffer locally | *Example:* 30 min |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security and privacy

| Threat | Control |
|---|---|
| Tenant reads another tenant's data | Tenant ID from the credential; every query and block path is scoped by tenant |
| Personal data in label values (emails, user IDs, IP addresses) | Distributor deny-list rules for known patterns; guidance to use logs or traces instead; deletion API per series |
| Write flooding as denial of service | Per-tenant sample rate limits and request size limits |
| Tampering with alert rules or silences | Role-based access, versioned rule groups, audit log of silences with owner and expiry |
| [Threat] | [Control] |

### 9.3 Observability of the platform

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Write success ratio | Accepted / received samples per tenant | Below 99.9% for 10 min |
| Freshness | Age of newest canary sample | Above 60 s |
| Head memory | Bytes in head per ingester | Above 80% of usable |
| Series creation rate | New series per tenant per minute | 10x the 7-day baseline |
| Block upload lag | Oldest block not yet uploaded | Above 4 h |
| Alert path | Canary alert received at test receiver | Missing for 10 min |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Ingesters | [$/node-month] | *Example:* 30 nodes with 64 GiB memory | [$] |
| Distributors, queriers, store gateways, rule evaluators | [$/node-month] | [Count] | [$] |
| Object storage | [$/TB-month] | *Example:* about 175 TB | [$] |
| Object storage requests | [$/million requests] | [Uploads plus query reads] | [$] |
| Meta-monitoring stack | [$/month] | 1 small deployment in another region | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Ingestion model | Central pull scraping; push to platform | Push to platform, pull only inside clusters | Works across networks and for short-lived jobs | The platform cannot see a target that silently stops sending (needs absence alerts) |
| Recent data | Disk-based TSDB; in-memory head with WAL | In-memory head | Rules and live dashboards read recent data most | Memory cost scales with active series |
| Storage of old data | Local disks on stateful nodes; object storage | Object storage | Capacity without servers, high durability | Higher first-byte latency, needs caching |
| Cardinality | Unlimited; global limit; per-tenant and per-metric limits | Per-tenant and per-metric | One team cannot harm the others | Teams must change instrumentation when they hit limits |
| Temporality | Store delta; store cumulative | Cumulative with conversion at the distributor | Rate and rollups expect cumulative | Stateful conversion for delta sources |
| Query language | New language; PromQL-compatible | PromQL-compatible | Existing dashboards and skills | Inherits its edge cases around staleness |
| What changes at 10x / 0.1x | 10x (1 billion active series, about 66.7 million samples/s): about 298 ingesters by the 3.4 formula, so split into independent cells of about 30 ingesters with tenants assigned to cells, and storage grows to about 1.75 PB. 0.1x (10 million series, about 667 000 samples/s): one replicated pair of single-node time-series servers with remote storage is simpler and cheaper | Review at each doubling of active series | Memory and cell count, not network, grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Write path, ingesters, object storage, query for two pilot tenants | Freshness and query targets met for 14 days | YYYY-MM-DD |
| M2 | Limits, usage API, rollups | Cardinality test tenant contained without effect on others | YYYY-MM-DD |
| M3 | Rule evaluator, alert router, meta-monitoring | Canary alert end-to-end within 90 s | YYYY-MM-DD |
| M4 | All tenants migrated, old servers retired | Dashboards and alerts parity signed off by each team | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Teams dual-write from their agents to the old server and the
platform for 30 days. Rule groups run in shadow mode on the platform and
their would-fire results are compared with the old alerts. Cut-over is per
team; rollback points dashboards and alert routing back to the old server,
which kept receiving data during dual-write.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Series churn from autoscaling doubles head memory | Medium | High | Aggregate pod labels at the distributor; churn alert | [@name] |
| R-02 | *Example:* Teams rely on 90-day raw retention for audits | Medium | Medium | Interview dashboard owners; audit data belongs in logs | [@name] |
| R-03 | *Example:* Remote write 2.0 is still an experimental specification | Medium | Low | Accept both 1.0 and 2.0 payloads; pin the revision in docs | [@name] |
| R-04 | *Example:* Shared platform becomes a single point of failure for all alerting | Low | High | Meta-monitoring in another failure domain; critical alerts also evaluated locally | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we accept native or exponential histograms from all agents, or convert at the distributor? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which tenants need data residency, and therefore a regional cell? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| Series | One metric name with one exact set of label values |
| Cardinality | Number of distinct series, overall or per metric |
| Head | The in-memory window of recent samples in an ingester |
| Block | An immutable file set covering a time range: chunks plus an index |
| Rollup | A lower-resolution copy that keeps combinable aggregates |
| Temporality | Whether a sum reports totals since start (cumulative) or changes since the last report (delta) |
| Dead man's switch | An alert that always fires; its absence means the alert path is broken |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| [OpenTelemetry Protocol (OTLP) specification](https://opentelemetry.io/docs/specs/otlp/) | Partial success and retry rules (4.2) |
| [OpenTelemetry metrics data model](https://opentelemetry.io/docs/specs/otel/metrics/data-model/), OpenTelemetry Specification (latest release v1.62.0, October 2026; page status mixed, core sections stable) | Point kinds, temporality, Summary as legacy (7.6) |
| [Prometheus metric and label naming](https://prometheus.io/docs/practices/naming/) | Cardinality guidance (7.3) |
| [Prometheus Remote-Write 2.0 specification](https://prometheus.io/docs/specs/prw/remote_write_spec_2_0/), marked experimental, 2.0-rc.6 as of October 2026 | Ingestion interface (4.1) |
| Pelkonen, Franklin, Teller, Cavallaro, Huang, Meza, Veeraraghavan, ["Gorilla: A Fast, Scalable, In-Memory Time Series Database"](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf), PVLDB 8(12): 1816 to 1827, 2015, doi:10.14778/2824032.2824078 | 7.2, 7.7 |
| Adams and co-authors, ["Monarch: Google's Planet-Scale In-Memory Time Series Database"](https://www.vldb.org/pvldb/vol13/p3181-adams.pdf), PVLDB 13(12): 3181 to 3194, 2020 | 7.7 |
| RFC 9457 Problem Details for HTTP APIs | Error bodies for refused requests, such as the rate-limit 429 (7.3) |
| Observability Specification (`ops/observability-spec`) | What each service must emit |
| Alerting Policy (`ops/alerting-policy`) | Paging rules, severities |
| Error Budget Policy (`ops/error-budget-policy`) | Burn-rate alerts in 4.3 |
| Stream Processing Engine System Design (`system-design/design-stream-processing-engine`) | Streaming alternative for rule evaluation |
| Distributed Tracing and Logging System Design (`system-design/design-distributed-tracing-logging`) | Traces and logs linked by exemplars (2.3) |
| Back-of-Envelope Estimation and Capacity Model (`system-design/back-of-envelope-estimation`) | Method for section 3 |

## Standard and licence note

Based on the OpenTelemetry metrics data model (OpenTelemetry Specification,
latest release v1.62.0 as of October 2026), the public Prometheus
documentation and the published Gorilla (PVLDB 2015) and Monarch (PVLDB
2020) papers cited in 12.3. The papers are paraphrased in 7.2 and 7.7 only;
no text, figures or code were copied, and all capacity numbers here are
illustrative examples computed from the stated assumptions. Check the current
specification versions before implementing. Original text written for this
collection. Template licence: MIT.