Metrics and Monitoring System Design Template (Reference Design)

System Design: [Metrics and monitoring platform name]

Reference design for a multi-tenant time-series metrics and alerting platform: ingestion over OTLP and Prometheus remote write, sharded in-memory ingesters with a write-ahead log, compressed blocks in object storage, downsampling and retention tiers, cardinality control, rule evaluation and alert routing. It follows the OpenTelemetry metrics data model (stable sections as of OpenTelemetry Specification v1.62.0, October 2026) and the compression ideas published in the Gorilla paper (PVLDB 8(12), 2015). It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: Each product team runs its own metrics server. Retention differs from 3 to 90 days, there is no global view across clusters, and two outages last quarter went unnoticed for 20 minutes because the metrics server of the affected cluster failed together with the cluster. One team added a request ID as a label, created 14 million series in an hour and took down the shared server for everyone.]

1.2 Proposed solution

[Example: A central, multi-tenant platform. Agents and SDKs push metrics over OTLP or Prometheus remote write to a stateless distributor, which validates them, enforces per-tenant limits and hashes each series to three ingesters. Ingesters keep the last two hours in compressed memory blocks with a write-ahead log, then flush immutable blocks to object storage. A compactor merges blocks and writes 5-minute and 1-hour rollups. A query frontend splits, caches and routes queries to ingesters (recent data) and store gateways (old data). A rule evaluator runs recording and alerting rules and sends alerts to an alert router that deduplicates, groups and notifies. A separate, small meta-monitoring stack watches the platform itself.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer: store each sample as a row of (series ID, timestamp, value) in a key-value store, 16 bytes per point before indexes. Show that at 6.7 million samples per second this is about 9.2 TB a day, then introduce delta-of-delta timestamps and XOR values and the two-hour in-memory block.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: logs and traces (separate pipelines; exemplars link to trace IDs only)]
• [Example: anomaly detection and forecasting on metrics]
• [Example: per-user product analytics (event data, not metrics)]
• [Example: billing-grade usage metering (needs exactly-once accounting)]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Ingestion rate

With replication each sample is written to three ingesters, so the ingester tier absorbs $3r = 2 \times 10^{7}$ sample appends per second. At an assumed 10 bytes per sample on the wire (protobuf plus snappy) the distributors receive about 66.7 MB/s. Consequence: the write path is CPU-bound on decompression and hashing, not network-bound, so distributors scale on CPU.

3.3 Storage and retention

Compression cuts the raw stream by $16 / 1.37 \approx 11.7$ times, which is what makes an in-memory recent window affordable. Rollups, not raw data, are 90% of the stored bytes, so the retention of the 5-minute tier is the main cost lever (section 9.4). Consequence: raw data goes to object storage and the 13-month tier is reviewed with dashboard owners before anyone asks for longer raw retention.

3.4 Memory for the recent window

The factor 1.25 is headroom for uneven series distribution and for series churn (pods restarting create new series while old ones are still in the head). Each ingester appends about $2 \times 10^{7} / 30 \approx 667\,000$ samples per second. Consequence: memory per series, not samples per second, decides the ingester count, so the series limit in 7.3 is a capacity control, not only a safety net.

3.5 Queries and rules

• Rule evaluations: $Q / 30 = 50\,000 / 30 \approx 1\,667$ per second, mostly over the last few minutes, so they are served from ingesters.
• A 7-day panel at 15 s resolution reads 40 320 points per series; at 5-minute resolution it reads 2 016, 20 times fewer. A 1 000-series panel at raw resolution reads 40.3 million points. Consequence: the query frontend routes ranges longer than 2 days to the 5-minute tier automatically.

3.6 Summary

4. API design

4.1 Interfaces

Every request carries a tenant identity derived from the authenticated credential, never from a free-text header set by the client.

4.2 Write responses and limits

A write that would create series beyond the tenant's limit is partly accepted: existing series are stored, new series are dropped, and the response says so. Because the stored part must not be sent again, this is not a retryable error. On OTLP the response is 200 OK with the partial_success field filled in (rejected_data_points and error_message), which the OTLP specification says clients must not retry; on Prometheus remote write it is a 400, which senders do not retry. Keep 429 with Retry-After for the samples-per-second rate limit (7.3), where the whole request was refused and a later retry is correct. Example OTLP response body, shown in the OTLP JSON encoding:

{"partialSuccess": {"rejectedDataPoints": "1240", "errorMessage": "Tenant team-checkout has 5000000 active series (limit 5000000). 1240 data points for new series were dropped; 18760 data points for existing series were stored. Retrying will not help: reduce label values or ask for a higher limit."}}

4.3 Alert rule format

Rules are versioned documents; a new rule group starts in shadow mode, where it evaluates and records what it would have fired without notifying.

{"namespace": "checkout", "group": "slo-burn", "interval_s": 30, "mode": "enforce", "rules": [{"alert": "CheckoutErrorBudgetFastBurn", "expr": "sum(rate(http_server_request_duration_seconds_count{service=\"checkout\",status_class=\"5xx\"}[5m])) / sum(rate(http_server_request_duration_seconds_count{service=\"checkout\"}[5m])) > 14.4 * 0.001", "for_s": 120, "labels": {"severity": "page"}, "annotations": {"runbook": "https://runbooks.example.com/checkout/error-budget"}}]}

The threshold 14.4 * 0.001 is a fast-burn alert for a 99.9% SLO: an error ratio 14.4 times the budget spends 2% of a 30-day budget in one hour.

5. Data model

5.1 Entities

A series is identified by its metric name plus the full set of label (attribute) pairs, including resource attributes such as service and cluster. Every distinct combination is a new series, which is why label values must come from small, bounded sets.

5.2 Storage choices

5.3 Access patterns

6. High-level design

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Write path

1. The distributor authenticates the tenant and checks request size, label count (Example: at most 30 labels per series), label name syntax and value length (Example: 1 024 bytes).
2. It converts delta-temporality sums to cumulative per series (7.6) and rejects samples older than the head window or too far in the future.
3. It hashes tenant plus sorted label set onto a consistent-hash ring and sends the sample to the three ingesters that own the token. The write succeeds when two of three acknowledge.
4. Each ingester appends to its write-ahead log, then to the in-memory head. Every two hours it cuts an immutable block, uploads it and truncates the WAL once upload is confirmed.

Because three replicas hold the same series, the compactor deduplicates overlapping blocks from the replicas, keeping one sample per timestamp.

7.2 Compression and the in-memory window

The encoding follows the scheme published for Gorilla (paraphrased):

• Timestamps are stored as the difference of consecutive differences ("delta of delta"). For a series collected every 15 s the deltas are 15, 15, 15, so the delta of delta is 0 and costs one bit. Small irregularities fall into short bit ranges with a short prefix; only large jumps use a long form.
• Values are XORed with the previous value. Identical values cost one bit; similar floats share sign, exponent and leading mantissa bits, so only the changed middle bits are stored with their position.

Example: timestamps 1 000, 1 015, 1 030, 1 045, 1 061 have deltas 15, 15, 15, 16 and deltas of delta 0, 0, 1. After the first two timestamps the next three cost 1 + 1 + 9 bits (a 2-bit prefix and a 7-bit value for the 1), or 11 bits instead of 192 bits uncompressed.

Chunks are aligned to two-hour windows. Decoding is sequential, which suits range scans and rules over the last minutes, and costs nothing for random access because nobody reads one sample in the middle of a chunk. The paper reports an average of 1.37 bytes per sample against 16 uncompressed; measure your own ratio, since counters with many distinct values compress worse than mostly constant gauges.

7.3 Cardinality control

Cardinality is the number of distinct series. It explodes multiplicatively: Example: http_server_request_duration_seconds with 5 methods, 10 status codes, 200 routes and 2 000 pods yields $5 \times 10 \times 200 \times 2\,000 = 2 \times 10^{7}$ series, 20% of the whole platform budget in one metric. Adding a user ID label makes it unbounded. Prometheus naming guidance warns against labels with many distinct values such as user IDs and email addresses; put those in logs or traces and link them with exemplars.

Limits are counted in the ingesters, where series are created, and the distributor reads them from a shared view refreshed every few seconds, so a tenant can overshoot by the series created in one refresh interval. Distributor-side relabelling rules let a tenant drop or aggregate a label (for example remove pod and keep deployment) before series are created.

7.4 Downsampling and retention

Rollups must keep aggregates that can be combined again. An average of averages is wrong when sample counts differ, so the store keeps sum and count and computes the average at query time. Counters are rolled up as the increase within the interval after handling resets, never as the raw counter value. Histograms are rolled up by adding bucket counts, which is only valid when bucket boundaries match; exponential histograms with different scales are first converted to the coarser scale. Percentiles cannot be rolled up from stored percentiles; keep the histogram.

7.5 Alert evaluation and watching the watcher

• Rule groups are sharded across evaluators by hash of namespace and group. Each group runs every interval_s; a missed evaluation is recorded as a metric and alerts on itself.
• An alert becomes pending when the expression returns a result and firing after it stays true for for_s. Alerts on absence (absent() or a canary series) catch a silent exporter.
• The alert router deduplicates by label fingerprint, groups related alerts into one notification, applies inhibition (a cluster-down alert suppresses its per-pod alerts) and silences with an owner and expiry.
• Meta-monitoring runs in a different region or provider with its own small storage. It probes the write path with a canary series, the query path and the alert path with a canary alert that must arrive at a test receiver every 5 minutes (a "dead man's switch"). If the canary stops arriving, the meta stack pages directly.

7.6 Temporality and the OpenTelemetry data model

The OpenTelemetry metrics data model defines Sum, Gauge, Histogram and ExponentialHistogram point kinds, plus Summary kept for compatibility and not recommended for new applications. Sums carry an aggregation temporality: cumulative points repeat the same start time, delta points advance it. This platform stores cumulative sums because rate functions and rollups expect them. The distributor keeps the last cumulative total per delta series and adds each delta to it; if the start time does not follow the previous end time it records a reset. That state lives in the distributor that receives the stream, so delta sources must stick to one distributor (by consistent hashing on the series at the load balancer) or send cumulative data, which is the safer default.

7.7 How real systems do it

How real systems do it (paraphrased, not copied). The Gorilla paper (Pelkonen and co-authors, PVLDB 8(12), 2015) describes the in-memory time-series store that sat in front of an HBase-backed monitoring system at a large social network. It argues that monitoring users care most about recent data and aggregates, so the store keeps the most recent 26 hours in memory, accepts losing a small amount of data on the write path in exchange for availability, and compresses timestamps with delta of delta and values with XOR. The authors report about 96% of timestamps compressed to one bit, roughly 51% of values to one bit, an average of 1.37 bytes per point against 16, and query latency 73 times lower than the previous system. Every write is streamed to two independent instances in separate regions, and queries move to the other region when one is unhealthy. Google's Monarch paper (Adams and co-authors, PVLDB 13(12), 2020) describes a planet-scale in-memory time-series system that keeps data in regional zones close to where it is produced, with global query and configuration planes on top, so a regional problem stays regional. This design keeps both lessons: the recent window in memory (7.2) and a monitoring path for the monitor that does not share its failure domain (7.5). Read the originals in 12.3.

8. Scalability and performance

Query latency budget for a 24-hour panel over 1 000 series (target p95 2 s):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and privacy

9.3 Observability of the platform

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Teams dual-write from their agents to the old server and the platform for 30 days. Rule groups run in shadow mode on the platform and their would-fire results are compared with the old alerts. Cut-over is per team; rollback points dashboards and alert routing back to the old server, which kept receiving data during dual-write.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on the OpenTelemetry metrics data model (OpenTelemetry Specification, latest release v1.62.0 as of October 2026), the public Prometheus documentation and the published Gorilla (PVLDB 2015) and Monarch (PVLDB 2020) papers cited in 12.3. The papers are paraphrased in 7.2 and 7.7 only; no text, figures or code were copied, and all capacity numbers here are illustrative examples computed from the stated assumptions. Check the current specification versions before implementing. Original text written for this collection. Template licence: MIT.