Distributed Tracing and Logging System Design Template (Reference Design)

System Design: [Tracing and logging platform name]

Reference design for a distributed tracing and log pipeline: how trace context crosses service and queue boundaries, how spans and logs are collected, which traces are kept (head, tail and hybrid sampling), how logs are correlated with traces, and how long each kind of data is retained. It uses W3C Trace Context (Recommendation, Level 1, 23 November 2021), W3C Baggage (Candidate Recommendation Snapshot, 30 May 2024) and OpenTelemetry (specification 1.62.0 as of October 2026, with the OTLP protocol and the Collector). It follows the structure of the System Design Document template, adds a 45-minute interview path, access-pattern and failure-mode tables and a cited "How real systems do it" box, and is pre-filled with a worked design to adapt. Replace every [bracketed] value and every row marked Example; values marked Example are assumptions, not benchmarks.

Document control

Approvals

Revision history

Contents

1. Overview
2. Requirements
3. Capacity estimation
4. API design
5. Data model
6. High-level design
7. Detailed design
8. Scalability and performance
9. Reliability, security and operations
10. Trade-offs and alternatives
11. Delivery plan and risks
12. Appendices

1. Overview

1.1 Problem statement

[Example: About 2 000 services emit logs in four formats and three tracing libraries that do not share context. During last month's checkout outage it took 70 minutes to connect a customer's failed payment to the service that timed out, because trace IDs stopped at the message queue and logs carried no trace ID at all. The logging bill grew 60% in a year, mostly from debug logs nobody reads.]

1.2 Proposed solution

[Example: Every service uses the OpenTelemetry SDK and propagates W3C traceparent, tracestate and baggage headers over HTTP, gRPC and message headers. SDKs head-sample 25% of traces with a parent-based, trace-ID-ratio sampler and attach the trace and span ID to every log record. The SDK instrumentation also records request, error and duration metrics for every request, sampled or not, and exports them as OTLP metrics. A Collector agent on each node forwards the metrics and sends sampled spans with a load-balancing exporter keyed by trace ID to a tail-sampling gateway tier. The gateway keeps every error and slow trace plus 5% of the rest and writes them to a trace store on object storage with a trace-ID index. Logs flow through the same agents to Kafka, then to a hot search index for 7 days and compressed object storage for 30 days.]

1.3 Decisions requested from reviewers

1.4 45-minute interview path

Use this order when the document serves an interview or a short design review. The written sections below are the long form of each step.

Start the default deep dive with the naive answer, keep 1% of traces chosen at random at the first service, and show what it misses: 99% of failed requests have no trace. Then add tail sampling, show that it needs every span of a trace on one machine, and size the buffer that requires.

2. Requirements

2.1 Core functional requirements

2.2 Non-functional requirements

2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design toward them.

• [Example: the metrics platform itself (time series storage and alerting; see system-design/design-metrics-monitoring)]
• [Example: continuous profiling (the OpenTelemetry profiles signal is still in development as of October 2026)]
• [Example: real-user monitoring in browsers and mobile apps]
• [Example: security event management (SIEM) correlation rules]
• [Item deferred, with the release that will take it]

3. Capacity estimation

3.1 Assumptions

3.2 Unsampled volume

At the average rate that is about 0.41 PB per day. Consequence: storing every span is not an option; sampling is the core of the design.

3.3 Tail-sampling buffer

The tail sampler must hold all spans of a trace until it decides. After 25% head sampling the gateway tier receives $3 \times 10^{6}$ spans/s (2.4 GB/s):

Spread over 24 gateway instances that is 3 GB each, or 6 GB with 2x headroom. Without head sampling the same buffer would be 288 GB. Consequence: hybrid sampling keeps the tail tier at a modest, fixed size.

3.4 Stored traces

The tail sampler keeps all error and slow traces and 5% of the rest:

At the average rate that is 117 750 spans/s, 8.1 TB per day raw, about 1.0 TB per day after 8x columnar compression and about 14.2 TB for 14 days of retention. The trace-ID index holds about 339 million traces per day at 40 B each, 13.6 GB per day or 190 GB for 14 days. Consequence: the trace store fits on object storage with a small index, not a large search cluster.

3.5 Logs

Consequence: logs, not traces, dominate storage; the cheapest saving is dropping debug logs at the source and indexing only 7 days.

3.6 Summary

4. API design

4.1 Interfaces

4.2 Trace context on the wire

W3C Trace Context Level 1 defines traceparent as four dash-separated fields: version (00), a 16-byte trace ID as 32 lowercase hex characters, an 8-byte parent span ID as 16 hex characters, and 8 bits of trace flags where the lowest bit means sampled. tracestate carries vendor-specific data as a list of at most 32 members. Trace Context Level 2 (Candidate Recommendation Draft, 28 March 2024) adds a random-trace-ID flag (bit value 2); check its status before relying on it.

GET /v1/orders/8812 HTTP/1.1
Host: orders.internal.example
traceparent: 00-7d3e1f9a5c2b4e8f9a0b1c2d3e4f5a6b-3c4d5e6f7a8b9c0d-01
tracestate: acme=q2v8n4kx
baggage: tenant.tier=gold,checkout.flow=express

Baggage carries application context (here a tenant tier) to downstream services; the W3C Baggage format sets minimum limits of 64 members and 8 192 bytes. Never put personal data or secrets in baggage: it is sent to every downstream service, including third parties.

4.3 Span as stored

The trace store keeps spans in the OTLP shape, flattened for columnar storage; times are Unix nanoseconds.

{"trace_id": "7d3e1f9a5c2b4e8f9a0b1c2d3e4f5a6b", "span_id": "9f8e7d6c5b4a3921", "parent_span_id": "3c4d5e6f7a8b9c0d", "name": "SELECT orders", "kind": "CLIENT", "start_unix_nano": 1793353447412000000, "end_unix_nano": 1793353447431000000, "status": "OK", "resource": {"service.name": "orders-api", "deployment.environment.name": "prod"}, "attributes": {"db.system.name": "postgresql"}, "links": []}

5. Data model

5.1 Entities

A trace is not stored as a row; it is the set of spans that share a trace ID. Log records point at spans by ID, and request metrics are series recorded for the same operations.

5.2 Storage choices

5.3 Access patterns

6. High-level design

Spans and logs share the node agent; they split into a trace path and a log path after it.

7. Detailed design

The deep dives in this section, each with options, choice and reason:

7.1 Context propagation

• Synchronous calls: SDK instrumentation for HTTP and gRPC injects and extracts traceparent, tracestate and baggage on every call. Edge proxies keep incoming context only from trusted internal callers and start a new trace for public traffic, so outside clients cannot choose trace IDs or force sampling.
• Messages: producers write the same fields into message headers (Kafka record headers, AMQP properties); consumers start a CONSUMER span whose parent is the producer span.
• Fan-in and batches: a job that processes 500 messages in one batch starts its own trace and adds a span link to each message's context, instead of choosing one parent arbitrarily.
• Threads and async code: use the SDK's context API to pass the active context into executors and callbacks; lost context is the main source of orphan spans (NFR-04).

7.2 Sampling

Head sampling uses ParentBased(root=TraceIdRatioBased(0.25)), so a child service never re-decides and traces stay complete. The OpenTelemetry specification marks TraceIdRatioBased stable but deprecated in favour of a ProbabilitySampler that is still in development as of version 1.62.0; it uses 56 bits of trace-ID randomness and records its threshold under the th key of the ot entry in tracestate. Plan to migrate when that sampler is stable in your SDKs.

Tail sampling needs every span of a trace on the same gateway instance. The agent uses the Collector's load-balancing exporter with the trace ID as the routing key, and the gateway runs the tail sampling processor with these policies:

The consequence to state openly: only 25% of failed requests have a stored trace. Every failure still appears exactly in the request metrics (7.3) and in logs with its trace ID, which is usually enough to find a representative stored trace.

7.3 Request metrics independent of sampling

With parent-based head sampling in the SDK, a span that is not sampled is not recorded or exported, so the agent sees only 25% of traces and cannot count errors from spans. The SDK instrumentation therefore records request count, error count and a duration histogram per service, operation and status class for every request (the HTTP, RPC and messaging duration metrics of the OpenTelemetry semantic conventions), whatever the sampling decision, and exports them as OTLP metrics through the agent. If you later move head sampling from the SDK to the agent, a span-to-metrics step in the agent can do the same job, at the cost of exporting every span to the node. With about 2 000 services, 50 operations each, 3 status classes and 15 histogram buckets that is up to 4.5 million series. Cap operation names at the source (route templates, not raw URLs), or cardinality, not spans, becomes the cost problem.

7.4 Log pipeline and correlation

Services log through the OpenTelemetry logs bridge for their logging library, so each record follows the stable OpenTelemetry log data model and carries TraceId, SpanId and TraceFlags when emitted inside a span. The agent batches logs to Kafka; parsers normalise severity to SeverityNumber, redact (7.6) and write to the hot index and object storage in parallel. Debug logs are off in production by default and can be enabled for one service for 30 minutes through a dynamic configuration flag.

7.5 Retention

Retention for logs that can identify a person must follow your data retention standard. For example, the storage-limitation principle in EU GDPR and UK GDPR Article 5(1)(e) requires that identifiable data is kept no longer than needed. This is a design note, not legal advice; check the current official text and a qualified professional.

7.6 Redaction and access

Redaction runs twice: SDK attribute filters stop known sensitive attributes (authorization headers, cookies, card numbers) at the source, and the gateway and log parsers apply pattern rules as a second line. Access to traces and logs is by team and environment; production payload attributes are visible only to on-call roles, and every query is logged.

7.7 How real systems do it

How real systems do it (paraphrased, not copied). Google's technical report "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" (Sigelman and colleagues, dapper-2010-1, April 2010) describes tracing added to a small set of shared libraries so application developers did not have to instrument code by hand. Spans were written to local log files, collected by per-host daemons and stored with one wide-table row per trace and one column per span; median collection delay was under 15 seconds. The authors report that a uniform sampling rate as low as 1 in 1 024 still gave enough data for high-volume services, describe work towards adaptive sampling driven by a target number of traces, and add a second sampling stage in the collectors that hashes the trace ID so whole traces, not single spans, are kept or dropped. That trace-ID-based decision is the same idea behind TraceIdRatioBased and the trace-ID routing in 7.2. Read the original in 12.3.

8. Scalability and performance

Overhead budget per service request (NFR-01, under 1 ms p99):

9. Reliability, security and operations

9.1 Failure modes

9.2 Security and privacy

9.3 Observability of the pipeline

9.4 Cost estimate

10. Trade-offs and alternatives

11. Delivery plan and risks

11.1 Milestones

11.2 Rollout and rollback

[Example: Roll out the SDK service by service behind a configuration flag, starting with the checkout path. The old log shipper runs in parallel for 30 days; rollback is turning the flag off, which stops export without a redeploy.]

11.3 Risks

12. Appendices

12.1 Open questions

12.2 Glossary

12.3 References

Standard and licence note

Based on W3C Trace Context (Recommendation, 23 November 2021), W3C Baggage (Candidate Recommendation Snapshot, 30 May 2024) and the OpenTelemetry specification 1.62.0 as published in October 2026. Baggage and Trace Context Level 2 are not yet Recommendations, and several OpenTelemetry samplers are still in development, so check the current status pages before implementing. The Dapper report is paraphrased in 7.7 only; no text, figures or code were copied, and every number in this template is an illustrative example computed for this design. The retention note is not legal advice. Original text written for this collection. Template licence: MIT.