Markdown Editor
打开编辑器

可观测性与数据系统设计套件

数据密集型基础设施参考设计：指标、链路追踪与日志、点击聚合、流、CDC、存储与向量搜索。

12 个模板，按使用顺序分为 3 步。在编辑器中打开任一模板，填写后可导出为 PDF、DOCX 或 HTML。

第 1 步

学习

  1. Data Store Selection Template (Relational, NoSQL, Search, Vector)

    标准

    Workload profile, store categories (relational, KV, document, wide-column, search, time-series, vector), weighted matrix, derived stores

  2. Replication and Consistency Models Template (Quorums, PACELC)

    标准

    Replication topology, sync vs async, N/W/R quorums, consistency model per operation, conflicts, failover and RPO, PACELC

    PACELC (Abadi 2012); Jepsen consistency map
第 2 步

参考设计

  1. Metrics and Monitoring System Design Template (Reference Design)

    完整

    Metrics and alerting design: OTLP and remote write ingestion, Gorilla-style compression, cardinality limits, rollups, alert evaluation, meta-monitoring

    OpenTelemetry metrics data model (Spec 1.62.0); Gorilla (PVLDB 2015); docs/48 design standard

  2. Distributed Tracing and Logging System Design Template (Reference Design)

    完整

    45-minute path, W3C trace context and baggage, async propagation, head, tail and hybrid sampling, span metrics, log correlation, retention tiers, failure modes

    W3C Trace Context (Rec 2021); W3C Baggage (CR 2024); OpenTelemetry 1.62.0

  3. Ad Click Aggregation System Design Template (Reference Design)

    完整

    45-minute path, click dedupe, event-time windows, exactly-once counting, exact and Count-Min top-K, batch reconciliation for billing, failure modes, 10x row

    Count-Min sketch (Cormode and Muthukrishnan 2005); Apache Kafka delivery semantics

  4. Stream Processing Engine System Design Template

    完整

    Stream processing engine: event time, watermarks, windows and triggers, late data, keyed state, barrier checkpoints, exactly-once sinks, backpressure, rescaling

  5. CDC Pipeline System Design Template

    完整

    CDC pipeline: log-based capture, watermark chunked snapshot, per-key order, schema evolution, outbox, search, cache and warehouse sinks, replay

  6. Object Storage System Design Template (S3-Like, Erasure Coding)

    完整

    S3-like blob store design: consistent metadata, packed volumes, replicas then RS(9,6) erasure coding across AZs, durability maths, repair, deletes

    Haystack (OSDI 2010); f4 (OSDI 2014); docs/48 design standard

  7. Distributed File System Design Template (GFS and HDFS Style)

    完整

    GFS/HDFS-style file system: metadata and chunk servers, chunk sizing, leases and write pipeline, record append consistency, re-replication, erasure coding

    GFS (Ghemawat et al., SOSP 2003); HDFS (Shvachko et al., MSST 2010)

  8. Vector Database System Design Template

    完整

    Vector search service: HNSW vs IVF-PQ sizing, re-ranking, recall vs latency tuning, filtered search by selectivity, inserts and deletes, sharding, failure modes

    HNSW (Malkov and Yashunin, TPAMI 2020); PQ (Jégou et al., TPAMI 2011)

  9. Feature Store System Design Template (Online, Offline, Point-in-Time)

    完整

    ML feature store: registry, batch and streaming pipelines, offline and online stores, point-in-time joins with TTL, training-serving skew, monitoring

第 3 步

运维

  1. Observability Specification Template (OpenTelemetry)

    完整

    Telemetry goals, OTel semantic conventions 1.44.0 baseline with stability per area, logs, metrics, traces, optional Profiles, dashboards, SLOs, alerts, privacy, retention, cost

    OpenTelemetry semantic conventions 1.44.0