Markdown Editor
Ouvrir l'éditeur

Kit conception d'observabilité et de systèmes de données

Conceptions de référence pour l'infrastructure de données : métriques, traces et logs, agrégation de clics, flux, CDC, stockage et recherche vectorielle.

12 modèles en 3 étapes, dans l'ordre d'utilisation. Ouvrez un modèle dans l'éditeur, complétez-le et exportez-le en PDF, DOCX ou HTML.

Étape 1

Apprentissage

  1. Data Store Selection Template (Relational, NoSQL, Search, Vector)

    Standard

    Workload profile, store categories (relational, KV, document, wide-column, search, time-series, vector), weighted matrix, derived stores

  2. Replication and Consistency Models Template (Quorums, PACELC)

    Standard

    Replication topology, sync vs async, N/W/R quorums, consistency model per operation, conflicts, failover and RPO, PACELC

    PACELC (Abadi 2012); Jepsen consistency map
Étape 2

Conceptions de référence

  1. Metrics and Monitoring System Design Template (Reference Design)

    Complet

    Metrics and alerting design: OTLP and remote write ingestion, Gorilla-style compression, cardinality limits, rollups, alert evaluation, meta-monitoring

    OpenTelemetry metrics data model (Spec 1.62.0); Gorilla (PVLDB 2015); docs/48 design standard

  2. Distributed Tracing and Logging System Design Template (Reference Design)

    Complet

    45-minute path, W3C trace context and baggage, async propagation, head, tail and hybrid sampling, span metrics, log correlation, retention tiers, failure modes

    W3C Trace Context (Rec 2021); W3C Baggage (CR 2024); OpenTelemetry 1.62.0

  3. Ad Click Aggregation System Design Template (Reference Design)

    Complet

    45-minute path, click dedupe, event-time windows, exactly-once counting, exact and Count-Min top-K, batch reconciliation for billing, failure modes, 10x row

    Count-Min sketch (Cormode and Muthukrishnan 2005); Apache Kafka delivery semantics

  4. Stream Processing Engine System Design Template

    Complet

    Stream processing engine: event time, watermarks, windows and triggers, late data, keyed state, barrier checkpoints, exactly-once sinks, backpressure, rescaling

  5. CDC Pipeline System Design Template

    Complet

    CDC pipeline: log-based capture, watermark chunked snapshot, per-key order, schema evolution, outbox, search, cache and warehouse sinks, replay

  6. Object Storage System Design Template (S3-Like, Erasure Coding)

    Complet

    S3-like blob store design: consistent metadata, packed volumes, replicas then RS(9,6) erasure coding across AZs, durability maths, repair, deletes

    Haystack (OSDI 2010); f4 (OSDI 2014); docs/48 design standard

  7. Distributed File System Design Template (GFS and HDFS Style)

    Complet

    GFS/HDFS-style file system: metadata and chunk servers, chunk sizing, leases and write pipeline, record append consistency, re-replication, erasure coding

    GFS (Ghemawat et al., SOSP 2003); HDFS (Shvachko et al., MSST 2010)

  8. Vector Database System Design Template

    Complet

    Vector search service: HNSW vs IVF-PQ sizing, re-ranking, recall vs latency tuning, filtered search by selectivity, inserts and deletes, sharding, failure modes

    HNSW (Malkov and Yashunin, TPAMI 2020); PQ (Jégou et al., TPAMI 2011)

  9. Feature Store System Design Template (Online, Offline, Point-in-Time)

    Complet

    ML feature store: registry, batch and streaming pipelines, offline and online stores, point-in-time joins with TTL, training-serving skew, monitoring

Étape 3

Exploitation

  1. Observability Specification Template (OpenTelemetry)

    Complet

    Telemetry goals, OTel semantic conventions 1.44.0 baseline with stability per area, logs, metrics, traces, optional Profiles, dashboards, SLOs, alerts, privacy, retention, cost

    OpenTelemetry semantic conventions 1.44.0