데이터 엔지니어링 키트
데이터 요구사항과 분류, 레이크하우스 및 스트리밍 설계, 데이터 프로덕트, ODCS 데이터 계약, 파이프라인 사양, 데이터 품질, 보존, 데이터 장애 포스트모템.
요구사항
Data Requirements Specification Template
표준Data entities, dictionary, ISO/IEC 25012 quality requirements, retention, privacy classification, lineage, interfaces and migration
ISO/IEC 25012:2008 characteristic namesData Classification Policy Template (FIPS 199, ISO/IEC 27002 5.12)
전체Classification levels and criteria, FIPS 199 impact method, personal and special-category flags, labelling, handling matrix incl. AI use, discovery, declassification
FIPS 199; ISO/IEC 27002:2022 5.12, 5.13; GDPR Art. 9, 10
설계
Data Platform Architecture Template (Lakehouse, Medallion, Lineage)
전체Lakehouse or warehouse architecture: consumers, ingestion, medallion layers, table formats, catalogue, OpenLineage 2-0-2 lineage, quality, security, cost
DAMA-DMBOK2 Revised (2024) knowledge areas; Apache Iceberg table spec v3; Delta Lake protocol; OpenLineage spec 2-0-2Streaming Solution Design Template (Kafka, Flink, AsyncAPI)
전체Latency SLOs, delivery guarantees, CDC sources, AsyncAPI 3.1.0 contracts, topics, watermarks, windows, state, exactly-once, capacity, lag
AsyncAPI 3.1.0; CloudEvents 1.0.2; Apache Kafka 4.x and Apache Flink 2.x conceptsData Product Specification Template (Bitol ODPS v1.1.0)
전체One data product: purpose, consumers, owner, input/output/management ports, ODCS contract links, SLOs, metadata, access, pricing, AI context, ODPS YAML
Bitol ODPS v1.1.0; ODCS v3.2.0
명세
Data Contract Template (ODCS v3.2.0)
전체Producer-consumer data contract: schema, semantic roles, quality rules, SLA properties, servers and variables, roles, pricing, AI context, ODCS v3.2.0 YAML
Bitol Open Data Contract Standard (ODCS) v3.2.0Data Pipeline Specification Template
전체Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook
Data Dictionary Template
표준Business and technical definitions per attribute: type, format, allowed values, source, owner, classification, code lists, derived fields
검증
운영
Data Retention and Deletion Standard Template (GDPR Art. 17, SP 800-88)
전체Retention rule method, deletion classes, standard periods, deletion per store (OLTP, Iceberg, Delta, search, logs, ML, backups), erasure propagation, evidence, sanitisation
GDPR Art. 5(1)(e), 17; NIST SP 800-88 Rev. 2; ISO/IEC 27555:2021Data Incident Postmortem Template (Lineage, GDPR Breach Check)
전체Data incident postmortem: SEV1-SEV4, impact, timeline, lineage blast radius, decisions affected, backfill, SLO breach, GDPR check, DQ rule gaps
OpenLineage 2-0-2; ODCS v3.2.0; GDPR Arts. 33-34