डेटा इंजीनियरिंग किट
डेटा आवश्यकताएँ और वर्गीकरण, लेकहाउस और स्ट्रीमिंग डिज़ाइन, डेटा प्रोडक्ट, ODCS डेटा कॉन्ट्रैक्ट, पाइपलाइन स्पेसिफ़िकेशन, डेटा गुणवत्ता, प्रतिधारण और डेटा इंसिडेंट पोस्टमॉर्टम।
आवश्यकताएँ
Data Requirements Specification Template
मानकData entities, dictionary, ISO/IEC 25012 quality requirements, retention, privacy classification, lineage, interfaces and migration
ISO/IEC 25012:2008 characteristic namesData Classification Policy Template (FIPS 199, ISO/IEC 27002 5.12)
पूर्णClassification levels and criteria, FIPS 199 impact method, personal and special-category flags, labelling, handling matrix incl. AI use, discovery, declassification
FIPS 199; ISO/IEC 27002:2022 5.12, 5.13; GDPR Art. 9, 10
डिज़ाइन
Data Platform Architecture Template (Lakehouse, Medallion, Lineage)
पूर्णLakehouse or warehouse architecture: consumers, ingestion, medallion layers, table formats, catalogue, OpenLineage 2-0-2 lineage, quality, security, cost
DAMA-DMBOK2 Revised (2024) knowledge areas; Apache Iceberg table spec v3; Delta Lake protocol; OpenLineage spec 2-0-2Streaming Solution Design Template (Kafka, Flink, AsyncAPI)
पूर्णLatency SLOs, delivery guarantees, CDC sources, AsyncAPI 3.1.0 contracts, topics, watermarks, windows, state, exactly-once, capacity, lag
AsyncAPI 3.1.0; CloudEvents 1.0.2; Apache Kafka 4.x and Apache Flink 2.x conceptsData Product Specification Template (Bitol ODPS v1.1.0)
पूर्णOne data product: purpose, consumers, owner, input/output/management ports, ODCS contract links, SLOs, metadata, access, pricing, AI context, ODPS YAML
Bitol ODPS v1.1.0; ODCS v3.2.0
स्पेसिफ़िकेशन
Data Contract Template (ODCS v3.2.0)
पूर्णProducer-consumer data contract: schema, semantic roles, quality rules, SLA properties, servers and variables, roles, pricing, AI context, ODCS v3.2.0 YAML
Bitol Open Data Contract Standard (ODCS) v3.2.0Data Pipeline Specification Template
पूर्णBatch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook
Data Dictionary Template
मानकBusiness and technical definitions per attribute: type, format, allowed values, source, owner, classification, code lists, derived fields
सत्यापन
संचालन
Data Retention and Deletion Standard Template (GDPR Art. 17, SP 800-88)
पूर्णRetention rule method, deletion classes, standard periods, deletion per store (OLTP, Iceberg, Delta, search, logs, ML, backups), erasure propagation, evidence, sanitisation
GDPR Art. 5(1)(e), 17; NIST SP 800-88 Rev. 2; ISO/IEC 27555:2021Data Incident Postmortem Template (Lineage, GDPR Breach Check)
पूर्णData incident postmortem: SEV1-SEV4, impact, timeline, lineage blast radius, decisions affected, backfill, SLO breach, GDPR check, DQ rule gaps
OpenLineage 2-0-2; ODCS v3.2.0; GDPR Arts. 33-34