Markdown Editor

डेटा इंजीनियरिंग किट

डेटा आवश्यकताएँ और वर्गीकरण, लेकहाउस और स्ट्रीमिंग डिज़ाइन, डेटा प्रोडक्ट, ODCS डेटा कॉन्ट्रैक्ट, पाइपलाइन स्पेसिफ़िकेशन, डेटा गुणवत्ता, प्रतिधारण और डेटा इंसिडेंट पोस्टमॉर्टम।

चरण 1

आवश्यकताएँ

  1. Data Requirements Specification Template

    मानक

    Data entities, dictionary, ISO/IEC 25012 quality requirements, retention, privacy classification, lineage, interfaces and migration

    ISO/IEC 25012:2008 characteristic names
  2. Data Classification Policy Template (FIPS 199, ISO/IEC 27002 5.12)

    पूर्ण

    Classification levels and criteria, FIPS 199 impact method, personal and special-category flags, labelling, handling matrix incl. AI use, discovery, declassification

    FIPS 199; ISO/IEC 27002:2022 5.12, 5.13; GDPR Art. 9, 10
चरण 2

डिज़ाइन

  1. Data Platform Architecture Template (Lakehouse, Medallion, Lineage)

    पूर्ण

    Lakehouse or warehouse architecture: consumers, ingestion, medallion layers, table formats, catalogue, OpenLineage 2-0-2 lineage, quality, security, cost

    DAMA-DMBOK2 Revised (2024) knowledge areas; Apache Iceberg table spec v3; Delta Lake protocol; OpenLineage spec 2-0-2
  2. Streaming Solution Design Template (Kafka, Flink, AsyncAPI)

    पूर्ण

    Latency SLOs, delivery guarantees, CDC sources, AsyncAPI 3.1.0 contracts, topics, watermarks, windows, state, exactly-once, capacity, lag

    AsyncAPI 3.1.0; CloudEvents 1.0.2; Apache Kafka 4.x and Apache Flink 2.x concepts
  3. Data Product Specification Template (Bitol ODPS v1.1.0)

    पूर्ण

    One data product: purpose, consumers, owner, input/output/management ports, ODCS contract links, SLOs, metadata, access, pricing, AI context, ODPS YAML

    Bitol ODPS v1.1.0; ODCS v3.2.0
चरण 3

स्पेसिफ़िकेशन

  1. Data Contract Template (ODCS v3.2.0)

    पूर्ण

    Producer-consumer data contract: schema, semantic roles, quality rules, SLA properties, servers and variables, roles, pricing, AI context, ODCS v3.2.0 YAML

    Bitol Open Data Contract Standard (ODCS) v3.2.0
  2. Data Pipeline Specification Template

    पूर्ण

    Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook

  3. Data Dictionary Template

    मानक

    Business and technical definitions per attribute: type, format, allowed values, source, owner, classification, code lists, derived fields

चरण 4

सत्यापन

  1. Data Quality Standard Template (ISO/IEC 25012, ODCS)

    पूर्ण

    Organisation DQ standard: ISO/IEC 25012 vocabulary, tiers, rule catalogue, severity, enforcement points, observability, scorecard, issues, waivers

    ISO/IEC 25012:2008 (names); ODCS v3.2.0; OpenLineage 2-0-2
चरण 5

संचालन

  1. Data Retention and Deletion Standard Template (GDPR Art. 17, SP 800-88)

    पूर्ण

    Retention rule method, deletion classes, standard periods, deletion per store (OLTP, Iceberg, Delta, search, logs, ML, backups), erasure propagation, evidence, sanitisation

    GDPR Art. 5(1)(e), 17; NIST SP 800-88 Rev. 2; ISO/IEC 27555:2021
  2. Data Incident Postmortem Template (Lineage, GDPR Breach Check)

    पूर्ण

    Data incident postmortem: SEV1-SEV4, impact, timeline, lineage blast radius, decisions affected, backfill, SLO breach, GDPR check, DQ rule gaps

    OpenLineage 2-0-2; ODCS v3.2.0; GDPR Arts. 33-34