Data Platform Architecture: [Platform name]

Solution architecture for an analytical data platform (lakehouse, warehouse or a combination): consumers and service levels, ingestion patterns, medallion storage layers, open table formats and compute engines, the semantic and serving layer, metadata, catalogue and lineage, data quality, security and masking, residency and retention, capacity and cost, operations and the delivery roadmap. The structure is aligned with the knowledge areas of DAMA-DMBOK 2nd Edition Revised (2024) and uses public specifications for table formats (Apache Iceberg table spec, Delta Lake protocol, Apache Hudi) and lineage (OpenLineage). Replace every [bracketed] value and every row marked Example. Worked examples follow a multi-country retailer building one lakehouse for finance, merchandising and customer analytics.

Document control

Approvals

Revision history

Contents

1. Context, scope and drivers
2. Consumers, use cases and service levels
3. Principles and key decisions
4. Logical architecture
5. Sources and ingestion patterns
6. Storage layers and modelling
7. Table formats and compute
8. Semantic and serving layer
9. Metadata, catalogue and lineage
10. Data quality and observability
11. Security and access control
12. Privacy, residency and retention
13. Capacity, cost and FinOps
14. Operations and reliability
15. Roadmap, risks and traceability
16. Glossary and references

1. Context, scope and drivers

1.1 Business drivers

1.2 Scope

1.3 Current state summary

Summarise the systems this platform replaces, their volumes and pain points, and link the Current State Assessment (architecture/current-state-assessment). State which legacy assets keep running during migration and for how long: coexistence drives cost.

2. Consumers, use cases and service levels

Express freshness as an objective that can be measured: the time between a business event at the source and its availability in the serving layer. Record the objectives formally in the SLO document (dev/slo-doc) and define the metrics that consumers see in Metric Definitions (ai-data/metric-definition).

3. Principles and key decisions

3.1 Principles

3.2 Key decisions

4. Logical architecture

Describe each zone: what it holds, who writes to it and who reads it. Keep the governance plane on the diagram; delivery teams skip it first and auditors ask about it first.

5. Sources and ingestion patterns

5.1 Source inventory

The Example total is 60 + 120 + 40 + 10 + 20 = 250 GB of compressed data landing in bronze per day. Section 13 uses this figure.

5.2 Pattern catalogue

5.3 Ingestion rules

1. Land data unchanged in bronze with load metadata: source system, extraction time, batch or offset identifier and schema version.
2. Make every load idempotent so a rerun of the same batch produces the same table state; see the idempotency section of the Data Pipeline Specification (ai-data/data-pipeline-spec).
3. Quarantine records that fail parsing in a separate table with the error reason; never drop them silently.
4. Capture schema changes and alert the owning domain before silver models break.
5. Encrypt in transit from the source to the landing zone; use private connectivity for sources inside the corporate network.

6. Storage layers and modelling

6.1 Layer contract

Bronze, silver and gold is the common medallion naming; some teams use raw, curated and consumption. Keep the names stable once teams rely on them.

6.2 Modelling standards

• Model gold marts dimensionally where the consumer is BI: facts at a declared grain, conformed dimensions, surrogate keys. Detail the model in Analytics and BI Solution Design (architecture/analytics-bi-solution-design).
• Keep entity definitions in the Data Dictionary (architecture/data-dictionary); keep golden records for customer, product and supplier in the master data hub, and consume them in silver rather than rebuilding matching logic in pipelines.
• Name tables <layer>_<domain>_<entity> or use one schema per layer and domain; choose one convention and enforce it in code review.
• Partition large tables on the column most queries filter on (usually a business date) and keep file sizes in the range the engines handle best (record the target size in section 7.3).

7. Table formats and compute

7.1 Table format comparison

7.2 Compute engines

7.3 Table maintenance

Time-travel retention must match the recovery objective in section 14 and the erasure obligations in section 12: a long retention of old snapshots keeps deleted personal data readable.

8. Semantic and serving layer

Keep metric definitions in one place: a semantic layer or metrics store that every tool queries. Open specifications for exchanging semantic models are emerging (for example Apache Ossie, incubating at the Apache Software Foundation, formerly Open Semantic Interchange); check their maturity before relying on them. Detail the semantic model in the Analytics and BI Solution Design.

9. Metadata, catalogue and lineage

9.1 Metadata scope

9.2 Lineage design

Emit lineage as OpenLineage run events from the orchestrator and from each engine integration. Each event names a job, a run identifier and the input and output datasets, with facets that carry schema, data quality results and SQL. Name datasets consistently (namespace plus physical name) so events from different tools join into one graph.

10. Data quality and observability

10.1 Quality rules by layer

10.2 Observability

Collect freshness, volume, schema-change and distribution signals for every gold table, and route alerts to the owning domain. Write the signals and dashboards into the Observability Specification (ops/observability-spec). Publish a quality score per data product in the catalogue so consumers can judge fitness for use.

11. Security and access control

11.1 Access model

11.2 Access matrix

Review access quarterly with the Access Review template (ops/access-review). Threats to the platform (exfiltration through exports, over-privileged service identities, poisoned source data) belong in the Threat Model (security/threat-model).

12. Privacy, residency and retention

13. Capacity, cost and FinOps

13.1 Storage estimate (Example)

At an Example object storage price of USD 0.023 per GB-month, 145 TB (145,000 GB) costs 145,000 x 0.023 = USD 3,335 per month before snapshots, non-production copies and backup. Add [20 %] for time-travel snapshots and [10 %] for non-production samples: 3,335 x 1.3 = USD 4,335.50 per month.

Consequence: bronze is 69 % of storage (100 of 145 TB). Moving bronze objects older than 90 days to an infrequent-access tier is the largest storage lever and must be tested against replay needs.

13.2 Compute estimate (Example)

USD 130 per day over 30 days is USD 3,900 per month. Compute and storage together are about USD 8,235.50 per month in this Example.

Consequence: bronze parsing and CDC merge is the largest compute item (24 of 52 node-hours, 46 %). Merge-on-read tables or fewer merge cycles per day reduce it; test the read-side cost before switching.

13.3 FinOps controls

• Tag every compute resource and storage location with domain, layer and environment; allocate cost monthly in the FinOps Cost Review (ops/finops-cost-review).
• Set budgets and auto-stop rules per workspace; alert at 80 % of budget.
• Report unit cost per data product (cost per refresh, cost per query).
• Compare the run cost with the legacy estate in the TCO Analysis (architecture/tco-analysis).

14. Operations and reliability

15. Roadmap, risks and traceability

15.1 Roadmap

15.2 Risks

15.3 Traceability to data management knowledge areas

16. Glossary and references

16.1 Glossary

16.2 References

Standard and licence note

Structure aligned with the knowledge areas of DAMA-DMBOK 2nd Edition Revised (DAMA International, 2024); only knowledge area names are used and no text, figures or definitions from the book are reproduced. Table format statements are based on the public Apache Iceberg table specification, the Delta Lake protocol and Apache Hudi documentation (Apache License 2.0), and lineage statements on the OpenLineage specification. This template is original text written for this collection. Check the current official version of each specification before use. Apache Iceberg and Apache Hudi are trademarks of the Apache Software Foundation, Delta Lake of LF Projects and DAMA-DMBOK of DAMA International; this template is not affiliated with or endorsed by them. Template licence: MIT.