Data Lineage Specification Template (OpenLineage)

Lineage use cases, capture granularity, OpenLineage 2-0-2 events and required facets, dataset and job naming, producers, store, consumers, lineage checks

Datentechnik & Daten-GovernanceVollständigOpenLineage specification 2-0-2lineage specificationAktualisiert 29.10.2026

Im Editor öffnen.md herunterladen

Anleitung: Use it when you introduce or standardise lineage capture across orchestrators and engines. Fill the use cases and the naming conventions first, because they decide what to capture and whether graphs from different tools join. The main pitfall is a lineage graph nobody trusts because producers silently stop emitting after upgrades.

Vorschau

Data Lineage Specification: [Organisation or platform name]

Specification for capturing, storing and using data lineage across a data platform: the questions lineage must answer, capture scope and granularity, the OpenLineage event model (specification 2-0-2) with required facets, dataset and job naming rules, producers and integration points, the lineage store and its consumers (impact analysis, incident triage, catalogue, privacy and audit), quality checks for lineage itself, and rollout. Replace every [bracketed] value and every row marked Example. Worked examples use Corvid Commerce, a fictional SaaS retailer with Airflow orchestration, Spark and dbt on an Apache Iceberg lakehouse and a Snowflake warehouse for finance.

Document control

Field	Value
Document ID	[SPEC-LIN-001]
Version	[0.1]
Status	[Draft / In review / Approved / Superseded]
Owner	[@platform-lead]
Approvers/reviewers	[@head-of-data-governance], [@analytics-engineering-lead], [@security-architect], [@data-protection-officer]
Scope	[Platforms, engines and environments covered]
Lineage standard	OpenLineage specification 2-0-2 (`https://openlineage.io/spec/2-0-2/OpenLineage.json`)
Lineage backend	[Tool name and version]
Effective from	YYYY-MM-DD
Next scheduled review	YYYY-MM-DD
Last updated	YYYY-MM-DD
Classification	[Internal]

Approvals

Role	Name	Decision	Date
Platform lead	[@name]	[Approve / Approve with conditions / Reject]	YYYY-MM-DD
Head of data governance	[@name]	[Decision]	YYYY-MM-DD
Security architect	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
1.0	YYYY-MM-DD	[@name]	Example: Approved; Airflow, Spark and dbt integrations mandatory
1.1	YYYY-MM-DD	[@name]	Example: Column-level lineage required for Gold datasets

1. Purpose and scope
2. Lineage use cases and questions
3. Capture scope and granularity
4. Event model
5. Required facets
6. Naming conventions
7. Producers and integration points
8. Lineage store and architecture
9. Consumers and access
10. Lineage quality and monitoring
11. Security and privacy
12. Rollout plan
13. Glossary and references
Standard and licence note

1. Purpose and scope

[Two to four sentences on the problem and the outcome.]

Example: When the Orders data product published wrong totals in [month], it took two days to find which dashboards and finance extracts had read the bad partition. Corvid Commerce will capture lineage at run time from the orchestrator and every engine that writes shared datasets, so that the owner of a broken dataset can list affected consumers within minutes, and governance can show where personal data flows.

Item	In scope	Out of scope (for now)
Engines	Example: Airflow, Spark, dbt, Kafka Connect sinks	Example: Ad hoc notebooks, spreadsheet exports
Stores	Example: Iceberg lakehouse, Snowflake finance warehouse, Kafka topics	Example: Operational databases (captured only as sources)
Environments	Example: Production	Example: Development and staging (separate namespace, not retained)
Consumers	Example: BI dashboards registered in the catalogue	Example: Personal BI workbooks

2. Lineage use cases and questions

ID	Use case	Question	Needs	Primary user
LUC-001	Impact analysis before change	Which datasets, jobs and dashboards read this column or table?	Column-level downstream	Producing team
LUC-002	Incident triage	A dataset is wrong: which upstream run produced it and what else did that run write?	Run-level, with run status and time	On-call data engineer
LUC-003	Root cause	Which source and transformation produced this value?	Column-level upstream with transformation type	Analytics engineer
LUC-004	Personal data mapping	Where does personal data from the customer system end up?	Dataset-level downstream plus classification tags	Data protection officer
LUC-005	Deletion and retention	Which copies must be deleted when a record is erased at source?	Dataset-level downstream, including extracts	Privacy engineering
LUC-006	Regulatory and audit evidence	How is this reported figure produced, from source to report?	Column-level, versioned, with code location	Finance controller, auditor
LUC-007	Cost and clean-up	Which datasets have no readers in 90 days?	Dataset-level downstream with last read time	Platform team
[LUC-nnn]	[Use case]	[Question]	[Granularity]	[User]

3. Capture scope and granularity

Level	Definition	Required for	Method
Dataset-level	Job X read datasets A and B and wrote dataset C	All shared datasets	Run events from engines
Run-level	Which run (with start, end, status) produced which version of C	All production jobs	Run events with run ID and parent run
Column-level	Column C.total derives from A.amount and B.rate, with transformation type	Silver and Gold datasets, key fields only (Example)	Column lineage facet from Spark and dbt integrations
Design-time	Declared inputs and outputs of a job, without a run	Jobs not yet scheduled, external systems	Job events or catalogue entries
Manual	Edge entered by a person with a reason	Systems that cannot emit, such as a vendor SFTP drop	Catalogue form, reviewed every 6 months

Rules:

Run-time lineage wins over manual lineage for the same edge. Manual edges show a "manual" marker and the date they were last confirmed.
Every shared dataset in the catalogue has at least one producing job in the lineage graph, or an approved exception.
Development and test runs are emitted to a separate namespace and are not shown to consumers.

4. Event model

OpenLineage defines three kinds of event:

Event	Purpose	When this platform emits it
Run event	State of one run of a job, with its input and output datasets	Every production job run: start and end, plus intermediate states where the integration sends them
Job event	Design-time description of a job, not tied to a run	When a job is deployed or its declared inputs and outputs change
Dataset event	Design-time metadata about a dataset, outside any run	When a dataset is created, its schema changes outside a job, or it is dropped

Core entities and identity:

Entity	Identified by	Notes
Job	Namespace and name	Name unique within the namespace (section 6)
Run	`runId`, a UUID generated by the client	UUIDv7 recommended by the specification; kept for the whole run
Dataset	Namespace and name derived from the physical location	Same dataset seen by two tools must get the same pair

Run event types in specification 2-0-2 are START, RUNNING, COMPLETE, ABORT, FAIL and OTHER. A run normally sends START and then exactly one of COMPLETE, FAIL or ABORT.

4.1 Example run event

Example: the dbt model that builds the Orders mart completes. Values are illustrative.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-10-30T06:12:44.120Z",
  "producer": "https://github.com/example-org/corvid-lineage-producer/v1.4.0",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
  "run": {
    "runId": "0192d5a8-7c3e-7b21-9f40-3a1c2e5d8b77",
    "facets": {
      "parent": {
        "_producer": "https://github.com/example-org/corvid-lineage-producer/v1.4.0",
        "_schemaURL": "https://openlineage.io/spec/facets/1-2-0/ParentRunFacet.json#/$defs/ParentRunFacet",
        "run": {"runId": "0192d5a1-0b2f-7e55-8c10-4f7a9d3e2c11"},
        "job": {"namespace": "airflow-prod", "name": "orders_daily.dbt_build"}
      }
    }
  },
  "job": {
    "namespace": "dbt-prod",
    "name": "sales.marts.fct_orders"
  },
  "inputs": [
    {"namespace": "iceberg://lake-prod", "name": "sales.staging.stg_orders"},
    {"namespace": "iceberg://lake-prod", "name": "sales.staging.stg_fx_rates"}
  ],
  "outputs": [
    {"namespace": "iceberg://lake-prod", "name": "sales.marts.fct_orders"}
  ]
}

The facet _schemaURL versions above are illustrative; record the versions your client library actually emits in section 5.

5. Required facets

Every facet carries _producer and _schemaURL. This platform requires the following:

ID	Facet	Attached to	Requirement	Serves	Notes
FAC-001	`parent`	Run	Required when the run is started by an orchestrator	LUC-002	Links a dbt or Spark run to the Airflow task
FAC-002	`nominalTime`	Run	Required for scheduled jobs	LUC-002	Scheduled time, not actual start
FAC-003	`errorMessage`	Run	Required on `FAIL`	LUC-002	Message and stack trace; scrub values (section 11)
FAC-004	`sql`	Job	Required for SQL jobs	LUC-003, LUC-006	Query text; no literals with personal data
FAC-005	`sourceCodeLocation`	Job	Required	LUC-006	Repository URL, path and commit
FAC-006	`jobType`	Job	Required	All	Batch or streaming, integration, job type
FAC-007	`schema`	Dataset	Required for outputs	LUC-001	Field names and types
FAC-008	`dataSource`	Dataset	Required	All	Name and URI of the source system
FAC-009	`columnLineage`	Output dataset	Required for Silver and Gold (Example)	LUC-001, LUC-003, LUC-006	Transformation type and subtype per input field
FAC-010	`ownership`	Dataset, job	Recommended	LUC-002	Owner names; catalogue remains the source of truth
FAC-011	`lifecycleStateChange`	Dataset	Required on create, drop, truncate, rename	LUC-005, LUC-007	Drives clean-up and deletion checks
FAC-012	`dataQualityAssertions` or `dataQualityMetrics`	Input dataset	Recommended	LUC-002	Links quality results to the run
FAC-013	`outputStatistics`	Output dataset	Recommended	LUC-002	Row counts and bytes written
FAC-014	`symlinks`	Dataset	Required where one dataset has two names (table and path)	All	Joins the graph across naming schemes
FAC-015	`tags`	Dataset	Recommended	LUC-004	Classification tags copied from the catalogue

5.1 Column-level lineage transformation types

The column lineage facet records, for each output field, the input fields and how they were used. The specification distinguishes:

Type	Subtypes	Meaning
`DIRECT`	`IDENTITY`, `TRANSFORMATION`, `AGGREGATION`	The output value is computed from the input value
`INDIRECT`	`JOIN`, `GROUP_BY`, `FILTER`, `SORT`, `WINDOW`, `CONDITIONAL`	The input affects the output without being part of its value

A masking flag marks inputs that were hashed or otherwise obfuscated on the way to the output; LUC-004 uses it to show where personal data is pseudonymised.

Example: fct_orders.order_total_eur has a DIRECT TRANSFORMATION from stg_orders.order_total and stg_fx_rates.rate, and an INDIRECT JOIN from stg_fx_rates.currency_code.

6. Naming conventions

6.1 Datasets

Store	Namespace pattern	Name pattern	Corvid value (Example)
PostgreSQL	`postgres://{host}:{port}`	`{database}.{schema}.{table}`	`postgres://orders-db.internal:5432`, `orders.public.order_lines`
Snowflake	`snowflake://{organization name}-{account name}`	`{database}.{schema}.{table}`	`snowflake://corvid-finance`, `FIN.CORE.REVENUE_DAILY`
BigQuery	`bigquery`	`{project id}.{dataset name}.{table name}`	Not used
S3	`s3://{bucket name}`	`{object key}`	`s3://corvid-vendor-drop`, `marketing/spend/2026-10-29.csv`
Kafka	`kafka://{bootstrap server host}:{port}`	`{topic}`	`kafka://kafka-prod.internal:9092`, `checkout.events.v2`
Iceberg tables in a catalogue (local rule)	`iceberg://{catalogue name}`	`{namespace}.{table}`	`iceberg://lake-prod`, `sales.marts.fct_orders`

The Snowflake row follows the organisation and account name form that the OpenLineage naming page recommends over the legacy account locator; switching later breaks existing graph links. Where the specification does not list a store (the Iceberg row), record a local rule here and attach a symlinks facet that maps the table to its storage path. Check the current naming page before adding a store.

6.2 Jobs

Producer	Job namespace	Job name pattern
Airflow	`airflow-[env]`	`{dag_id}.{task_id}`
Spark	`spark-[env]`	`{appName}.{command}.{table}` as produced by the integration
dbt	`dbt-[env]`	`{project}.{folder}.{model}` (local rule)
Kafka Connect	`connect-[env]`	`{connector name}` (local rule)

Rules: one namespace per producer and environment; never reuse a job name for a different job; renames are announced and recorded as a symlink or alias in the backend.

7. Producers and integration points

ID	Producer	Integration	Version	Lineage level	Owner	Status
PRD-001 (Example)	Airflow	Airflow OpenLineage provider	[version]	Dataset, run, parent	Platform team	Live
PRD-002 (Example)	Spark	OpenLineage Spark integration	[version]	Dataset, column	Platform team	Live
PRD-003 (Example)	dbt	OpenLineage dbt integration run as a wrapper	[version]	Dataset, column, sql	Analytics engineering	Live
PRD-004 (Example)	Kafka Connect	Custom emitter from connector config	[version]	Dataset	Streaming team	Planned
PRD-005 (Example)	Vendor SFTP drop	Manual edge in catalogue	Not applicable	Dataset	Marketing data steward	Live, reviewed every 6 months
[PRD-nnn]	[Producer]	[Integration]	[version]	[Level]	[@owner]	[Status]

Producers send events over HTTP to the collector endpoint with an API key per producer, or to a Kafka topic when the network does not allow direct HTTP.

8. Lineage store and architecture

Component	Choice	Retention	Notes
Collector	[Tool]	Not applicable	Validates events against the 2-0-2 schema; rejects events without `run.runId` or dataset names
Graph store	[Tool]	Example: Run events 400 days; current graph kept indefinitely	Long enough to answer year-end audit questions
Dead letter store	[Tool]	Example: 30 days	Reviewed daily by the platform on-call
Catalogue sync	[Connector]	Not applicable	Pull every 15 minutes (see `data/data-catalog-metadata-standard` section 9)

9. Consumers and access

Consumer	Use cases	Access path	Access level
Producing teams	LUC-001	Pull request check lists downstream assets and owners of changed columns	All shared assets
On-call data engineers	LUC-002	Incident tool links from an alert to the run and its downstream datasets	All
Analytics engineers	LUC-003	Catalogue lineage view, column level	All
Data protection officer	LUC-004, LUC-005	Privacy data map filtered by personal data tags	All, including restricted datasets
Finance and auditors	LUC-006	Exported, versioned lineage report for named figures	Named Gold datasets only
Platform team	LUC-007	Unused dataset report	All

The impact check in pull requests works as follows:

10. Lineage quality and monitoring

ID	Check	Definition	Target (Example)	Action on breach
LQ-001	Producer coverage	Shared datasets with at least one producing job in the graph, divided by shared datasets	98%	Ticket to dataset owner
LQ-002	Run completion	Runs with a `START` and a terminal event within 24 hours, divided by runs started	99.5%	Investigate integration
LQ-003	Orphan datasets	Datasets read by jobs but with no producer and no manual edge	0 for Gold	Add producer or manual edge
LQ-004	Name collisions	Same physical table under two names without a symlink	0	Fix naming in producer
LQ-005	Column lineage coverage	Key fields of Silver and Gold datasets with column lineage	100% Gold, 90% Silver	Owner action
LQ-006	Freshness	Time from run end to event in the graph	95th percentile under 5 minutes	Platform alert
LQ-007	Invalid events	Events in the dead letter store per day	Under 0.1% of events	Fix producer

Example: in week 2026-W43 Corvid had 1,240 shared datasets, of which 1,221 had a producer in the graph, giving LQ-001 coverage of 98.5%.

11. Security and privacy

ID	Control	Detail
LSEC-001	No data values in facets	SQL text and error messages are scrubbed of literals; producers must not send sample rows
LSEC-002	Authenticated producers	One API key per producer, rotated every [90] days, scoped to its namespaces
LSEC-003	Access follows the catalogue	Users see lineage nodes for datasets they may discover in the catalogue; restricted nodes show as "restricted" with owner only
LSEC-004	Integrity	Events are append-only; manual edges carry author and date
LSEC-005	Retention	Run history kept as in section 8, then deleted

Lineage is processing metadata, but job names, SQL and error messages can still contain personal data. Treat the lineage store at the highest classification of the metadata it may hold.

12. Rollout plan

Phase	Window	Scope	Use cases delivered	Exit criteria
1 (Example)	Months 1 to 2	Airflow and dbt, dataset-level, collector and graph store	LUC-002, LUC-007	LQ-001 above 80%, LQ-002 above 99%
2 (Example)	Months 3 to 4	Spark, column-level for Gold, catalogue sync	LUC-001, LUC-003	Impact check live in CI for the Sales domain
3 (Example)	Months 5 to 6	Kafka Connect, manual edges, privacy data map	LUC-004, LUC-005	DPO signs off the personal data flow map
4 (Example)	Months 7 to 9	Audit export, column lineage for Silver	LUC-006	Finance controller accepts the report for one reported figure

13. Glossary and references

13.1 Glossary

Term	Meaning in this document
Lineage	The record of which processes read and wrote which data, and when
Run event	A message that reports the state of one execution of a job and its inputs and outputs
Facet	A named, versioned block of extra metadata attached to a run, job or dataset
Namespace	The prefix that, with a name, identifies a job or dataset uniquely
Column-level lineage	Lineage between individual fields, with how each input was used
Impact analysis	Finding everything downstream that a change could affect
Symlink	An alternative name for the same dataset, used to join graphs

13.2 References

Reference	Version or date checked	Use
OpenLineage specification, JSON schema, https://openlineage.io/spec/2-0-2/OpenLineage.json	2-0-2; checked 2026-10-30	Event types, required fields of run, job and dataset
OpenLineage object model, https://openlineage.io/docs/spec/object-model	Checked 2026-10-30	Run, job and dataset events; facet families
OpenLineage naming conventions, https://openlineage.io/docs/spec/naming	Checked 2026-10-30	Dataset and job naming patterns in section 6
OpenLineage column lineage facet, https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet	Checked 2026-10-30	Transformation types and subtypes, masking flag
[Lineage backend documentation]	[Version]	[Storage and API]

Standard and licence note

Based on the OpenLineage specification 2-0-2 and its published facet and naming documentation (OpenLineage, an LF AI and Data Foundation project, Apache License 2.0). Event types, field names, facet names and naming patterns are used as identifiers; no specification text is reproduced. The use cases, requirements, quality checks, local naming rules and examples are original, and Corvid Commerce is fictional. This template is original text written for this collection. Check the current OpenLineage specification and the documentation of your integrations before use. Template licence: MIT.