Data Retention and Deletion Standard Template (GDPR Art. 17, SP 800-88)

Data Retention and Deletion Standard: [Organisation or platform name]

Technical standard for how long data is kept on a data platform and how it is removed: the method for writing retention rules (trigger event, period, legal basis, holds), deletion classes, standard retention periods, deletion mechanics per store (operational databases, lakehouse tables with snapshot expiry, vacuum and deletion vectors, backups, search indexes, logs and machine learning features), propagation of erasure requests under GDPR (Regulation (EU) 2016/679) Article 17, verification evidence, and media sanitisation following NIST SP 800-88 Rev. 2 (September 2025). The concepts of deletion classes and deletion rules follow the structure of ISO/IEC 27555:2021 (guidelines on PII deletion), cited by name only. Replace every [bracketed] value and every row marked Example. This is a documentation format, not legal advice.

Document control

Approvals

Revision history

Contents

• 1. Purpose, scope and principles
• 2. Roles and responsibilities
• 3. Retention rule method
• 4. Deletion classes
• 5. Standard retention periods
• 6. Deletion mechanics per store
• 7. Legal holds and suspension
• 8. Erasure request propagation
• 9. Verification and deletion evidence
• 10. Media sanitisation
• 11. Exceptions and monitoring
• 12. Traceability matrix
• 13. Glossary and references
• Standard and licence note

1. Purpose, scope and principles

1.1 Purpose

This standard makes sure that data is kept only as long as a purpose or an obligation requires, that deletion happens on schedule in every copy of the data, and that [Organisation name] can prove it. It supports the storage limitation principle of GDPR Article 5(1)(e), the right to erasure in Article 17, and the time limits recorded in the record of processing activities under Article 30(1)(f) (compliance/gdpr-ropa).

1.2 Scope

All data stores that hold organisation or customer data: operational databases, event streams, the lakehouse and warehouse, derived marts, search indexes, caches, logs and traces, feature stores, model training sets, exports, backups and archives, including those operated by processors.

1.3 Principles

1. Every dataset has a rule. No dataset is released to production without a retention rule ID in its data contract (data/data-contract).
2. Rules are expressed as trigger plus period. Calendar dates alone are not used.
3. Deletion means unrecoverable by normal means in every primary store, and aged out of backups within the stated backup window.
4. Holds override deletion, and only holds do.
5. Deletion is automated and produces evidence; manual deletion is an exception.
6. Derived data follows its source: lineage drives propagation.

2. Roles and responsibilities

3. Retention rule method

3.1 Rule structure

Each retention rule has these fields:

3.2 Writing a rule

1. Identify the purpose and any legal obligation to keep the data.
2. Choose the trigger that best reflects when the purpose ends (for example account closure, not account creation).
3. Set the period to the minimum that satisfies the purpose or the obligation; record the source of any statutory period.
4. Choose the action at end. Prefer deletion; anonymisation needs a completed assessment (compliance/de-identification-assessment).
5. Assign a deletion class.
6. Get approval from the data owner and, for personal data, the DPO.

3.3 Rule register

The register is the single source of rule IDs and periods. The record of processing activities (compliance/gdpr-ropa), the records schedule (compliance/iso-15489-records-schedule), DPIAs and data sharing agreements quote the rule ID and its period exactly as written here; when a period changes, they change in the same release. Rules RET-010 to RET-015 are the platform defaults in section 5.

4. Deletion classes

Rules:

• A dataset with mixed classes is split, or takes the class of its longest-lived element only when the owner records why splitting is impractical.
• Personal data never sits in DC-5.

5. Standard retention periods

6. Deletion mechanics per store

6.1 Operational databases (OLTP)

• Hard delete rows by primary key, or delete by partition for time-based rules.
• Soft-delete flags are allowed only as a short grace period ([30] days); a scheduled job then hard deletes.
• Cascade to child tables or nullify foreign keys deliberately; check orphaned rows.
• Run vacuum or compaction according to the engine so freed pages are reused; remember that write-ahead logs, replicas and point-in-time recovery windows still hold the data until they roll over.
• Change data capture streams carry the delete as an event; downstream consumers must act on it.

6.2 Lakehouse tables: Apache Iceberg

Iceberg tables are versioned: each commit creates a snapshot, and old snapshots keep references to data files that contain deleted rows.

Configure snapshot retention so step 3 runs within the deletion deadline. Time travel stops working for expired snapshots, which is the intended effect.

Metadata residue: manifest files record per-file column statistics, including lower and upper bounds. With the default metrics mode (truncate(16)) the bounds of a string column such as email are real values (the first 16 characters), and they stay in manifests until snapshot expiry removes every manifest that lists the old file. For personal-data columns, set write.metadata.metrics.column.<column> to none or counts (bounds are then not persisted), or count manifest expiry in the residual window. Changing the metrics mode affects new files only; rewrite existing files to clear old bounds.

6.3 Lakehouse tables: Delta Lake

Notes: the vacuum threshold must be measured from the purge commit, not the original delete. If change data feed is enabled, its files also hold deleted values until vacuumed. Shallow and deep clones are separate copies and need their own deletion. Time travel to versions before the purge stops working once VACUUM removes their files; this is the intended effect.

Column statistics: exclude personal-data columns from statistics collection (list only non-personal columns in delta.dataSkippingStatsColumns, or order the schema so they fall outside delta.dataSkippingNumIndexedCols), or include the log retention period in the residual window. Changing these properties affects statistics written from then on; existing log entries keep their statistics until they age out.

6.4 Apache Hudi and other formats

Apache Hudi (current release line 1.2.x) keeps older file versions for time travel and incremental queries. A delete writes a new file version or, for merge-on-read tables, a log entry that compaction later merges. The cleaner service removes older file versions according to the configured cleaning policy (for example by number of commits or file versions retained), and timeline archival moves old instants out of the active timeline. Set the cleaning policy so the residual window fits the deletion deadline, and verify the configuration names against the documentation of the version you run. For any other format, record the equivalent of: logical delete, rewrite, removal of unreferenced files, and metadata history expiry.

6.5 Warehouses and marts

• Rebuild derived models from cleaned sources; incremental dbt models need an explicit delete step or a periodic full refresh.
• Warehouse time travel and fail-safe features keep data for a set period after deletion: record that period here: [N days].
• Snapshots (slowly changing dimension history) must apply erasure to historical rows, not only the current row.

6.6 Search indexes and caches

• Delete documents by ID; force merge or segment rewrite so deleted documents leave disk ([schedule]).
• Use index lifecycle policies for time-based rules (DC-2).
• Caches use TTLs no longer than the class period; invalidate on erasure.

6.7 Logs, traces and observability

• Avoid personal data in logs by design (security/security-logging-standard).
• Apply retention at the log platform by index or bucket lifecycle.
• Erasure requests do not usually require editing immutable security logs; document that position, the basis and the retention period.

6.8 Machine learning features and models

• Feature store tables follow source rules; online stores need TTLs.
• Training sets are snapshots: record which rule applies and delete at the end of the period.
• Where a model was trained on data later erased, record whether retraining is required, using the risk assessment in the model documentation (ai-data/model-card).
• Embeddings and vector indexes of personal content are deleted with the source document.

6.9 Backups and archives

• Backups are not edited to remove single records. They age out on the rolling window in ops/backup-policy (RET-014).
• On restore, replay the erasure log (section 8.4) before the restored system goes live, so erased records do not reappear.
• Archives under DC-4 are deleted at the end of the statutory period by a scheduled job with evidence.

6.10 Store register

7. Legal holds and suspension

• Legal issues holds through the hold register with scope, reason, start date and review date.
• Deletion jobs check the hold register before deleting and skip held data, logging each skip.
• Holds are reviewed [quarterly]; on release, deletion resumes and any overdue data is deleted within [30] days.
• Erasure requests for held data are answered with the reason for restriction, as advised by Legal.

8. Erasure request propagation

8.1 Workflow

The sequence below shows one request from receipt to response. Each store completes its own ticket and returns evidence before the response is sent.

8.2 Resolving the subject

• Map the request to stable subject keys (customer ID, email hash, device IDs) through an identity resolution table.
• Use lineage (for example OpenLineage events or the catalogue) to list every dataset containing those keys or derived from them.
• Include free-text stores (tickets, chat logs) by search.

8.3 Execution

• Batch erasure requests per store per day to limit rewrite cost on lakehouse tables.
• The physical purge deadline (sections 6.2 and 6.3) must fall inside the response deadline.
• Where data is kept under an exception (for example statutory invoices), restrict processing and record the reason.

8.4 Erasure log

Keep a minimal erasure log (subject key hash, request ID, date, stores completed) so erasure can be reapplied after a backup restore. The log itself holds no more personal data than necessary and has its own retention rule.

9. Verification and deletion evidence

Verification rules:

• Run a verification query per store after each purge cycle; failures open an incident with severity per ops/incident-severity-matrix (SEV3 by default, SEV2 if personal data is overdue by more than the response deadline).
• Sample [5] percent of erasure requests each quarter for end-to-end testing across stores, including a restore test.

10. Media sanitisation

• Choose the sanitisation method (Rev. 2 section 3.1) per media and classification: Clear (logical techniques against simple, non-invasive recovery through the normal interface), Purge (recovery infeasible with state-of-the-art laboratory techniques, media possibly reusable) or Destroy (as Purge, media no longer usable; not applicable to logical or virtual storage). Cryptographic erase is a purge technique where encryption was enabled from first use and keys are managed properly. See IEEE 2883 for media-specific techniques.
• Restricted data media: Purge or Destroy.
• Cloud storage: rely on the provider's documented sanitisation attestations and on cryptographic erase of customer-managed keys.
• Keep a sanitisation record for each item: media identifier, data classification, method, tool, operator, verification result and date.
• Vendors performing destruction provide a certificate per batch.

11. Exceptions and monitoring

12. Traceability matrix

13. Glossary and references

13.1 Glossary

13.2 References

Standard and licence note

Based on Regulation (EU) 2016/679 (GDPR) and NIST SP 800-88 Rev. 2 (September 2025), with structure aligned with ISO/IEC 27555:2021, which is cited by name only; no ISO text, tables or definitions are reproduced. Lakehouse mechanics refer to the Apache Iceberg specification and the Delta Lake documentation (Apache-2.0 projects), described in original wording. NIST publications are US government works and EU legislation is free to reuse. All text and examples are original and use a fictional organisation. Check the current official versions, the consolidated GDPR text and the documentation of the engine versions you run before use. This is a documentation format, not legal advice. Template licence: MIT.