# System Design: [Key management and secrets service name]

> Reference design for an internal key management service (KMS) and secrets
> service: a root of trust in hardware security modules (HSMs), a key
> hierarchy, envelope encryption with data keys, key versions and rotation,
> key states, a secrets store with leases and dynamic credentials, policy and
> audit, caching, and multi-region availability. Key lifecycle and
> cryptoperiods follow NIST SP 800-57 Part 1 Revision 5, "Recommendation for
> Key Management: Part 1 - General" (May 2020); Revision 6 was released as an
> initial public draft on 5 December 2025 and is not final as of October
> 2026. It follows the structure of the System Design Document template,
> adds a 45-minute interview path, access-pattern and failure-mode tables and
> a cited "How real systems do it" box, and is pre-filled with a worked
> design to adapt. Replace every `[bracketed]` value and every row marked
> *Example*; values marked *Example* are assumptions, not benchmarks. This is
> a design format, not security certification advice; confirm cryptographic
> choices with your security team against the current NIST publications.

<!--
Guidance: How to use this template
- Separate the two products early: a KMS that never releases its top keys
  and performs cryptographic operations, and a secrets service that stores
  and hands out credentials. They share policy, audit and the root of trust,
  but have different load and failure profiles.
- Fill the call-rate estimate in 3.2 before choosing HSM counts. Naive
  envelope encryption with one KMS call per object is the most common
  sizing mistake.
- Decide the failure behaviour (section 7.6) with the data owners: a KMS
  outage makes encrypted data unreadable, so availability is a security
  requirement here.
- Pair this design with the cryptography and secrets management standards
  listed in 12.3; those set policy, this document sets the system.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-KMS-001] |
| System / feature | [Internal KMS and secrets service] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Confidential] |
| Difficulty | Advanced |
| Key management basis | [NIST SP 800-57 Part 1 Rev. 5 (May 2020); Rev. 6 initial public draft tracked] |
| HSM validation required | [FIPS 140-3 level, per security policy] |
| Related | [Cryptography standard], [Secrets management standard], [ADR: HSM vendor] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Security architect | [@name] | [Decision] | YYYY-MM-DD |
| Cryptography reviewer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Compliance | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Added data key reuse per writer after HSM sizing showed 120 HSMs for the naive design |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added dynamic database credentials to the secrets service |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* About 3 000 services keep database passwords and API tokens in
environment files and deployment variables, rotated by hand once a year at
best. Data at rest is encrypted by each storage system with its own static
key, so no one can say which keys protect which data, rotate them, or prove
who used them. An auditor asked for a key inventory and an access log; we
could provide neither.]

### 1.2 Proposed solution

[*Example:* A regional KMS holds customer master keys (KEKs) whose key
material never leaves HSMs in plaintext. Services encrypt data with data
encryption keys (DEKs) generated by the KMS and store each DEK wrapped under
a KEK next to the data (envelope encryption). Writers reuse one DEK for at
most 5 minutes and readers cache unwrapped DEKs for 5 minutes, so the KMS
sees thousands rather than millions of calls per second. KEKs have versions;
rotation adds a version for new encryption while old versions still
decrypt. A secrets service, encrypted by the same KMS, stores static secrets
with versions and issues short-lived dynamic credentials. Workloads
authenticate with their platform identity, every operation is authorised by
policy and written to an append-only audit log.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* One KMS call per object, or data key reuse and caching? | Reuse per writer for 5 min; cache unwrapped DEKs for 5 min | 3.2, 7.2 |
| Q-02 | *Example:* What does rotation mean for stored data? | New KEK version yearly; re-wrap DEKs, do not re-encrypt data | 7.3 |
| Q-03 | *Example:* Behaviour when the KMS is unreachable? | Fail closed for new keys; serve cached DEKs until their TTL | 7.6 |
| Q-04 | *Example:* Static secrets or dynamic credentials for databases? | Dynamic credentials with 1 h leases where the database supports it | 7.5 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

Use this order when the document serves an interview or a short design
review. The written sections below are the long form of each step.

| Minutes | Cover | Output on the whiteboard |
|---|---|---|
| 0 to 5 | Scope: keys never leave HSMs, envelope encryption, secrets, audit; below-the-line items (2.3) | Three to five requirements |
| 5 to 10 | Estimates: naive KMS calls, calls with reuse and caching, HSM count (section 3) | "Caching cuts 1.2 million calls/s to about 51 000" |
| 10 to 15 | Interface: `GenerateDataKey`, `Decrypt`, `GetSecret` (section 4) | Three calls with their inputs |
| 15 to 25 | High-level design: API tier, HSM tier, key store, secrets store, audit (section 6) | One diagram |
| 25 to 40 | One deep dive: envelope encryption and caching (default, 7.2), or rotation and key states (7.3, 7.4), or availability versus security (7.6) | Options, choice, reason |
| 40 to 45 | Failures (HSM loss, region loss, key compromise), the 10x row in section 10 | Two failure rows |

Start the default deep dive with the naive answer: send every object to the
KMS to encrypt. Show that payload size limits and call volume make that
impossible, then introduce envelope encryption, then show that even one KMS
call per object is too many and introduce DEK reuse and caching with their
limits.

## 2. Requirements

<!-- Guidance: Split core requirements from below-the-line items so the design stays focused. Non-functional targets need numbers and a way to measure them. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Create, describe, enable, disable and schedule deletion of KEKs | Must |
| FR-02 | *Example:* Generate data keys returned in plaintext and wrapped form | Must |
| FR-03 | *Example:* Decrypt (unwrap) data keys, bound to an encryption context | Must |
| FR-04 | *Example:* Automatic and on-demand KEK rotation with versions | Must |
| FR-05 | *Example:* Store, version and retrieve static secrets | Must |
| FR-06 | *Example:* Issue dynamic database credentials with leases | Should |
| FR-07 | *Example:* Sign and verify with asymmetric keys | Should |
| FR-08 | *Example:* Import key material from an external HSM (bring your own key) | Could |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Key confidentiality | *Example:* KEK and root key material never exists in plaintext outside HSMs | Design review, HSM configuration audit |
| NFR-02 | Latency | *Example:* p99 at or below 10 ms for `Decrypt` within a region | API timing |
| NFR-03 | Availability | *Example:* 99.99% per region for decrypt operations | Successful responses |
| NFR-04 | Audit | *Example:* Every KMS and secret operation logged, tamper-evident, kept 1 year | Log completeness checks |
| NFR-05 | Durability | *Example:* No loss of any key version that still protects data | Backup restore drill |
| NFR-06 | Separation of duties | *Example:* No single person can export, delete or restore root key material | Quorum controls, drill |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

Name these explicitly so reviewers and interviewers do not pull the design
toward them.

- [*Example:* public certificate issuance and TLS certificate lifecycle (see the PKI architecture)]
- [*Example:* end-user password storage (handled by the identity provider)]
- [*Example:* payment HSM functions such as PIN translation]
- [*Example:* migration of every algorithm to post-quantum schemes (tracked in the PQC migration plan)]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Back-of-the-envelope numbers decide the shape of the design. Show inputs, formulas and results; round generously and say which way you rounded. -->

### 3.1 Assumptions

All figures are per region.

| Input | Symbol | Value | Source |
|---|---|---|---|
| Encrypted object writes at peak | $w$ | *Example:* 200 000/s | Storage metrics |
| Encrypted object reads at peak | $r$ | *Example:* 1 000 000/s | Storage metrics |
| Writer instances | $n_w$ | *Example:* 1 000 | Deployment inventory |
| DEK reuse window per writer | $T$ | *Example:* 300 s | Policy (7.2) |
| Reader DEK cache hit ratio | $h$ | *Example:* 0.95 | Load test with 5 min TTL |
| Workload instances fetching secrets | $i$ | *Example:* 60 000 (3 000 services x 20) | Platform inventory |
| Secrets per instance, lease length | | *Example:* 20 secrets, 1 h | Secrets inventory |
| HSM symmetric operations per second | $c$ | *Example:* 10 000 | Vendor data sheet, confirm by test |
| Audit event size | | *Example:* 500 B | Log schema |

### 3.2 KMS call rate

Naive design, one KMS call per object:

$$
O_{naive} = w + r = 2 \times 10^{5} + 10^{6} = 1.2 \times 10^{6} \ \text{ops/s} \Rightarrow \frac{1.2 \times 10^{6}}{10\,000} = 120 \ \text{HSMs}
$$

With DEK reuse and reader caching:

$$
O_{gen} = \frac{n_w}{T} = \frac{1\,000}{300} \approx 3.3 \ \text{/s}, \qquad
O_{dec} = r (1 - h) = 10^{6} \times 0.05 = 50\,000 \ \text{/s}
$$

Secrets add $60\,000 \times 20 / 3\,600 \approx 333$ reads/s for lease
renewals and about 400/s during a deploy that restarts 10% of instances
within 5 minutes. The total is about 50 737 operations/s:

$$
N_{HSM} = \frac{50\,737}{10\,000} \times 1.5 \approx 7.6 \Rightarrow 8 \ \text{HSMs per region}
$$

Each DEK then protects about $200\,000 / 1\,000 \times 300 = 60\,000$
objects, far below the limit of $2^{32}$ AES-GCM invocations with random IVs
per key in NIST SP 800-38D section 8.3.

### 3.3 Storage and audit

- Secrets: 100 000 secrets x 10 versions x 4 KiB is about 4.1 GB, small.
- Data keys: one new DEK per writer every 5 minutes is 288 000 wrapped DEKs
  per day and about 105 million per year; they live with the data, not in the
  KMS.
- Re-wrap after a KEK rotation: 105 million DEKs at 2 000 re-wraps/s take
  about 14.6 hours, a background job, not an outage.
- Audit: 50 737 events/s x 86 400 s x 500 B is about 2.19 TB per day before
  compression, about 0.8 PB per year. Cache hits are not KMS events, which is
  why the audit volume stays manageable; the log goes to append-only storage
  with compression and tiering.

### 3.4 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Naive KMS calls | 1.2 million/s, 120 HSMs | Never call the KMS per object |
| Calls with reuse and caching | About 51 000/s | 8 HSMs per region with 50% headroom |
| Objects per DEK | About 60 000 | Well inside the AES-GCM invocation limit |
| Re-wrap after rotation | About 14.6 h | Rotation is a background job |
| Audit | About 2.2 TB per day raw | Append-only, compressed, tiered storage |

## 4. API design

<!-- Guidance: Define the contract first: resources, methods, request and response shapes, errors, idempotency, pagination, rate limits and versioning. Use RFC 9457 problem details for errors. -->

### 4.1 Operations

| Interface | Operation | Purpose |
|---|---|---|
| KMS | `CreateKey(spec, usage, policy)` | New KEK in state pre-activation or active |
| KMS | `GenerateDataKey(key_id, context) -> {plaintext, wrapped}` | Envelope encryption: DEK for local use plus wrapped copy to store |
| KMS | `Decrypt(wrapped, context) -> plaintext` | Unwrap a DEK; key ID and version are read from the wrapped blob |
| KMS | `Encrypt(key_id, plaintext <= 4 KiB, context)` | Small payloads only, such as a secret value |
| KMS | `ReEncrypt(wrapped, new_key_id)` | Re-wrap under the current version without exposing the DEK to the caller |
| KMS | `RotateKey(key_id)`, `DisableKey`, `ScheduleKeyDeletion(days)` | Lifecycle |
| Secrets | `PutSecret(path, value)`, `GetSecret(path, version?)` | Static secrets |
| Secrets | `IssueCredential(role) -> {username, password, lease}` | Dynamic credentials |
| Secrets | `RenewLease(id)`, `RevokeLease(id)` | Lease control |

### 4.2 Wrapped key format

The wrapped DEK is self-describing so `Decrypt` needs no key ID from the
caller, and the encryption context is bound as additional authenticated data
(AAD): unwrapping with a different context fails.

```json
{"format": 1, "keyId": "kek-orders-eu", "keyVersion": 4, "alg": "AES-256-GCM", "iv": "[base64, 12 bytes]", "ciphertext": "[base64 wrapped DEK]", "tag": "[base64, 16 bytes]", "contextHash": "[sha-256 of canonical context]"}
```

```json
{"context": {"service": "orders", "table": "invoices", "tenant": "t_8812"}}
```

Errors use RFC 9457 problem details; `AccessDenied`, `KeyDisabled`,
`ContextMismatch` and `Throttled` are distinct types so clients can tell a
policy problem from a capacity problem.

## 5. Data model

<!-- Guidance: Choose storage by access pattern, not by habit. Show entities, keys, partitioning and the queries each store must answer. -->

### 5.1 Entities

KEK versions are stored only as blobs wrapped by an HSM-held domain key; DEKs
are not stored in the KMS at all.

```mermaid
erDiagram
  KEY {
    string key_id PK
    string usage
    string state
    string policy_id FK
    string rotation_period
    int current_version
  }
  KEY_VERSION {
    string key_id FK
    int version PK
    string state
    blob wrapped_material
    datetime activated_at
    datetime deactivated_at
  }
  POLICY {
    string policy_id PK
    string document
    int revision
  }
  SECRET {
    string path PK
    int current_version
    string kek_id FK
  }
  SECRET_VERSION {
    string path FK
    int version PK
    blob ciphertext
    datetime created_at
  }
  LEASE {
    string lease_id PK
    string role
    datetime expires_at
  }
  KEY ||--|{ KEY_VERSION : has
  POLICY ||--o{ KEY : governs
  KEY ||--o{ SECRET : protects
  SECRET ||--|{ SECRET_VERSION : has
```

### 5.2 Storage choices

| Data | Store | Key | Why |
|---|---|---|---|
| Key metadata and wrapped versions | *Example:* Replicated consensus-based store, per region with cross-region copies | `key_id`, `version` | Small, strongly consistent, must never lose a version |
| Secrets | Same store, values encrypted under a KEK | `path`, `version` | Versioned, read-heavy, small |
| Leases | Same store with TTL index | `lease_id` | Expiry drives revocation |
| Audit log | Append-only object storage with hash chaining | Time, region | Tamper-evident, large |
| Domain (root) keys | Inside HSMs only, backed up as quorum-protected shares | Not addressable | Never exported in plaintext |

### 5.3 Access patterns

| Access pattern | Frequency | Key or index that serves it | Partition key rationale |
|---|---|---|---|
| Unwrap a DEK | About 50 000/s | `key_id` and `version` from the wrapped blob, metadata cached on API hosts | Key metadata is tiny and cached; only the HSM call is remote |
| Generate a DEK | About 3/s steady, more at deploys | `key_id` current version | Same cache |
| Read a secret | About 333/s, 400/s in deploy storms | `path` and version | Partitioned by path prefix (team) so one team's storm stays local |
| Expire leases | Continuous | TTL index on `expires_at` | Time-ordered index, scanned by revocation workers |
| Find all data under a key version | Rotation, compromise | Inventory of wrapped DEKs reported by storage systems, keyed by `key_id` and version | The KMS cannot know where data lives; storage systems report it |
| Answer an auditor | Rare | Audit log by key, principal and time | Indexed copy in the security analytics platform |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. Name components by responsibility. -->

```mermaid
flowchart LR
  wl[Workload with platform identity] -->|"mTLS + identity token"| api[KMS and secrets API]
  api --> authz[Policy engine]
  api --> meta[(Key and secret store)]
  api -->|"wrap, unwrap, generate"| hsm[HSM cluster]
  api --> audit[[Audit log]]
  wl --> lib[Encryption SDK with DEK cache]
  lib --> api
  lib -->|"ciphertext + wrapped DEK"| store[(Application storage)]
  dyn[Dynamic credential engine] --> db[(Databases)]
  api --> dyn
  rot[Rotation and re-wrap jobs] --> api
  ceremony[Key ceremony, quorum of custodians] --> hsm
```

| Component | Responsibility | Scales by |
|---|---|---|
| KMS and secrets API | Authenticate, authorise, call HSMs, return results, write audit | Stateless hosts |
| Policy engine | Evaluate key and secret policies (who, which operation, which context) | Cached policies on each host |
| HSM cluster | Hold domain keys, wrap and unwrap KEK versions and DEKs | Operations per second (3.2) |
| Key and secret store | Metadata, wrapped key versions, secrets, leases | Small, consensus replicated |
| Encryption SDK | Envelope encryption, DEK reuse and cache, AAD handling | With workloads |
| Dynamic credential engine | Create and revoke database users per lease | Leases per second |
| Rotation and re-wrap jobs | Scheduled rotation, re-wrap of stored DEKs | Background |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. Show the sequence of calls and the failure behaviour of each step. -->

The deep dives in this section, each with options, choice and reason:

| Deep dive | Options considered | Choice | Reason | Section |
|---|---|---|---|---|
| Key hierarchy | One key per service in software, HSM-held root with wrapped KEKs, everything in HSM slots | HSM-held domain key wraps KEK versions stored outside | Unlimited keys, keys never in plaintext outside HSMs | 7.1 |
| Data encryption | KMS encrypts data, one DEK per object, DEK reuse with caching | DEK reuse per writer, cache on readers | 120 HSMs down to 8 | 7.2 |
| Rotation | Re-encrypt all data, new version plus re-wrap, new version only | New version, re-wrap in background | Old data stays readable, no data rewrite | 7.3 |
| Unreachable KMS | Fail open, fail closed, bounded cache | Bounded cache, then fail closed | Availability without unlimited exposure | 7.6 |

### 7.1 Key hierarchy

| Level | Key | Where it lives | Lifetime (*Example*) |
|---|---|---|---|
| 0 | Domain (root) key | Only inside HSMs; backup split into shares held by custodians | Rotated per policy, with ceremony |
| 1 | KEK version (customer master key) | Wrapped by the domain key in the key store; unwrapped only inside HSMs | Originator use 1 year, kept while data needs it |
| 2 | DEK | Plaintext in workload memory for at most 5 minutes; wrapped copy stored with the data | One writer window |
| Data | Objects, rows, files | Application storage, encrypted with AES-256-GCM | As long as the data |

### 7.2 Envelope encryption with reuse and caching

```mermaid
sequenceDiagram
  participant W as Writer SDK
  participant K as KMS API
  participant H as HSM
  participant S as Storage
  participant R as Reader SDK
  W->>K: GenerateDataKey(kek-orders-eu, context)
  K->>H: generate DEK, wrap under KEK version 4
  H-->>K: plaintext DEK, wrapped DEK
  K-->>W: plaintext DEK, wrapped DEK
  W->>W: encrypt objects for up to 5 min with this DEK
  W->>S: ciphertext + wrapped DEK
  R->>S: read object
  S-->>R: ciphertext + wrapped DEK
  R->>R: cache lookup by wrapped DEK
  alt cache miss
    R->>K: Decrypt(wrapped DEK, context)
    K->>H: unwrap with KEK version 4
    H-->>K: plaintext DEK
    K-->>R: plaintext DEK
  end
  R->>R: decrypt object
```

Rules for reuse and caching:

- A writer uses one DEK for at most 5 minutes or a set number of objects,
  whichever comes first, and for one encryption context only.
- Each object gets a fresh random 96-bit IV; the count per DEK stays far
  below the SP 800-38D section 8.3 limit (3.2).
- Reader caches are keyed by the exact wrapped DEK bytes and context, hold
  plaintext DEKs in memory only, and expire after 5 minutes. A disabled KEK
  therefore stops all new decryption within 5 minutes.
- The trade-off: a compromised workload host exposes the DEKs in its cache,
  which cover at most 5 minutes of data per context. Highly sensitive data
  sets a shorter window or turns caching off.

### 7.3 Rotation and re-wrapping

Rotation creates a new KEK version that becomes the only one used for new
wrapping; old versions remain able to unwrap. Data is not re-encrypted:
re-wrapping a DEK under the new version (`ReEncrypt`) changes only the small
wrapped blob. A background job re-wraps every stored DEK when policy
requires retiring an old version, about 14.6 hours for a year of DEKs
(3.3). DEKs themselves are rotated implicitly every writer window.

NIST SP 800-57 Part 1 Rev. 5 section 5.3.6 discusses cryptoperiods by key
type. For symmetric data-encryption keys it suggests an originator-usage
period on the order of a day or a week when large volumes are encrypted
quickly, up to about two years for smaller volumes, and a recipient-usage
period of up to three years beyond that; for symmetric key-wrapping keys it
links the period to how many keys are wrapped. Set your own periods in the
cryptography standard and record them per key in the rotation period field.

### 7.4 Key states

The KMS tracks each key version through the states described in NIST SP
800-57 Part 1 Rev. 5 section 7. The diagram shows the transitions this
service allows.

```mermaid
stateDiagram-v2
  [*] --> PreActivation: CreateKey
  PreActivation --> Active: activate
  PreActivation --> Destroyed: cancel
  Active --> Suspended: suspend
  Suspended --> Active: resume
  Active --> Deactivated: rotation or end of use period
  Suspended --> Deactivated: retire
  Active --> Compromised: compromise declared
  Suspended --> Compromised: compromise declared
  Deactivated --> Compromised: compromise declared
  Compromised --> Destroyed: data re-protected
  Deactivated --> Destroyed: no data depends on it
  Destroyed --> [*]
```

| State | Encrypt or wrap | Decrypt or unwrap | Notes |
|---|---|---|---|
| Pre-activation | No | No | Created, not yet usable |
| Active | Yes | Yes | Current version of a KEK |
| Suspended | No | No | Temporary hold, for example during an investigation |
| Deactivated | No | Yes | Old version kept for existing data |
| Compromised | No | Only under break-glass, to re-protect data | Triggers the compromise runbook (9.1) |
| Destroyed | No | No | Key material deleted after a waiting period (for example 30 days) |

### 7.5 Secrets service

- **Static secrets**: stored encrypted under a KEK, versioned, read by path.
  Rotation writes a new version; consumers read "current" and the old
  version stays valid for an overlap window agreed with the target system.
- **Dynamic credentials**: for supported databases the engine creates a user
  with the role's grants for each lease (1 hour, renewable) and drops it on
  expiry or revocation. Leaked credentials expire on their own, and audit
  shows which workload held which user.
- **Delivery**: a sidecar or SDK fetches secrets at start and renews leases;
  secrets never go into images, repositories or environment files.
- **Authentication**: workloads prove identity with the platform's workload
  identity (signed service account token or certificate), never with a
  secret that itself needs storing.

### 7.6 Availability versus security

| Situation | Behaviour | Rationale |
|---|---|---|
| KMS unreachable, DEK in reader cache | Serve until cache TTL | Short outages invisible to users |
| KMS unreachable, cache miss | Fail closed with a retryable error | No way to decrypt without the key |
| KMS unreachable, writer needs a new DEK | Keep current DEK up to a hard maximum (for example 15 min), then fail closed | Bounded exposure |
| Region lost | Clients fail over to the paired region, where KEK versions are replicated under that region's domain key | Data stays readable |
| HSM cluster degraded | Throttle bulk jobs (re-wrap) before interactive calls | Protect latency for users |

### 7.7 How real systems do it

> **How real systems do it** (paraphrased, not copied). The AWS document
> "AWS KMS cryptographic details" describes a tiered service: API hosts that
> terminate TLS and authorise requests, and a fleet of FIPS 140-3 validated
> HSMs. A KMS key is a logical container for one or more HSM backing keys;
> each backing key version is exported only in encrypted form, wrapped by
> HSM-held domain keys, and stored in durable storage. Rotation adds a new
> backing key for new encryption while older ones still decrypt. The document
> lists domain keys as rotated daily (relaxed at times to at most weekly),
> derives a fresh encryption key from the backing key for each encryption,
> and describes data keys returned in plaintext and encrypted form for
> envelope encryption, with an optional encryption context bound as
> additional authenticated data. The document also says direct encryption
> under a KMS key suits only data under about 4 KB, which is why large data
> uses data keys. Read the original in 12.3; this box keeps only the general
> idea.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| HSM operations | DEK reuse and caching; HSMs added in pairs | *Example:* 8 HSMs per region at 50% headroom |
| API hosts | Stateless, policy and metadata cached | [Requests per host] |
| Hot keys | One KEK per service and context, not one global key | Per-key throttles |
| Deploy storms | Jittered lease renewal, per-team throttles | *Example:* 400 secret reads/s |
| Audit volume | Only cache misses produce KMS events; compression | Storage cost |

Decrypt latency budget (target p99 10 ms):

| Step | Budget |
|---|---|
| mTLS session reuse and authentication | 1 ms |
| Policy evaluation (cached) | 0.5 ms |
| HSM unwrap including queueing | 4 ms |
| Audit write (asynchronous, durable queue) | 1 ms |
| Network round trip in region | 1 ms |
| Headroom | 2.5 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes with their blast radius and recovery time, the threats, and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| HSM | Device failure | Health checks, error rate | Capacity of one of 8 HSMs | Remaining HSMs serve; replace and re-sync from cluster | *Example:* 0 (capacity only) |
| HSM cluster | Loss of all HSMs in a region | Synthetic decrypt probes | All cache misses in the region | Fail over to paired region; restore from quorum backup | *Example:* 5 min failover |
| Key store | Replica loss | Consensus health | None while a majority stays | Replace node | *Example:* 30 min |
| KEK | Accidental deletion | Deletion scheduled event | All data under that key | Mandatory waiting period with alerts; restore before expiry | *Example:* Minutes, during the waiting period |
| KEK version | Compromise | Detection or report | Data wrapped under that version | Mark compromised, rotate, re-wrap or re-encrypt affected data, review audit for misuse | *Example:* Per runbook, hours |
| Secrets engine | Database unreachable for dynamic credentials | Issue failures | New credentials for that database | Existing leases continue; extend renewals | *Example:* 15 min |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [Time] |

### 9.2 Security

| Threat | Control |
|---|---|
| Insider exports root keys | Domain keys non-exportable; backup shares require a quorum (for example 3 of 5 custodians) in a recorded ceremony |
| Stolen workload credential | Short-lived workload identity; key policies scoped to context; anomaly alerts on unusual key use |
| Confused deputy (service decrypts another tenant's data) | Encryption context with tenant ID bound as AAD and required by policy |
| Audit tampering | Hash-chained, append-only log copied to a separate security account |
| Ciphertext swapped between records | AAD includes record identity, so a moved ciphertext fails to decrypt |
| Future quantum attacks | AES-256 for symmetric keys; asymmetric keys inventoried for migration per the PQC plan |
| [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric or log | Alert threshold |
|---|---|---|
| Decrypt latency | p99 per region | Above 10 ms for 10 min |
| HSM utilisation | Operations per HSM | Above 60% of tested capacity |
| Access denials | Denied operations by principal and key | Spike vs. baseline |
| Unusual use | Key used from a new service or context | Any, for high-sensitivity keys |
| Rotation compliance | Keys past their rotation date | Any |
| Lease health | Expired but not revoked dynamic users | Any |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| HSMs | [$/HSM-month] | *Example:* 8 per region x 2 regions | [$] |
| API hosts | [$/host-month] | [Count] | [$] |
| Audit storage | [$/TB-month] | *Example:* About 2.2 TB per day before compression | [$] |
| **Total** | | | **[$]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Reviewers will propose the alternatives anyway; answer them here. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Build or buy | Cloud provider KMS; self-run open-source secrets manager; build on HSMs | [Choice] | [Reason] | [Cost] |
| DEK scope | Per object; per writer window; per table | Per writer window | 120 HSMs down to 8 | Larger exposure per stolen DEK |
| Reader cache TTL | None; 1 min; 5 min | 5 min | 95% hit ratio | Revocation takes up to 5 min to bite |
| Rotation | Re-encrypt data; re-wrap DEKs | Re-wrap | No data rewrite | Old KEK versions kept while data needs them |
| Secrets | Static only; dynamic where possible | Dynamic where possible | Leaked credentials expire | Target systems must support user creation |
| What changes at 10x / 0.1x | 10x (about 507 000 operations/s): about 76 HSMs by the 3.2 formula, so add an intermediate per-service key held in HSM-backed memory on hardened KMS hosts, or longer reuse windows, and accept a larger trusted computing base. 0.1x (about 5 000 operations/s): one HSM pair per region, or a cloud provider KMS | Review at each doubling of decrypt rate | HSM operations grow first | [Note] |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | HSM cluster, key ceremony, KMS API, audit | Ceremony recorded; restore from backup shares tested | YYYY-MM-DD |
| M2 | Encryption SDK with reuse and cache; first storage system | Decrypt p99 under 10 ms at test load | YYYY-MM-DD |
| M3 | Secrets service, static secrets, migration of top 100 services | No secrets in repositories for those services | YYYY-MM-DD |
| M4 | Dynamic credentials, automatic rotation, second region | Region failover drill passes | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Storage systems adopt the SDK one at a time. New data is written
with KMS envelopes while old data stays under the old static key; a
background job re-encrypts old data. Rollback keeps both readers in place,
so data written either way remains readable.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* KMS outage makes data unreadable | Medium | High | Two regions, caches, drills | [@name] |
| R-02 | *Example:* Loss of custodian shares | Low | High | Shares in separate locations; annual verification | [@name] |
| R-03 | *Example:* Teams bypass the SDK and call the KMS per object | Medium | Medium | Per-principal throttles; SDK as the supported path | [@name] |
| R-04 | *Example:* NIST SP 800-57 Rev. 6 final changes cryptoperiod or algorithm guidance | Medium | Low | Track the draft; review the cryptography standard on publication | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary and open questions. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Which FIPS 140-3 security level do contracts require for the HSMs? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Do any tenants require keys held in their own HSM (external key store)? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Glossary

| Term | Meaning |
|---|---|
| KMS | Key management service that performs operations with keys it never releases |
| HSM | Hardware security module: tamper-resistant device that stores keys and performs cryptography |
| KEK | Key-encryption (key-wrapping) key that protects other keys |
| DEK | Data encryption key used directly on data |
| Envelope encryption | Encrypting data with a DEK and storing the DEK wrapped by a KEK |
| Encryption context | Non-secret key-value data bound to a ciphertext as AAD |
| Cryptoperiod | Time span during which a key is authorised for use |
| Lease | Time-limited grant of a dynamic credential |
| [Term] | [Meaning] |

### 12.3 References

| Reference | Use |
|---|---|
| NIST [SP 800-57 Part 1 Rev. 5](https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final), "Recommendation for Key Management: Part 1 - General", May 2020, [doi:10.6028/NIST.SP.800-57pt1r5](https://doi.org/10.6028/NIST.SP.800-57pt1r5) | Cryptoperiods (5.3.6), key states (7) |
| NIST [SP 800-57 Part 1 Rev. 6 initial public draft](https://csrc.nist.gov/pubs/sp/800/57/pt1/r6/ipd), 5 December 2025 (comments closed 5 February 2026; not final as of October 2026) | Upcoming changes, including post-quantum algorithms |
| NIST [SP 800-38D](https://csrc.nist.gov/pubs/sp/800/38/d/final), GCM and GMAC, November 2007 (revision in progress) | IV and invocation limits (8.3) |
| NIST [SP 800-38F](https://csrc.nist.gov/pubs/sp/800/38/f/final), Methods for Key Wrapping, December 2012 | Key wrapping modes |
| NIST [FIPS 140-3](https://csrc.nist.gov/pubs/fips/140-3/final), Security Requirements for Cryptographic Modules, 22 March 2019 | HSM validation |
| AWS, ["AWS KMS cryptographic details"](https://docs.aws.amazon.com/kms/latest/cryptographic-details/intro.html), AWS documentation (accessed October 2026) | 7.7 |
| RFC 9457 Problem Details for HTTP APIs | Error format |
| Secrets Management Standard (`security/secrets-management-standard`) | Policy for secret types, rotation and storage |
| Cryptography Standard (`security/cryptography-standard`) | Approved algorithms and cryptoperiods |
| PKI and Certificate Architecture (`security/pki-certificate-architecture`) | Certificates, out of scope here |
| Post-Quantum Cryptography Migration Plan (`security/pqc-migration-plan`) | Asymmetric key migration |
| Authorization Service Design (`system-design/design-authorization-service`) | Policy evaluation patterns |
| Distributed Lock and Coordination Service Design (`system-design/design-distributed-lock-coordination`) | Consensus store for key metadata |

## Standard and licence note

Based on NIST SP 800-57 Part 1 Rev. 5 (May 2020), NIST SP 800-38D, SP 800-38F
and FIPS 140-3, which are public US government publications, paraphrased
here in original words with section numbers only. Revision 6 of SP 800-57
Part 1 was an initial public draft as of October 2026; check the NIST page
for the current status before relying on cryptoperiod or algorithm guidance.
The AWS documentation is paraphrased in 7.7 only; no text, figures or tables
were copied, and all numbers marked *Example* are illustrative. This is a
design format, not security certification advice. Original text written for
this collection. Template licence: MIT.