# Kubernetes Cluster Design: [Cluster or fleet name]

> Design record for one Kubernetes cluster or a fleet of clusters built to the same pattern: tenancy model, managed or self-managed control plane, version and upgrade cadence, control plane availability, node pools, networking and NetworkPolicy, ingress and Gateway API, identity and RBAC, Pod Security Admission, admission policy, secrets, storage, observability, backup of etcd and workloads, cost and capacity, and the hardening baseline. Based on the Kubernetes documentation (kubernetes.io, CC BY 4.0, release 1.37 current as of October 2026) and the NSA and CISA Kubernetes Hardening Guidance version 1.2 (August 2022). The CIS Kubernetes Benchmark is named as an external baseline only. Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Write one record per cluster pattern. A fleet of identical clusters (for example one per region) shares one record with a per-cluster inventory in section 1.4.
- This record decides how the cluster is built and governed. The landing zone that hosts it is in Landing Zone Design (`architecture/landing-zone-design`); the paved road that teams use to deploy onto it is in Platform Golden Path (`architecture/platform-golden-path`); the platform team charter is in Internal Developer Platform Charter (`architecture/internal-developer-platform-charter`).
- Day-2 procedures live elsewhere: minor-version upgrades in Kubernetes Cluster Upgrade Runbook (`ops/kubernetes-cluster-upgrade-runbook`), node OS and image patching in Patch Management Policy (`ops/patch-management-policy`), application releases in Deployment Runbook (`ops/deployment-runbook`).
- Check every version against kubernetes.io/releases on the day you approve the record. Kubernetes ships about three minor releases a year and each is patched for about one year.
- Example organisation: Corvid Commerce, a fictional SaaS company. Delete every *Example* row before approval.
-->

## Document control

<!-- Guidance: Identify the cluster or fleet, the platform owner and the version line. A change of tenancy model, hosting model or CNI is a major version of this record. -->

| Field | Value |
|---|---|
| Document ID | [KCD-001] |
| Cluster or fleet name | [prod-eu fleet] |
| Environments covered | [Production / Staging / Development] |
| Kubernetes minor version at approval | [1.37] |
| Hosting model | [Managed service name / Self-managed with kubeadm / Distribution name] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@platform-lead] |
| Approvers/reviewers | [@head-of-platform], [@security-architect], [@sre-lead], [@finops-lead] |
| Last updated | YYYY-MM-DD |
| Next review | YYYY-MM-DD (at least every minor-version upgrade) |
| Classification | [Internal / Confidential] |
| Related documents | [Landing zone design], [Golden path], [Upgrade runbook], [Backup policy] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Head of platform | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| Security architect | [@name] | [Decision] | YYYY-MM-DD |
| SRE lead | [@name] | [Decision] | YYYY-MM-DD |
| FinOps lead | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved for the prod-eu fleet at Kubernetes 1.37 |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Gateway API replaces the retired Ingress NGINX controller |

## Contents

<!-- Guidance: Keep this list aligned with the headings. -->

1. [Context, scope and requirements](#1-context-scope-and-requirements)
2. [Tenancy model](#2-tenancy-model)
3. [Hosting model](#3-hosting-model)
4. [Version and upgrade cadence](#4-version-and-upgrade-cadence)
5. [Control plane and availability](#5-control-plane-and-availability)
6. [Node pools and compute](#6-node-pools-and-compute)
7. [Networking: CNI and NetworkPolicy](#7-networking-cni-and-networkpolicy)
8. [Ingress and Gateway API](#8-ingress-and-gateway-api)
9. [Identity, authentication and RBAC](#9-identity-authentication-and-rbac)
10. [Workload security and admission policy](#10-workload-security-and-admission-policy)
11. [Secrets](#11-secrets)
12. [Storage](#12-storage)
13. [Observability and audit logging](#13-observability-and-audit-logging)
14. [Backup and recovery](#14-backup-and-recovery)
15. [Cost and capacity](#15-cost-and-capacity)
16. [Hardening baseline](#16-hardening-baseline)
17. [Decisions, risks and open issues](#17-decisions-risks-and-open-issues)
18. [Glossary and references](#18-glossary-and-references)

## 1. Context, scope and requirements

<!-- Guidance: State why this cluster exists, what runs on it and the non-functional requirements that drive every later choice. Pitfall: designing a generic cluster with no stated workload, so every decision becomes a matter of taste. -->

### 1.1 Purpose

State the business purpose of the cluster or fleet in two or three sentences: which products run on it, who operates it and what it replaces. *Example:* Corvid Commerce runs its storefront, checkout and catalogue services on the prod-eu fleet, operated by the platform team as a shared runtime for twelve product teams, replacing a set of virtual machine scale sets.

### 1.2 In scope and out of scope

| Item | In scope | Notes |
|---|---|---|
| Cluster control plane, node pools, cluster add-ons | Yes | This record |
| Network landing zone, VPC or VNet, DNS zones | No | Landing Zone Design (`architecture/landing-zone-design`) |
| Application deployment pipeline | No | Platform Golden Path (`architecture/platform-golden-path`) |
| Data stores outside the cluster (managed databases) | No | [Database design record] |
| *Example:* GPU inference pool | Yes | Section 6, pool `np-gpu` |

### 1.3 Requirements

List the requirements with an ID so later sections can trace to them.

| ID | Requirement | Source | Design section |
|---|---|---|---|
| KCD-REQ-001 *(Example)* | Storefront availability 99.95 % monthly | SLO document | 5, 6 |
| KCD-REQ-002 *(Example)* | Survive loss of one availability zone without manual action | Resilience tier 1 | 5, 6, 12 |
| KCD-REQ-003 *(Example)* | Card data never processed in this cluster | PCI DSS scoping decision | 2, 7 |
| KCD-REQ-004 *(Example)* | Restore a deleted namespace within 4 hours, data loss at most 1 hour | Backup policy | 14 |
| KCD-REQ-005 *(Example)* | No workload runs as root or with host access | Security standard | 10 |
| KCD-REQ-006 *(Example)* | Cluster cost allocated per team monthly | FinOps standard | 15 |

### 1.4 Cluster inventory

| Cluster | Environment | Region and zones | Minor version | Owner |
|---|---|---|---|---|
| prod-eu-1 *(Example)* | Production | eu-west, 3 zones | 1.37 | Platform team |
| prod-eu-2 *(Example)* | Production | eu-central, 3 zones | 1.37 | Platform team |
| stg-eu-1 *(Example)* | Staging | eu-west, 3 zones | 1.37 | Platform team |
| dev-eu-1 *(Example)* | Development | eu-west, 1 zone | 1.37 | Platform team |

## 2. Tenancy model

<!-- Guidance: Decide how teams and workloads share clusters before anything else; RBAC, network policy, quotas and cost allocation all follow from it. Pitfall: treating namespaces as a hard security boundary for mutually untrusted tenants. -->

Kubernetes offers two broad models, with variants between them:

- **Namespace per tenant on a shared cluster (soft multi-tenancy).** Tenants are teams of the same organisation who trust each other to a degree. Isolation comes from namespaces, RBAC, NetworkPolicy, ResourceQuota, LimitRange and Pod Security Admission. Cost is lower and operations are simpler, but the control plane, the node kernel and cluster-scoped resources are shared.
- **Cluster per tenant (hard isolation).** Each tenant or trust zone gets its own cluster. Use it for untrusted code, regulated data, customer-dedicated deployments or noisy workloads that cannot share a control plane. Cost and fleet management effort rise with the number of clusters.
- **Variants.** Dedicated node pools per tenant on a shared control plane (with taints and node affinity), sandboxed runtimes for untrusted code, or virtual control planes per tenant.

For SaaS tenancy of end customers, align this section with Multi-Tenancy Design (`architecture/multi-tenancy-design`), which owns the customer isolation model.

| Decision criterion | Namespace per team | Cluster per trust zone | Weight |
|---|---|---|---|
| Trust between tenants | Same organisation, shared standards | Different organisations or regulated scope | High |
| Blast radius of a control plane fault | Whole fleet tenant set | One tenant | High |
| Cost of idle capacity | Low (bin packing) | Higher | Medium |
| Operational effort (upgrades, add-ons) | One cluster per environment | One per tenant per environment | Medium |
| Cluster-scoped resources needed by tenants (CRDs, webhooks) | Hard to share safely | Free per tenant | Medium |

**Decision.** *Example:* Corvid Commerce uses one shared cluster per environment and region, with one namespace set per product team. Payment tokenisation stays outside Kubernetes (KCD-REQ-003). A separate cluster is created for any workload that must run third party code.

### 2.1 Namespace contract

Every tenant namespace is created by the platform, never by hand, and carries:

| Object | Purpose | Example value |
|---|---|---|
| Labels `team`, `cost-centre`, `data-class` | Ownership, cost and policy selection | `team=checkout` *(Example)* |
| Pod Security Admission labels | Enforce the workload security level | `enforce=restricted` |
| ResourceQuota | Cap CPU, memory, object counts | 40 CPU, 160 GiB requests *(Example)* |
| LimitRange | Default requests and limits for containers | 100m CPU, 128 MiB *(Example)* |
| Default deny NetworkPolicy | Block traffic not explicitly allowed | Section 7 |
| RoleBindings to team groups | Least privilege access | Section 9 |

## 3. Hosting model

<!-- Guidance: Choose between a provider-managed control plane and a self-managed one with explicit criteria. Pitfall: choosing self-managed for flexibility and then lacking the people to patch etcd and the API server on time. -->

| Criterion | Managed control plane | Self-managed (kubeadm or distribution) |
|---|---|---|
| Control plane patching and etcd operations | Provider | Platform team |
| Control plane SLA | Provider SLA, check tier | Own design |
| Version availability | Provider lags upstream by weeks to months | Upstream on release day |
| Control over API server flags and admission plugins | Limited | Full |
| On-premises or air-gapped sites | Rarely possible | Possible |
| Staff needed | Lower | Higher, with etcd skills |

Record the provider or distribution, its support policy for Kubernetes minor versions and any extended support option with its price. Managed services often support a minor version for longer than the upstream project through paid extended support, which is a cost decision, not a substitute for upgrading.

| Field | Value |
|---|---|
| Hosting model chosen | [Managed: provider and service / Self-managed: tool and version] |
| Provider version support window | [Months per minor version, standard and extended] |
| Control plane SLA | [Percentage, tier] |
| Reason | [Two or three sentences linked to KCD-REQ IDs] |

*Example:* Corvid Commerce chooses a managed control plane with the provider's zone-redundant tier because the platform team has four engineers and no etcd on-call capacity.

## 4. Version and upgrade cadence

<!-- Guidance: Commit to a version policy that keeps every cluster on a supported minor version. Pitfall: upgrading only when the provider forces it, which turns every upgrade into an emergency. -->

The Kubernetes project maintains release branches for the three most recent minor releases and gives each minor release about one year of patch support (for 1.19 and newer). Upgrades must go one minor version at a time; skipping minor versions is not supported. The version skew policy limits how far kubelets and other components may lag the API server. The step-by-step procedure is in the upgrade runbook.

Release status checked on kubernetes.io/releases on 2026-10-24:

| Minor version | Latest patch on 2026-10-24 | End of life (upstream) |
|---|---|---|
| 1.37 | 1.37.1 | 2027-10-28 |
| 1.36 | 1.36.5 | 2027-06-28 |
| 1.35 | 1.35.9 | 2027-02-28 |
| 1.34 | 1.34.12 | 2026-10-27 |

1.34 receives its final patches until its end of life on 2026-10-27.

Check the current table at kubernetes.io/releases before approval; 1.38 is the next minor release.

Policy for this cluster or fleet:

1. Run a minor version that is supported both upstream and by the provider, and never one within [90] days of its end of life.
2. Apply patch releases within [30] days of release, or, when the patch fixes a security issue, within the remediation deadline set by Vulnerability Management Policy (`security/vulnerability-management-policy`), delivered through the cadences of Patch Management Policy (`ops/patch-management-policy`).
3. Adopt a new minor version in development within [30] days of the provider making it available, in staging within [60] days and in production within [120] days. *Example values.*
4. Keep all clusters of one fleet on the same minor version, except during a rollout window of at most [30] days.
5. Upgrade node images at least [weekly] through the node pool rotation in section 6.

| Planned upgrade | Target environment | Window | Status |
|---|---|---|---|
| 1.36 to 1.37 *(Example)* | Development | YYYY-MM-DD | Done |
| 1.36 to 1.37 *(Example)* | Production | YYYY-MM-DD | Planned, see upgrade runbook |
| 1.37 to 1.38 *(Example)* | Development | After provider availability | Not started |

## 5. Control plane and availability

<!-- Guidance: Describe how the API server and etcd survive failures, and what the cluster does when the control plane is unavailable. Pitfall: assuming workloads stop when the control plane is down; running pods continue, but scaling, scheduling and deployments stop. -->

For a managed service, record the availability tier and its zone layout. For a self-managed cluster, record:

- API server instances: at least three, spread across zones, behind a load balancer with health checks.
- etcd: an odd number of members (three or five) across failure domains, on dedicated fast storage, stacked with the control plane or as an external cluster, with peer and client TLS.
- Controller manager and scheduler: run with leader election on each control plane node.
- Access to the API server endpoint: private endpoint preferred, with any public endpoint restricted to approved source ranges.

| Failure | Expected behaviour | Recovery | Requirement |
|---|---|---|---|
| One control plane zone lost *(Example)* | API remains available on the other zones | Automatic | KCD-REQ-002 |
| etcd quorum lost *(Example)* | API writes fail and the API server reports unhealthy; running pods continue | Restore from snapshot, section 14 | KCD-REQ-004 |
| API server overload *(Example)* | API Priority and Fairness sheds low priority traffic | Tune flow schemas, scale | KCD-REQ-001 |
| Provider control plane incident *(Example)* | Deployments blocked | Follow provider status, freeze changes | KCD-REQ-001 |

Configure API Priority and Fairness so that controllers and system traffic keep priority over bulk clients such as CI jobs or badly written operators.

## 6. Node pools and compute

<!-- Guidance: Define each node pool with its purpose, machine type, scaling bounds, taints and update method. Pitfall: one large general pool where system add-ons compete with tenant workloads. -->

| Pool | Purpose | Machine type | Min to max nodes | Zones | Taints | OS image |
|---|---|---|---|---|---|---|
| np-system *(Example)* | Cluster add-ons (DNS, CNI agents, ingress, monitoring) | 4 vCPU, 16 GiB | 3 to 6 | 3 | `CriticalAddonsOnly` | Provider hardened image |
| np-general *(Example)* | Stateless tenant workloads | 8 vCPU, 32 GiB | 6 to 60 | 3 | None | Provider hardened image |
| np-memory *(Example)* | Search and cache workloads | 8 vCPU, 64 GiB | 3 to 12 | 3 | `workload=memory:NoSchedule` | Provider hardened image |
| np-gpu *(Example)* | Recommendation inference | 1 GPU, 8 vCPU | 0 to 4 | 2 | `nvidia.com/gpu:NoSchedule` | GPU image |
| np-spot *(Example)* | Batch and CI runners | Mixed | 0 to 40 | 3 | `capacity=spot:NoSchedule` | Provider hardened image |

Rules for node pools:

- Spread every production pool across at least three zones, and use topology spread constraints in workloads so replicas follow.
- Use node autoscaling with bounds that the capacity plan in section 15 justifies. Set expanders and scale-down delays deliberately.
- Replace nodes instead of patching them in place: new node image, surge upgrade, drain, delete. The patch cadence comes from Patch Management Policy (`ops/patch-management-policy`).
- Reserve capacity for system daemons through kubelet system and kube reserved settings or the provider equivalent.
- Require PodDisruptionBudgets for every production workload with more than one replica, so drains during upgrades do not breach the SLO.
- Disable SSH to nodes by default; use a break-glass procedure with audit when node access is needed.

## 7. Networking: CNI and NetworkPolicy

<!-- Guidance: Record the pod and service address plan, the CNI plugin and how NetworkPolicy is enforced. Pitfall: a CNI that does not enforce NetworkPolicy, so policies are accepted by the API but have no effect. -->

### 7.1 Address plan

| Range | CIDR | Size | Notes |
|---|---|---|---|
| Node subnet *(Example)* | 10.40.0.0/20 | 4,096 addresses | Per cluster, from the landing zone plan |
| Pod range *(Example)* | 10.64.0.0/14 | 262,144 addresses | Overlay or routable, per CNI choice |
| Service range *(Example)* | 10.96.0.0/16 | 65,536 addresses | Internal only |

Make sure pod and service ranges do not overlap with any network the cluster must reach, including peered networks and on-premises ranges. Plan for dual-stack if IPv6 is a requirement.

### 7.2 CNI choice

| Criterion | [Option A] | [Option B] |
|---|---|---|
| Enforces NetworkPolicy | [Yes / No] | [Yes / No] |
| Supports extended policy (layer 7, DNS-based, cluster-wide) | [Detail] | [Detail] |
| Pod IPs routable in the VPC | [Yes / No] | [Yes / No] |
| Encryption between nodes | [Detail] | [Detail] |
| Observability of flows | [Detail] | [Detail] |
| Supported by the hosting provider | [Yes / No] | [Yes / No] |

### 7.3 NetworkPolicy baseline

1. Every tenant namespace starts with a default deny for ingress and egress.
2. Allow DNS egress to the cluster DNS service explicitly.
3. Allow ingress only from the gateway namespace or from named client namespaces, using namespace and pod selectors.
4. Restrict egress to the internet through an egress gateway or proxy with an allow list for regulated namespaces.
5. Block access to the cloud instance metadata endpoint from pods that do not need it; use workload identity instead (section 9).

```yaml
# Example: default deny for a tenant namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: checkout
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
```

```mermaid
flowchart LR
  internet["Internet clients"] --> lb["Cloud load balancer"]
  lb --> gw["Gateway namespace: Gateway API implementation"]
  gw --> checkout["checkout namespace"]
  gw --> catalog["catalog namespace"]
  checkout --> dns["Cluster DNS"]
  checkout --> egress["Egress gateway"]
  egress --> psp["Payment provider API"]
  catalog --> db["Managed database outside cluster"]
```

## 8. Ingress and Gateway API

<!-- Guidance: Choose how north-south traffic reaches workloads and who owns each layer. Pitfall: building new work on a controller that has been retired upstream. -->

The Ingress API remains available but is feature-frozen; the Kubernetes project points new designs at Gateway API, which separates infrastructure roles (GatewayClass for the provider, Gateway for the platform team, HTTPRoute and other routes for application teams). The community Ingress NGINX controller was retired upstream in March 2026 after a best-effort maintenance period announced in November 2025, so plan a migration if any cluster still runs it.

| Layer | Resource | Owner | Notes |
|---|---|---|---|
| Implementation | GatewayClass | Platform team | [Implementation name and version] |
| Shared entry point | Gateway per environment | Platform team | TLS termination, listeners, allowed route namespaces |
| Application routing | HTTPRoute, GRPCRoute | Product teams | Created through the golden path |
| Certificates | [Certificate controller or provider service] | Platform team | Automated issuance and renewal |
| Web application firewall | [Service] | Security team | In front of the load balancer |

Record the implementation chosen and check the current Gateway API release and its conformance report for that implementation at gateway-api.sigs.k8s.io.

## 9. Identity, authentication and RBAC

<!-- Guidance: Describe how humans, pipelines and workloads authenticate and what they may do. Pitfall: shared admin kubeconfigs and long-lived static tokens that survive staff changes. -->

### 9.1 Human and pipeline access

- Authenticate humans through the corporate identity provider using OIDC, with groups mapped to Kubernetes groups. No static client certificates or tokens for people.
- Pipelines deploy through a federated workload identity of the CI system, scoped to their namespaces, never through a personal account.
- Keep cluster-admin for a break-glass group with just-in-time elevation and alerting on use.
- Review role bindings at least quarterly through Access Review (`ops/access-review`).

| Role | Kubernetes binding | Scope | Who |
|---|---|---|---|
| platform-admin *(Example)* | cluster-admin (break-glass only) | Cluster | On-call platform engineer with approval |
| platform-operator *(Example)* | Custom ClusterRole, no secrets read | Cluster | Platform team |
| team-deployer *(Example)* | Role: manage workloads, read logs | Team namespaces | Team CI identity |
| team-developer *(Example)* | Role: read workloads, exec in non-prod only | Team namespaces | Team members |
| auditor *(Example)* | ClusterRole view without secrets | Cluster | Security and audit |

### 9.2 Workload identity

- Give each workload its own ServiceAccount; disable automounting of the token where the pod does not call the API server.
- Use bound, short-lived projected service account tokens (the default in current releases), never legacy long-lived token Secrets.
- Map ServiceAccounts to cloud identities through the provider's workload identity feature instead of static cloud keys in Secrets.
- Never grant `escalate`, `bind` or `impersonate` verbs, or wildcards on secrets, to tenant roles.

## 10. Workload security and admission policy

<!-- Guidance: Set the Pod Security Standards level per namespace and the policy rules enforced at admission. Pitfall: running audit mode forever and never moving to enforce. -->

### 10.1 Pod Security Admission

The built-in Pod Security Admission controller enforces the three Pod Security Standards levels (Privileged, Baseline, Restricted) per namespace through labels, in three modes: enforce, audit and warn.

| Namespace class | enforce | audit | warn | Notes |
|---|---|---|---|---|
| Tenant namespaces | restricted | restricted | restricted | Default for all product teams |
| Platform add-on namespaces | baseline | restricted | restricted | Exceptions recorded per add-on |
| Node agents (CNI, storage, monitoring) | privileged | baseline | baseline | Only named system namespaces |

Pin the policy version label (for example `pod-security.kubernetes.io/enforce-version: v1.37`) to the cluster minor version after testing, so that new checks in a later release do not break workloads without warning.

### 10.2 Admission policy

Use ValidatingAdmissionPolicy (CEL rules, built in) or a policy engine for rules that Pod Security Standards do not cover:

| Policy ID | Rule | Mode | Exception path |
|---|---|---|---|
| KCD-ADM-001 *(Example)* | Images only from the approved registry | Deny | None |
| KCD-ADM-002 *(Example)* | Images referenced by digest in production | Deny | Platform approval |
| KCD-ADM-003 *(Example)* | Image signature and provenance verified | Deny in prod, audit elsewhere | Security approval |
| KCD-ADM-004 *(Example)* | CPU and memory requests set | Deny | None |
| KCD-ADM-005 *(Example)* | Required labels `team`, `cost-centre` present | Deny | None |
| KCD-ADM-006 *(Example)* | No Service of type LoadBalancer outside the gateway namespace | Deny | Platform approval |
| KCD-ADM-007 *(Example)* | Production Deployments with 2 or more replicas have a PodDisruptionBudget | Warn | Not applicable |

Set webhook failure policies deliberately: a fail-closed webhook that is down blocks all deployments, a fail-open one silently skips checks. Exclude system namespaces from tenant webhooks to avoid deadlocks.

## 11. Secrets

<!-- Guidance: Describe where secrets live, how they reach pods and how they are protected at rest. Pitfall: base64-encoded Secrets in Git, which is encoding, not encryption. -->

- Encrypt Secret objects at rest in etcd using an envelope encryption provider backed by a key management service (KMS v2 provider or the managed service equivalent).
- Keep the source of truth for secrets in an approved external store and sync or mount them into pods through a secrets operator or CSI driver. The organisation rules for stores, rotation and leak response are in Secrets Management Standard (`security/secrets-management-standard`).
- Restrict `get`, `list` and `watch` on Secrets to the workloads and operators that need them; note that `list` returns secret values.
- Prefer workload identity to any static cloud credential.

| Secret class | Store | Delivery to pod | Rotation |
|---|---|---|---|
| Database credentials *(Example)* | External secret store, dynamic | Secrets operator | Per lease, 24 hours |
| TLS certificates *(Example)* | Certificate controller | Secret mounted by gateway | Automatic before expiry |
| Third party API keys *(Example)* | External secret store | CSI driver | 90 days or on leak |

## 12. Storage

<!-- Guidance: List the storage classes, what they are for and how they behave on zone failure. Pitfall: zonal volumes for a stateful set that must survive a zone loss. -->

| StorageClass | Backing | Access mode | Zone behaviour | Reclaim policy | Use |
|---|---|---|---|---|---|
| standard-rwo *(Example)* | Provider block, SSD | ReadWriteOnce | Zonal, volume binding WaitForFirstConsumer | Delete | Caches, scratch |
| retained-rwo *(Example)* | Provider block, SSD | ReadWriteOnce | Zonal | Retain | Stateful services with backups |
| shared-rwx *(Example)* | Provider file service | ReadWriteMany | Regional | Retain | Shared media uploads |

State the stance on databases in the cluster. *Example:* Corvid Commerce runs primary databases as managed services outside the cluster and allows in-cluster state only for caches and search replicas that can be rebuilt. Enable volume snapshots through the CSI snapshot controller where backups rely on them.

## 13. Observability and audit logging

<!-- Guidance: Cover metrics, logs, traces and the Kubernetes audit log, and who receives them. Pitfall: no API server audit log, so a compromised credential leaves no trace. -->

Operational telemetry (metrics, logs, traces, dashboards, SLO alerts) follows Observability Specification (`ops/observability-spec`); this section records only the cluster-specific choices. Retention and protection of security-relevant logs, including the Kubernetes audit log, follow Security Logging Standard (`security/security-logging-standard`); the retention values below are *Example* values to align with it:

| Signal | Source | Destination | Retention |
|---|---|---|---|
| Cluster and node metrics *(Example)* | Metrics agent, kube-state-metrics | Metrics backend | 13 months downsampled |
| Container logs *(Example)* | Node log agent | Log platform | 30 days hot |
| Traces *(Example)* | OpenTelemetry Collector | Tracing backend | 14 days |
| Kubernetes audit log *(Example)* | API server audit policy | SIEM | 12 months |
| Events *(Example)* | Event exporter | Log platform | 30 days |

Write an audit policy that records metadata for all requests, request bodies for changes to RBAC, admission configuration and workloads, and never the bodies of Secret or token requests. Alert on use of the break-glass group, on new ClusterRoleBindings and on exec into production pods.

## 14. Backup and recovery

<!-- Guidance: Separate control plane state (etcd) from workload state (manifests and volumes) and prove both with restore tests. Pitfall: backing up etcd but not persistent volumes, or the reverse. -->

The organisation rules for backup tiers, copies, immutability and restore testing are in Backup Policy (`ops/backup-policy`). This section applies them to the cluster.

| Item | Method | Frequency | Retention | Restore test |
|---|---|---|---|---|
| etcd (self-managed) | `etcdctl snapshot save`, encrypted, copied off cluster | Every [1] hour | [7] days | Quarterly into an isolated cluster |
| etcd (managed) | Provider responsibility; confirm in contract | Provider | Provider | Not possible; rely on workload backup |
| Cluster configuration | Git repository (GitOps source of truth) | Every change | Git history | Rebuild staging from Git twice a year |
| Namespaced objects | Backup tool export per namespace | Every [1] hour | [30] days | Monthly namespace restore *(Example)* |
| Persistent volumes | CSI snapshots plus copy to another region | Every [1] hour | [30] days | Monthly |

Recovery scenarios to document and test:

1. Restore a deleted namespace with its volumes (KCD-REQ-004).
2. Rebuild a cluster from Git and backups after loss of a region.
3. Restore etcd after quorum loss (self-managed only), following the documented etcd restore procedure for the running version.

The restore steps for one system follow Backup and Restore Procedure (`ops/backup-restore-procedure`); evidence of a disaster recovery test goes in DR Test Report (`ops/dr-test-report`).

## 15. Cost and capacity

<!-- Guidance: Show how the cluster is sized, how cost is allocated to teams and which levers keep it efficient. Pitfall: requests set far above real use, so the cluster scales out for capacity that is never used. -->

Sizing and forecasts follow Capacity Plan (`ops/capacity-plan`); cost allocation follows Cost Allocation and Tagging Standard (`ops/cost-allocation-tagging-standard`).

| Measure | Formula | Target | Current |
|---|---|---|---|
| CPU request utilisation | sum(CPU used) / sum(CPU requested) | above [60] % | 48 % *(Example)* |
| Node allocation | sum(CPU requested) / sum(node allocatable CPU) | [70] to [85] % | 74 % *(Example)* |
| Idle cost share | idle node cost / total cluster cost | below [20] % | 26 % *(Example)* |
| Cost allocated to a team | share of requests (or max of requests and use) per namespace times node cost, plus shared add-on cost split by the same ratio | 100 % allocated | 92 % *(Example)* |

Levers: right-size requests from observed use, scale pools to zero when idle, use spot capacity for interruptible batch work, use committed-use discounts for the stable baseline, and review the top ten namespaces by cost monthly in FinOps Cost Review (`ops/finops-cost-review`).

## 16. Hardening baseline

<!-- Guidance: Name the baselines this cluster is assessed against and record how compliance is measured. Pitfall: copying a benchmark into the design; reference it and keep the evidence in the scan results. -->

The cluster is assessed against:

- **NSA and CISA Kubernetes Hardening Guidance, version 1.2 (August 2022).** Its main themes map to this record: pod security (section 10), network separation and hardening (sections 7 and 8), authentication and authorisation (section 9), audit logging and threat detection (section 13), and upgrading and application security practices (sections 4 and 6).
- **CIS Kubernetes Benchmark**, or the provider-specific CIS Benchmark for the managed service, in the version matching the cluster's minor version. Obtain it from the Center for Internet Security; this template does not reproduce or paraphrase any of its recommendations.
- **Organisation security standards**, including the cloud posture review in Cloud Security Posture Review (`security/cloud-security-posture-review`).

| Control area | Evidence | Frequency | Owner |
|---|---|---|---|
| Benchmark scan of nodes and control plane *(Example)* | Scanner report, deviations recorded | Weekly | Platform team |
| Pod Security Admission violations *(Example)* | Audit annotations report | Weekly | Platform team |
| Image vulnerability scan *(Example)* | Registry scan report, routed to vulnerability management | Daily | Product teams |
| RBAC review *(Example)* | Access review record | Quarterly | Security team |
| Restore test *(Example)* | DR test report | Quarterly | SRE |

Record every accepted deviation in section 17 with an owner and an expiry date. Image and node vulnerabilities are prioritised and given deadlines under Vulnerability Management Policy (`security/vulnerability-management-policy`).

## 17. Decisions, risks and open issues

<!-- Guidance: Log the main decisions with alternatives considered and the risks you accept. Pitfall: decisions that live only in chat threads, so the next team reopens them. -->

| ID | Type | Description | Owner | Due or expiry |
|---|---|---|---|---|
| KCD-DEC-001 *(Example)* | Decision | Managed control plane, zone-redundant tier (section 3) | [@platform-lead] | Review at next major version |
| KCD-DEC-002 *(Example)* | Decision | Gateway API with [implementation] replaces Ingress NGINX | [@platform-lead] | Not applicable |
| KCD-RSK-001 *(Example)* | Risk | Shared cluster lets a kernel exploit cross team namespaces | [@security-architect] | Mitigated by restricted PSA and node patching; review YYYY-MM-DD |
| KCD-RSK-002 *(Example)* | Risk | Provider ships new minor versions late, shrinking the upgrade window | [@platform-lead] | Track monthly |
| KCD-DEV-001 *(Example)* | Deviation | Monitoring agent needs host access in a privileged namespace | [@sre-lead] | YYYY-MM-DD |
| KCD-ISS-001 *(Example)* | Open issue | Choose the policy engine or stay with built-in CEL policies | [@platform-lead] | YYYY-MM-DD |

Design decisions with lasting effect should also be written as Architecture Decision Records (`architecture/adr-full`).

## 18. Glossary and references

<!-- Guidance: Define the terms readers outside the platform team need, and list sources with the version you checked. -->

### 18.1 Glossary

| Term | Meaning |
|---|---|
| Control plane | API server, etcd, scheduler and controllers that manage the cluster |
| etcd | Consistent key-value store holding all cluster state |
| CNI | Container Network Interface plugin that provides pod networking |
| NetworkPolicy | Kubernetes API object that allows or denies pod traffic, enforced by the CNI |
| Gateway API | Kubernetes API family for role-oriented traffic routing |
| Pod Security Standards | Three policy levels (Privileged, Baseline, Restricted) for pod settings |
| Pod Security Admission | Built-in admission controller that applies the standards per namespace |
| ValidatingAdmissionPolicy | Built-in admission policy written in CEL |
| PodDisruptionBudget | Limit on voluntary disruptions to a set of pods |
| Version skew policy | Rules for how far component versions may differ from the API server |
| Workload identity | Mapping a ServiceAccount to a cloud identity without static keys |

### 18.2 References

| Reference | Version or date checked | Use |
|---|---|---|
| Kubernetes Releases, kubernetes.io/releases | Checked 2026-10-24: 1.37.1 latest, 1.34 end of life 2026-10-27 | Version table and support period |
| Kubernetes Version Skew Policy, kubernetes.io/releases/version-skew-policy | Checked 2026-10-24 | Skew limits and upgrade order |
| Kubernetes documentation: Pod Security Standards, Pod Security Admission, Network Policies, Gateway API, Encrypting Confidential Data at Rest, Auditing, Multi-tenancy | Checked 2026-10-24 | Concepts summarised in original words |
| Ingress NGINX Retirement: What You Need to Know, kubernetes.io blog | 2025-11-11 | Retirement timeline |
| NSA and CISA, Kubernetes Hardening Guidance | Version 1.2, August 2022 | Hardening themes in section 16 |
| CIS Kubernetes Benchmark, Center for Internet Security | Version matching the cluster | Named only |
| Gateway API project, gateway-api.sigs.k8s.io | Check current release | Implementation conformance |

## Standard and licence note

Based on the Kubernetes documentation published at kubernetes.io by The Kubernetes Authors under the Creative Commons Attribution 4.0 International licence (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/); concepts are summarised in original words and no documentation text is copied. Also based on the NSA and CISA Kubernetes Hardening Guidance version 1.2 (August 2022), a US Government publication. The CIS Kubernetes Benchmark is named as an external baseline only; none of its content is reproduced or paraphrased, and it must be obtained from the Center for Internet Security under its own terms. Kubernetes is a registered trademark of The Linux Foundation. Other product and service names are trademarks of their owners; this template is not affiliated with or endorsed by them. This template is original text written for this collection. Check the current Kubernetes release table and the current editions of the referenced guidance before use. Template licence: MIT.