Kubernetes Cluster Design Template (Tenancy, Security, Operations)

Kubernetes Cluster Design: [Cluster or fleet name]

Design record for one Kubernetes cluster or a fleet of clusters built to the same pattern: tenancy model, managed or self-managed control plane, version and upgrade cadence, control plane availability, node pools, networking and NetworkPolicy, ingress and Gateway API, identity and RBAC, Pod Security Admission, admission policy, secrets, storage, observability, backup of etcd and workloads, cost and capacity, and the hardening baseline. Based on the Kubernetes documentation (kubernetes.io, CC BY 4.0, release 1.37 current as of October 2026) and the NSA and CISA Kubernetes Hardening Guidance version 1.2 (August 2022). The CIS Kubernetes Benchmark is named as an external baseline only. Replace every [bracketed] value and every row marked Example.

Document control

Approvals

Revision history

Contents

1. Context, scope and requirements
2. Tenancy model
3. Hosting model
4. Version and upgrade cadence
5. Control plane and availability
6. Node pools and compute
7. Networking: CNI and NetworkPolicy
8. Ingress and Gateway API
9. Identity, authentication and RBAC
10. Workload security and admission policy
11. Secrets
12. Storage
13. Observability and audit logging
14. Backup and recovery
15. Cost and capacity
16. Hardening baseline
17. Decisions, risks and open issues
18. Glossary and references

1. Context, scope and requirements

1.1 Purpose

State the business purpose of the cluster or fleet in two or three sentences: which products run on it, who operates it and what it replaces. Example: Corvid Commerce runs its storefront, checkout and catalogue services on the prod-eu fleet, operated by the platform team as a shared runtime for twelve product teams, replacing a set of virtual machine scale sets.

1.2 In scope and out of scope

1.3 Requirements

List the requirements with an ID so later sections can trace to them.

1.4 Cluster inventory

2. Tenancy model

Kubernetes offers two broad models, with variants between them:

• Namespace per tenant on a shared cluster (soft multi-tenancy). Tenants are teams of the same organisation who trust each other to a degree. Isolation comes from namespaces, RBAC, NetworkPolicy, ResourceQuota, LimitRange and Pod Security Admission. Cost is lower and operations are simpler, but the control plane, the node kernel and cluster-scoped resources are shared.
• Cluster per tenant (hard isolation). Each tenant or trust zone gets its own cluster. Use it for untrusted code, regulated data, customer-dedicated deployments or noisy workloads that cannot share a control plane. Cost and fleet management effort rise with the number of clusters.
• Variants. Dedicated node pools per tenant on a shared control plane (with taints and node affinity), sandboxed runtimes for untrusted code, or virtual control planes per tenant.

For SaaS tenancy of end customers, align this section with Multi-Tenancy Design (architecture/multi-tenancy-design), which owns the customer isolation model.

Decision. Example: Corvid Commerce uses one shared cluster per environment and region, with one namespace set per product team. Payment tokenisation stays outside Kubernetes (KCD-REQ-003). A separate cluster is created for any workload that must run third party code.

2.1 Namespace contract

Every tenant namespace is created by the platform, never by hand, and carries:

3. Hosting model

Record the provider or distribution, its support policy for Kubernetes minor versions and any extended support option with its price. Managed services often support a minor version for longer than the upstream project through paid extended support, which is a cost decision, not a substitute for upgrading.

Example: Corvid Commerce chooses a managed control plane with the provider's zone-redundant tier because the platform team has four engineers and no etcd on-call capacity.

4. Version and upgrade cadence

The Kubernetes project maintains release branches for the three most recent minor releases and gives each minor release about one year of patch support (for 1.19 and newer). Upgrades must go one minor version at a time; skipping minor versions is not supported. The version skew policy limits how far kubelets and other components may lag the API server. The step-by-step procedure is in the upgrade runbook.

Release status checked on kubernetes.io/releases on 2026-10-24:

1.34 receives its final patches until its end of life on 2026-10-27.

Check the current table at kubernetes.io/releases before approval; 1.38 is the next minor release.

Policy for this cluster or fleet:

1. Run a minor version that is supported both upstream and by the provider, and never one within [90] days of its end of life.
2. Apply patch releases within [30] days of release, or, when the patch fixes a security issue, within the remediation deadline set by Vulnerability Management Policy (security/vulnerability-management-policy), delivered through the cadences of Patch Management Policy (ops/patch-management-policy).
3. Adopt a new minor version in development within [30] days of the provider making it available, in staging within [60] days and in production within [120] days. Example values.
4. Keep all clusters of one fleet on the same minor version, except during a rollout window of at most [30] days.
5. Upgrade node images at least [weekly] through the node pool rotation in section 6.

5. Control plane and availability

For a managed service, record the availability tier and its zone layout. For a self-managed cluster, record:

• API server instances: at least three, spread across zones, behind a load balancer with health checks.
• etcd: an odd number of members (three or five) across failure domains, on dedicated fast storage, stacked with the control plane or as an external cluster, with peer and client TLS.
• Controller manager and scheduler: run with leader election on each control plane node.
• Access to the API server endpoint: private endpoint preferred, with any public endpoint restricted to approved source ranges.

Configure API Priority and Fairness so that controllers and system traffic keep priority over bulk clients such as CI jobs or badly written operators.

6. Node pools and compute

Rules for node pools:

• Spread every production pool across at least three zones, and use topology spread constraints in workloads so replicas follow.
• Use node autoscaling with bounds that the capacity plan in section 15 justifies. Set expanders and scale-down delays deliberately.
• Replace nodes instead of patching them in place: new node image, surge upgrade, drain, delete. The patch cadence comes from Patch Management Policy (ops/patch-management-policy).
• Reserve capacity for system daemons through kubelet system and kube reserved settings or the provider equivalent.
• Require PodDisruptionBudgets for every production workload with more than one replica, so drains during upgrades do not breach the SLO.
• Disable SSH to nodes by default; use a break-glass procedure with audit when node access is needed.

7. Networking: CNI and NetworkPolicy

7.1 Address plan

Make sure pod and service ranges do not overlap with any network the cluster must reach, including peered networks and on-premises ranges. Plan for dual-stack if IPv6 is a requirement.

7.2 CNI choice

7.3 NetworkPolicy baseline

1. Every tenant namespace starts with a default deny for ingress and egress.
2. Allow DNS egress to the cluster DNS service explicitly.
3. Allow ingress only from the gateway namespace or from named client namespaces, using namespace and pod selectors.
4. Restrict egress to the internet through an egress gateway or proxy with an allow list for regulated namespaces.
5. Block access to the cloud instance metadata endpoint from pods that do not need it; use workload identity instead (section 9).

# Example: default deny for a tenant namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: checkout
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

8. Ingress and Gateway API

The Ingress API remains available but is feature-frozen; the Kubernetes project points new designs at Gateway API, which separates infrastructure roles (GatewayClass for the provider, Gateway for the platform team, HTTPRoute and other routes for application teams). The community Ingress NGINX controller was retired upstream in March 2026 after a best-effort maintenance period announced in November 2025, so plan a migration if any cluster still runs it.

Record the implementation chosen and check the current Gateway API release and its conformance report for that implementation at gateway-api.sigs.k8s.io.

9. Identity, authentication and RBAC

9.1 Human and pipeline access

• Authenticate humans through the corporate identity provider using OIDC, with groups mapped to Kubernetes groups. No static client certificates or tokens for people.
• Pipelines deploy through a federated workload identity of the CI system, scoped to their namespaces, never through a personal account.
• Keep cluster-admin for a break-glass group with just-in-time elevation and alerting on use.
• Review role bindings at least quarterly through Access Review (ops/access-review).

9.2 Workload identity

• Give each workload its own ServiceAccount; disable automounting of the token where the pod does not call the API server.
• Use bound, short-lived projected service account tokens (the default in current releases), never legacy long-lived token Secrets.
• Map ServiceAccounts to cloud identities through the provider's workload identity feature instead of static cloud keys in Secrets.
• Never grant escalate, bind or impersonate verbs, or wildcards on secrets, to tenant roles.

10. Workload security and admission policy

10.1 Pod Security Admission

The built-in Pod Security Admission controller enforces the three Pod Security Standards levels (Privileged, Baseline, Restricted) per namespace through labels, in three modes: enforce, audit and warn.

Pin the policy version label (for example pod-security.kubernetes.io/enforce-version: v1.37) to the cluster minor version after testing, so that new checks in a later release do not break workloads without warning.

10.2 Admission policy

Use ValidatingAdmissionPolicy (CEL rules, built in) or a policy engine for rules that Pod Security Standards do not cover:

Set webhook failure policies deliberately: a fail-closed webhook that is down blocks all deployments, a fail-open one silently skips checks. Exclude system namespaces from tenant webhooks to avoid deadlocks.

11. Secrets

• Encrypt Secret objects at rest in etcd using an envelope encryption provider backed by a key management service (KMS v2 provider or the managed service equivalent).
• Keep the source of truth for secrets in an approved external store and sync or mount them into pods through a secrets operator or CSI driver. The organisation rules for stores, rotation and leak response are in Secrets Management Standard (security/secrets-management-standard).
• Restrict get, list and watch on Secrets to the workloads and operators that need them; note that list returns secret values.
• Prefer workload identity to any static cloud credential.

12. Storage

State the stance on databases in the cluster. Example: Corvid Commerce runs primary databases as managed services outside the cluster and allows in-cluster state only for caches and search replicas that can be rebuilt. Enable volume snapshots through the CSI snapshot controller where backups rely on them.

13. Observability and audit logging

Operational telemetry (metrics, logs, traces, dashboards, SLO alerts) follows Observability Specification (ops/observability-spec); this section records only the cluster-specific choices. Retention and protection of security-relevant logs, including the Kubernetes audit log, follow Security Logging Standard (security/security-logging-standard); the retention values below are Example values to align with it:

Write an audit policy that records metadata for all requests, request bodies for changes to RBAC, admission configuration and workloads, and never the bodies of Secret or token requests. Alert on use of the break-glass group, on new ClusterRoleBindings and on exec into production pods.

14. Backup and recovery

The organisation rules for backup tiers, copies, immutability and restore testing are in Backup Policy (ops/backup-policy). This section applies them to the cluster.

Recovery scenarios to document and test:

1. Restore a deleted namespace with its volumes (KCD-REQ-004).
2. Rebuild a cluster from Git and backups after loss of a region.
3. Restore etcd after quorum loss (self-managed only), following the documented etcd restore procedure for the running version.

The restore steps for one system follow Backup and Restore Procedure (ops/backup-restore-procedure); evidence of a disaster recovery test goes in DR Test Report (ops/dr-test-report).

15. Cost and capacity

Sizing and forecasts follow Capacity Plan (ops/capacity-plan); cost allocation follows Cost Allocation and Tagging Standard (ops/cost-allocation-tagging-standard).

Levers: right-size requests from observed use, scale pools to zero when idle, use spot capacity for interruptible batch work, use committed-use discounts for the stable baseline, and review the top ten namespaces by cost monthly in FinOps Cost Review (ops/finops-cost-review).

16. Hardening baseline

The cluster is assessed against:

• NSA and CISA Kubernetes Hardening Guidance, version 1.2 (August 2022). Its main themes map to this record: pod security (section 10), network separation and hardening (sections 7 and 8), authentication and authorisation (section 9), audit logging and threat detection (section 13), and upgrading and application security practices (sections 4 and 6).
• CIS Kubernetes Benchmark, or the provider-specific CIS Benchmark for the managed service, in the version matching the cluster's minor version. Obtain it from the Center for Internet Security; this template does not reproduce or paraphrase any of its recommendations.
• Organisation security standards, including the cloud posture review in Cloud Security Posture Review (security/cloud-security-posture-review).

Record every accepted deviation in section 17 with an owner and an expiry date. Image and node vulnerabilities are prioritised and given deadlines under Vulnerability Management Policy (security/vulnerability-management-policy).

17. Decisions, risks and open issues

Design decisions with lasting effect should also be written as Architecture Decision Records (architecture/adr-full).

18. Glossary and references

18.1 Glossary

18.2 References

Standard and licence note

Based on the Kubernetes documentation published at kubernetes.io by The Kubernetes Authors under the Creative Commons Attribution 4.0 International licence (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/); concepts are summarised in original words and no documentation text is copied. Also based on the NSA and CISA Kubernetes Hardening Guidance version 1.2 (August 2022), a US Government publication. The CIS Kubernetes Benchmark is named as an external baseline only; none of its content is reproduced or paraphrased, and it must be obtained from the Center for Internet Security under its own terms. Kubernetes is a registered trademark of The Linux Foundation. Other product and service names are trademarks of their owners; this template is not affiliated with or endorsed by them. This template is original text written for this collection. Check the current Kubernetes release table and the current editions of the referenced guidance before use. Template licence: MIT.