Kubernetes Cluster Upgrade Runbook Template (Minor Version)

Kubernetes Cluster Upgrade Runbook: [Cluster name] [source version] to [target version]

Runbook for one minor-version upgrade of a Kubernetes cluster: support window and version skew check, deprecated API scan, add-on and CRD compatibility, etcd backup, control plane first and nodes second, surge and drain with PodDisruptionBudget checks, verification, rollback limits and communication. It includes a worked example of an upgrade from 1.36 to 1.37. Based on the Kubernetes release and version skew policy documentation (kubernetes.io, CC BY 4.0), checked on 2026-10-24 when 1.37 was the newest minor release. Replace every [bracketed] value and every row marked Example.

Document control

Approvals

Revision history

Contents

1. Scope and summary
2. Roles and contacts
3. Support window and version skew check
4. Release notes and deprecated API scan
5. Add-on, CRD and operator compatibility
6. Workload disruption readiness
7. Backup and rollback limits
8. Go or no-go checklist
9. Procedure: control plane
10. Procedure: node pools
11. Verification
12. Rollback and recovery
13. Communication plan
14. Worked example: 1.36 to 1.37
15. Outcome and closure
16. Glossary and references

1. Scope and summary

State the scope in a short paragraph: which cluster, which control plane and node pools, which add-ons move with the upgrade and which do not. Name the trigger (cadence, approaching end of life, needed feature, security fix) and the business deadline.

Rules for this runbook:

1. Upgrade one minor version per change. Never skip a minor version.
2. Upgrade staging first and let it run for at least [5] business days before production.
3. Upgrade the control plane first, then node pools. Kubelets must never be newer than the API server.
4. Freeze application deployments to the cluster for the window, except emergency fixes approved by the upgrade lead.
5. Stop and roll back at any failed verification gate; do not improvise in production.

2. Roles and contacts

If the upgrade causes customer impact, declare an incident and hand command to the incident commander; roles then follow Incident Commander Handbook (ops/incident-commander-handbook) and severity follows Incident Severity Matrix (ops/incident-severity-matrix).

3. Support window and version skew check

3.1 Support window

Check the release table at kubernetes.io/releases and the provider's version calendar on the day of planning and again on the day before the window.

3.2 Version skew

The Kubernetes version skew policy, as published on 2026-10-24, sets these limits relative to the API server:

While API servers in a highly available control plane run mixed versions, the allowed range for the other components narrows to the oldest API server. Some providers and tools impose tighter limits than the upstream policy; record them.

Any kubelet or kube-proxy already three minor versions behind the API server must be upgraded before the control plane can move.

4. Release notes and deprecated API scan

4.1 Release notes review

Read the release notes and the "urgent upgrade notes" of the target minor version and of every patch in between, plus the Kubernetes deprecated API migration guide. Record each item that affects this cluster.

4.2 Deprecated API scan

Use at least two sources, because each sees a different part of the picture:

1. Live cluster usage. Query the API server metric apiserver_requested_deprecated_apis and audit events annotated as deprecated over a period that covers monthly jobs (at least [30] days of audit data). These show which clients still call deprecated versions.
2. Stored objects and manifests. Scan Git repositories, Helm releases and rendered manifests with a deprecated API detection tool, and render every chart with the target version's API list.
3. Admission webhooks. Check that webhooks match new resource versions; confirm that no webhook sets matchPolicy: Exact, so that requests for equivalent API versions are still intercepted (Equivalent is the default in admissionregistration.k8s.io/v1).

# Example: list deprecated API calls seen by the API server
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Exit criterion: no client of a version removed in the target release in the last [30] days of audit data, and no stored manifest using one.

5. Add-on, CRD and operator compatibility

For each CRD-backed operator, record the stored versions (status.storedVersions of the CRD) and confirm that the new operator release still serves them or migrates them. Upgrade operators in staging first and run their own upgrade notes.

6. Workload disruption readiness

Draining a node evicts its pods through the eviction API, which honours PodDisruptionBudgets. Check before the window:

1. Every production workload with two or more replicas has a PodDisruptionBudget.
2. No PodDisruptionBudget has ALLOWED DISRUPTIONS at 0 in steady state, unless the owner has agreed a manual step.
3. Single-replica workloads are listed and their owners accept a short outage, or scale to two replicas for the window.
4. Stateful workloads with quorum (databases, queues, search) have a PodDisruptionBudget that keeps quorum and an owner on standby.
5. Workloads use readiness probes so that traffic shifts only to ready pods.
6. Pods with local storage (emptyDir) are known; their data is lost on drain.

# Example: find budgets that currently allow no disruption
kubectl get pdb -A -o wide | awk 'NR==1 || $5 == 0'

7. Backup and rollback limits

Take and verify these backups no more than [2] hours before the window:

Rollback limits to accept before approval:

• Control plane. Downgrading the control plane to the previous minor version is not a supported path upstream, and managed services generally do not offer it. Once the API server has written objects in the new version, the only full rollback is restoring etcd from the snapshot (self-managed) or rebuilding the cluster from Git and backups. Treat the control plane step as the point of no return.
• Nodes. Node pools can be rolled back by creating a new pool at the previous version (allowed, because kubelets may be older than the API server) and moving workloads to it.
• Add-ons. Roll back add-ons through their own release history, provided their CRD versions allow it.
• Data written during the window. Restoring etcd discards cluster changes after the snapshot; workloads' external data is not affected.

8. Go or no-go checklist

Decision: [Go / No-go], recorded by [@upgrade-lead] at [YYYY-MM-DD HH:MM].

9. Procedure: control plane

Record the start and end time of each step in the execution log.

1. Upgrade add-ons that must move first (section 5), one at a time, verifying each. Verify: add-on pods ready, no new errors in their logs.
2. Upgrade the API server.

   • Managed service: start the control plane upgrade to the target patch through the provider interface or infrastructure code. Verify: the provider reports the target version and the cluster as healthy.

   • Self-managed with kubeadm: follow the kubeadm upgrade page on kubernetes.io for the target minor, in this order.

     1. On every node, switch the package repository to the target minor.
     2. On the first control plane node: upgrade the kubeadm package to the target patch, check it with kubeadm version, run sudo kubeadm upgrade plan, then sudo kubeadm upgrade apply v[target]. Upgrade the CNI plugin if its provider requires it (section 5).
     3. On each other control plane node, one at a time: upgrade the kubeadm package, then run sudo kubeadm upgrade node.
     4. Then, on each control plane node in turn: kubectl drain <node> --ignore-daemonsets, upgrade the kubelet and kubectl packages to the target patch, run sudo systemctl daemon-reload and sudo systemctl restart kubelet, then kubectl uncordon <node>.
     5. Self-managed worker nodes that are upgraded in place instead of replaced follow the same pattern in section 10: upgrade kubeadm, run sudo kubeadm upgrade node, drain, upgrade kubelet and kubectl, restart kubelet, uncordon.

     Verify: each API server reports the target version, and kubectl get nodes shows every control plane node Ready at the target kubelet version.

3. Controller manager, scheduler and cloud controller manager. These move with the control plane on managed services and kubeadm. Verify: their leases renew and no crash loops occur.
4. Check API health.

kubectl version
kubectl get --raw='/readyz?verbose'
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

5. Gate 1. Service owners confirm their dashboards show no regression for [15] minutes. If any check fails, stop and go to section 12.

10. Procedure: node pools

Upgrade pools in this order: system pool, then general pools, then special pools (GPU, memory, spot). For each pool:

1. Set surge settings. Example: max surge 1 node per zone, max unavailable 0, so capacity never drops.
2. Start the pool upgrade (managed) or create a replacement pool at the target version (self-managed or blue-green). Kubelets must not be upgraded across minor versions while pods run on the node: drain the node first. For kubeadm nodes upgraded in place, work one node (or one zone batch) at a time: upgrade the kubeadm package, run sudo kubeadm upgrade node, drain (step 3), upgrade the kubelet and kubectl packages to the target patch, run sudo systemctl daemon-reload and sudo systemctl restart kubelet, then kubectl uncordon <node>.
3. Drain each old node.

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m

4. Watch evictions. If a drain waits on a PodDisruptionBudget for more than [15] minutes, page the workload owner; do not force eviction of quorum workloads.
5. Verify each zone batch: new nodes Ready at the target version, pods rescheduled and ready, no pending pods.
6. Delete old nodes or the old pool once it is empty.
7. Gate 2 per pool: service owners confirm no regression for [15] minutes.

11. Verification

12. Rollback and recovery

After any rollback, update the change record, keep the cluster in its current state until a review decides the next step, and record the cause in a postmortem using Postmortem (dev/postmortem).

13. Communication plan

14. Worked example: 1.36 to 1.37

Example: Corvid Commerce upgrades the prod-eu-1 cluster (managed control plane, four node pools) from 1.36.5 to 1.37.1. Kubernetes 1.37.0 was released upstream on 2026-08-26; on 2026-10-24 the upstream table lists 1.37, 1.36, 1.35 and 1.34 as maintained, with 1.34 reaching end of life on 2026-10-27.

Totals for the example: 33 nodes replaced (3 + 24 + 6), window used 4 hours 35 minutes of the 6 hours booked, no customer impact.

15. Outcome and closure

16. Glossary and references

16.1 Glossary

16.2 References

Standard and licence note

Based on the Kubernetes release, version skew policy and task documentation published at kubernetes.io by The Kubernetes Authors under the Creative Commons Attribution 4.0 International licence (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/). The skew limits are summarised in original words and no documentation text is copied. Kubernetes is a registered trademark of The Linux Foundation. Other product and service names are trademarks of their owners; this template is not affiliated with or endorsed by them. This template is original text written for this collection. Check the current release table, skew policy and release notes for the target version before use. Template licence: MIT.