# Kubernetes Cluster Upgrade Runbook: [Cluster name] [source version] to [target version]

> Runbook for one minor-version upgrade of a Kubernetes cluster: support window and version skew check, deprecated API scan, add-on and CRD compatibility, etcd backup, control plane first and nodes second, surge and drain with PodDisruptionBudget checks, verification, rollback limits and communication. It includes a worked example of an upgrade from 1.36 to 1.37. Based on the Kubernetes release and version skew policy documentation (kubernetes.io, CC BY 4.0), checked on 2026-10-24 when 1.37 was the newest minor release. Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Copy this runbook once per cluster and per minor-version step. An upgrade from 1.35 to 1.37 is two runbooks executed in order, because skipping minor versions is not supported.
- The cluster's version policy and target cadence live in Kubernetes Cluster Design (`ops/kubernetes-cluster-design`), section 4. This runbook executes one step of that policy.
- Raise the change through Change Request (`ops/change-request`) and announce customer-visible windows with Maintenance Notice (`ops/maintenance-notice`). Application releases during the window follow Deployment Runbook (`ops/deployment-runbook`) and are normally frozen.
- Node image and OS patching between minor upgrades follows Patch Management Policy (`ops/patch-management-policy`).
- Managed services hide the control plane steps behind one API call or console action; keep the sections anyway and record what the provider does and what you verify.
- Example organisation: Corvid Commerce, a fictional SaaS company. Delete every *Example* row before use.
-->

## Document control

<!-- Guidance: Identify the cluster, the versions and the change record. Each execution of the runbook is a version of this document with its outcome recorded. -->

| Field | Value |
|---|---|
| Document ID | [KUR-001] |
| Cluster | [prod-eu-1] |
| Environment | [Production / Staging / Development] |
| Source version (control plane and nodes) | [1.36.5] |
| Target version | [1.37.1] |
| Hosting model | [Managed service / kubeadm / Distribution] |
| Change request | [CHG-0000] |
| Planned window | [YYYY-MM-DD HH:MM to HH:MM time zone] |
| Upgrade lead | [@upgrade-lead] |
| Version | [0.1] |
| Status | [Draft / Approved / In progress / Completed / Rolled back] |
| Owner | [@platform-lead] |
| Approvers/reviewers | [@sre-lead], [@service-owner], [@change-manager] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Platform lead | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| SRE lead | [@name] | [Decision] | YYYY-MM-DD |
| Change manager | [@name] | [Decision] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved after staging rehearsal |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Executed in production, outcome recorded in section 15 |

## Contents

<!-- Guidance: Keep this list aligned with the headings. -->

1. [Scope and summary](#1-scope-and-summary)
2. [Roles and contacts](#2-roles-and-contacts)
3. [Support window and version skew check](#3-support-window-and-version-skew-check)
4. [Release notes and deprecated API scan](#4-release-notes-and-deprecated-api-scan)
5. [Add-on, CRD and operator compatibility](#5-add-on-crd-and-operator-compatibility)
6. [Workload disruption readiness](#6-workload-disruption-readiness)
7. [Backup and rollback limits](#7-backup-and-rollback-limits)
8. [Go or no-go checklist](#8-go-or-no-go-checklist)
9. [Procedure: control plane](#9-procedure-control-plane)
10. [Procedure: node pools](#10-procedure-node-pools)
11. [Verification](#11-verification)
12. [Rollback and recovery](#12-rollback-and-recovery)
13. [Communication plan](#13-communication-plan)
14. [Worked example: 1.36 to 1.37](#14-worked-example-136-to-137)
15. [Outcome and closure](#15-outcome-and-closure)
16. [Glossary and references](#16-glossary-and-references)

## 1. Scope and summary

<!-- Guidance: Say what is upgraded, in which order and why now. Pitfall: bundling the minor upgrade with unrelated changes (CNI swap, new add-ons), which makes failures hard to attribute. -->

State the scope in a short paragraph: which cluster, which control plane and node pools, which add-ons move with the upgrade and which do not. Name the trigger (cadence, approaching end of life, needed feature, security fix) and the business deadline.

| Item | Upgraded in this change | From | To |
|---|---|---|---|
| Control plane *(Example)* | Yes | 1.36.5 | 1.37.1 |
| Node pool np-system *(Example)* | Yes | 1.36.5 | 1.37.1 |
| Node pool np-general *(Example)* | Yes | 1.36.5 | 1.37.1 |
| Node pool np-memory *(Example)* | Yes | 1.36.5 | 1.37.1 |
| Node pool np-gpu *(Example)* | Deferred, within skew | 1.36.5 | 1.36.5 |
| CNI plugin *(Example)* | Yes, required by its support matrix | [x.y] | [x.z] |
| Cluster DNS, metrics server *(Example)* | Yes, provider managed | Provider | Provider |

Rules for this runbook:

1. Upgrade one minor version per change. Never skip a minor version.
2. Upgrade staging first and let it run for at least [5] business days before production.
3. Upgrade the control plane first, then node pools. Kubelets must never be newer than the API server.
4. Freeze application deployments to the cluster for the window, except emergency fixes approved by the upgrade lead.
5. Stop and roll back at any failed verification gate; do not improvise in production.

## 2. Roles and contacts

<!-- Guidance: Name a single upgrade lead who makes the go, pause and rollback calls. Pitfall: several engineers running commands in parallel without a shared log. -->

| Role | Responsibility | Name | Contact |
|---|---|---|---|
| Upgrade lead | Runs the procedure, decides go, pause or rollback | [@name] | [Channel or phone] |
| Second engineer | Verifies each step, keeps the execution log | [@name] | [Contact] |
| Service owners on standby | Check their services at verification gates | [@names] | [Contact] |
| Incident commander on call | Takes command if the upgrade causes an incident | [@name] | [Paging route] |
| Communications | Status page and internal updates | [@name] | [Contact] |
| Provider support | Escalation for managed control plane issues | [Plan and case link] | [Contact] |

If the upgrade causes customer impact, declare an incident and hand command to the incident commander; roles then follow Incident Commander Handbook (`ops/incident-commander-handbook`) and severity follows Incident Severity Matrix (`ops/incident-severity-matrix`).

## 3. Support window and version skew check

<!-- Guidance: Prove the target is supported and every component will stay within the skew policy at every moment of the upgrade. Pitfall: a node pool left three minor versions behind, which then blocks the next control plane upgrade. -->

### 3.1 Support window

Check the release table at kubernetes.io/releases and the provider's version calendar on the day of planning and again on the day before the window.

| Check | Value | Source | Result |
|---|---|---|---|
| Target minor version supported upstream | [1.37, end of life YYYY-MM-DD] | kubernetes.io/releases | [Pass / Fail] |
| Target patch is the latest patch of the minor | [1.37.1] | kubernetes.io/releases | [Pass / Fail] |
| Target available and supported by provider | [Yes, standard support until YYYY-MM-DD] | Provider calendar | [Pass / Fail] |
| Source version end of life | [YYYY-MM-DD] | kubernetes.io/releases | [Information] |

### 3.2 Version skew

The Kubernetes version skew policy, as published on 2026-10-24, sets these limits relative to the API server:

| Component | Allowed relative to kube-apiserver |
|---|---|
| Other kube-apiserver instances (HA) | Within one minor version of each other |
| kubelet | Not newer; up to three minor versions older |
| kube-proxy | Not newer; up to three minor versions older, and within three minor versions of the kubelet on the same node |
| kube-controller-manager, kube-scheduler, cloud-controller-manager | Not newer; expected to match, may be one minor version older |
| kubectl | Within one minor version, older or newer |

While API servers in a highly available control plane run mixed versions, the allowed range for the other components narrows to the oldest API server. Some providers and tools impose tighter limits than the upstream policy; record them.

| Component | Current version | Allowed after control plane reaches target | Action |
|---|---|---|---|
| kubelet, pool np-general *(Example)* | 1.36.5 | 1.37, 1.36, 1.35, 1.34 | Upgrade in section 10 |
| kubelet, pool np-memory *(Example)* | 1.36.5 | Same | Upgrade in section 10 |
| kubelet, pool np-gpu *(Example)* | 1.36.5 | Same | Defer to next window, still within skew |
| kubectl in CI images *(Example)* | 1.35 | 1.38, 1.37, 1.36 | Update CI image to 1.37 before the window |

Any kubelet or kube-proxy already three minor versions behind the API server must be upgraded before the control plane can move.

## 4. Release notes and deprecated API scan

<!-- Guidance: Find every API version, flag and behaviour that the target release removes or changes, and fix it before the window. Pitfall: scanning only Git manifests and missing objects created by operators or by hand. -->

### 4.1 Release notes review

Read the release notes and the "urgent upgrade notes" of the target minor version and of every patch in between, plus the Kubernetes deprecated API migration guide. Record each item that affects this cluster.

| ID | Change in target release | Affects | Action | Owner | Status |
|---|---|---|---|---|---|
| KUR-RN-001 *(Example)* | [Removed API version or changed default] | [Charts, operators, manifests] | [Migrate to new version] | [@name] | [Done / Open] |
| KUR-RN-002 *(Example)* | [Feature gate graduated or removed] | [Component flags] | [Remove flag] | [@name] | [Status] |
| KUR-RN-003 *(Example)* | [Changed default in kubelet] | [Node pools] | [Test in staging] | [@name] | [Status] |

### 4.2 Deprecated API scan

Use at least two sources, because each sees a different part of the picture:

1. **Live cluster usage.** Query the API server metric `apiserver_requested_deprecated_apis` and audit events annotated as deprecated over a period that covers monthly jobs (at least [30] days of audit data). These show which clients still call deprecated versions.
2. **Stored objects and manifests.** Scan Git repositories, Helm releases and rendered manifests with a deprecated API detection tool, and render every chart with the target version's API list.
3. **Admission webhooks.** Check that webhooks match new resource versions; confirm that no webhook sets `matchPolicy: Exact`, so that requests for equivalent API versions are still intercepted (`Equivalent` is the default in `admissionregistration.k8s.io/v1`).

```bash
# Example: list deprecated API calls seen by the API server
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
```

| Finding ID | Object or client | Deprecated version | Replacement | Fix owner | Fixed in staging |
|---|---|---|---|---|---|
| KUR-API-001 *(Example)* | [Chart or controller name] | [group/version] | [group/version] | [@team] | [Yes / No] |

Exit criterion: no client of a version removed in the target release in the last [30] days of audit data, and no stored manifest using one.

## 5. Add-on, CRD and operator compatibility

<!-- Guidance: Check every component that talks to the API server against its support matrix for the target version. Pitfall: an operator that stores CRD objects in a version it stops serving after its own upgrade. -->

| Component | Installed version | Supports target? | Required version | Upgrade order | Source |
|---|---|---|---|---|---|
| CNI plugin *(Example)* | [x.y] | No | [x.z] | Before control plane | Vendor compatibility matrix |
| Gateway API implementation *(Example)* | [x.y] | Yes | Same | No change | Vendor notes |
| Policy engine or admission webhooks *(Example)* | [x.y] | Yes from [x.z] | [x.z] | Before control plane | Release notes |
| Certificate controller *(Example)* | [x.y] | Yes | Same | No change | Release notes |
| Storage CSI drivers *(Example)* | [x.y] | Yes | Same | After control plane | Provider |
| Monitoring stack *(Example)* | [x.y] | Yes | Same | No change | Release notes |
| Backup tool *(Example)* | [x.y] | Yes from [x.z] | [x.z] | Before control plane | Release notes |
| Autoscaler *(Example)* | Version tied to minor | No | Version matching 1.37 | With control plane | Project docs |

For each CRD-backed operator, record the stored versions (`status.storedVersions` of the CRD) and confirm that the new operator release still serves them or migrates them. Upgrade operators in staging first and run their own upgrade notes.

## 6. Workload disruption readiness

<!-- Guidance: Make sure draining nodes cannot take a service below its capacity or quorum. Pitfall: a PodDisruptionBudget that allows zero disruptions, which blocks the drain and stalls the upgrade for hours. -->

Draining a node evicts its pods through the eviction API, which honours PodDisruptionBudgets. Check before the window:

1. Every production workload with two or more replicas has a PodDisruptionBudget.
2. No PodDisruptionBudget has `ALLOWED DISRUPTIONS` at 0 in steady state, unless the owner has agreed a manual step.
3. Single-replica workloads are listed and their owners accept a short outage, or scale to two replicas for the window.
4. Stateful workloads with quorum (databases, queues, search) have a PodDisruptionBudget that keeps quorum and an owner on standby.
5. Workloads use readiness probes so that traffic shifts only to ready pods.
6. Pods with local storage (`emptyDir`) are known; their data is lost on drain.

```bash
# Example: find budgets that currently allow no disruption
kubectl get pdb -A -o wide | awk 'NR==1 || $5 == 0'
```

| Workload | Namespace | Replicas | PDB | Allowed disruptions | Action |
|---|---|---|---|---|---|
| search-replica *(Example)* | catalog | 3 | minAvailable 3 | 0 | Change to maxUnavailable 1 before window |
| legacy-report *(Example)* | finance | 1 | None | Not applicable | Owner accepts 2 minute restart |

## 7. Backup and rollback limits

<!-- Guidance: Take the backups that recovery depends on and state plainly what cannot be rolled back. Pitfall: assuming a control plane can be downgraded after a failed upgrade. -->

Take and verify these backups no more than [2] hours before the window:

| Backup | Method | Location | Verified by |
|---|---|---|---|
| etcd snapshot (self-managed) | `etcdctl snapshot save`, then `etcdutl snapshot status` | Encrypted bucket outside cluster | [@name] |
| Namespaced objects and volumes | Backup tool run for all namespaces | Backup vault | [@name] |
| Cluster configuration | Git commit hash of the GitOps repository | Git | [@name] |
| Provider settings | Export of cluster and node pool configuration | Change record attachment | [@name] |

Rollback limits to accept before approval:

- **Control plane.** Downgrading the control plane to the previous minor version is not a supported path upstream, and managed services generally do not offer it. Once the API server has written objects in the new version, the only full rollback is restoring etcd from the snapshot (self-managed) or rebuilding the cluster from Git and backups. Treat the control plane step as the point of no return.
- **Nodes.** Node pools can be rolled back by creating a new pool at the previous version (allowed, because kubelets may be older than the API server) and moving workloads to it.
- **Add-ons.** Roll back add-ons through their own release history, provided their CRD versions allow it.
- **Data written during the window.** Restoring etcd discards cluster changes after the snapshot; workloads' external data is not affected.

## 8. Go or no-go checklist

<!-- Guidance: Decide go or no-go at a fixed time before the window with the named approvers. Pitfall: proceeding with open items because the window is booked. -->

| ID | Check | Evidence | Result |
|---|---|---|---|
| KUR-GO-01 | Staging upgraded to the same target and stable for [5] business days | Staging runbook record | [Pass / Fail] |
| KUR-GO-02 | Version support and skew checks pass (section 3) | Section 3 tables | [Pass / Fail] |
| KUR-GO-03 | No removed API in use (section 4) | Scan output | [Pass / Fail] |
| KUR-GO-04 | Add-ons at compatible versions (section 5) | Section 5 table | [Pass / Fail] |
| KUR-GO-05 | PodDisruptionBudgets allow disruption (section 6) | Command output | [Pass / Fail] |
| KUR-GO-06 | Backups taken and verified (section 7) | Backup job IDs | [Pass / Fail] |
| KUR-GO-07 | Spare quota for surge nodes in each zone | Quota report | [Pass / Fail] |
| KUR-GO-08 | No open SEV1 or SEV2 incident on the cluster (Incident Severity Matrix, `ops/incident-severity-matrix`) | Incident tool | [Pass / Fail] |
| KUR-GO-09 | Change approved and maintenance notice sent | Change record | [Pass / Fail] |
| KUR-GO-10 | Error budget of affected SLOs not exhausted | SLO dashboard | [Pass / Fail] |

Decision: [Go / No-go], recorded by [@upgrade-lead] at [YYYY-MM-DD HH:MM].

## 9. Procedure: control plane

<!-- Guidance: Upgrade the API servers first, then controller manager and scheduler, verifying after each step. Pitfall: upgrading the control plane while admission webhooks cannot handle the new API versions. -->

Record the start and end time of each step in the execution log.

1. **Upgrade add-ons that must move first** (section 5), one at a time, verifying each. *Verify:* add-on pods ready, no new errors in their logs.
2. **Upgrade the API server.**
   - Managed service: start the control plane upgrade to the target patch through the provider interface or infrastructure code. *Verify:* the provider reports the target version and the cluster as healthy.
   - Self-managed with kubeadm: follow the kubeadm upgrade page on kubernetes.io for the target minor, in this order.
     1. On every node, switch the package repository to the target minor.
     2. On the first control plane node: upgrade the `kubeadm` package to the target patch, check it with `kubeadm version`, run `sudo kubeadm upgrade plan`, then `sudo kubeadm upgrade apply v[target]`. Upgrade the CNI plugin if its provider requires it (section 5).
     3. On each other control plane node, one at a time: upgrade the `kubeadm` package, then run `sudo kubeadm upgrade node`.
     4. Then, on each control plane node in turn: `kubectl drain <node> --ignore-daemonsets`, upgrade the `kubelet` and `kubectl` packages to the target patch, run `sudo systemctl daemon-reload` and `sudo systemctl restart kubelet`, then `kubectl uncordon <node>`.
     5. Self-managed worker nodes that are upgraded in place instead of replaced follow the same pattern in section 10: upgrade `kubeadm`, run `sudo kubeadm upgrade node`, drain, upgrade `kubelet` and `kubectl`, restart kubelet, uncordon.

     *Verify:* each API server reports the target version, and `kubectl get nodes` shows every control plane node `Ready` at the target kubelet version.
3. **Controller manager, scheduler and cloud controller manager.** These move with the control plane on managed services and kubeadm. *Verify:* their leases renew and no crash loops occur.
4. **Check API health.**

```bash
kubectl version
kubectl get --raw='/readyz?verbose'
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
```

5. **Gate 1.** Service owners confirm their dashboards show no regression for [15] minutes. If any check fails, stop and go to section 12.

## 10. Procedure: node pools

<!-- Guidance: Replace nodes with surge capacity, draining old nodes gracefully. Pitfall: in-place kubelet minor upgrades, which the project does not support; always drain first. -->

Upgrade pools in this order: system pool, then general pools, then special pools (GPU, memory, spot). For each pool:

1. **Set surge settings.** *Example:* max surge 1 node per zone, max unavailable 0, so capacity never drops.
2. **Start the pool upgrade** (managed) or create a replacement pool at the target version (self-managed or blue-green). Kubelets must not be upgraded across minor versions while pods run on the node: drain the node first. For kubeadm nodes upgraded in place, work one node (or one zone batch) at a time: upgrade the `kubeadm` package, run `sudo kubeadm upgrade node`, drain (step 3), upgrade the `kubelet` and `kubectl` packages to the target patch, run `sudo systemctl daemon-reload` and `sudo systemctl restart kubelet`, then `kubectl uncordon <node>`.
3. **Drain each old node.**

```bash
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=15m
```

4. **Watch evictions.** If a drain waits on a PodDisruptionBudget for more than [15] minutes, page the workload owner; do not force eviction of quorum workloads.
5. **Verify each zone batch:** new nodes `Ready` at the target version, pods rescheduled and ready, no pending pods.
6. **Delete old nodes** or the old pool once it is empty.
7. **Gate 2** per pool: service owners confirm no regression for [15] minutes.

| Pool | Nodes | Surge | Start | End | Issues |
|---|---|---|---|---|---|
| np-system *(Example)* | 3 | 1 per zone | HH:MM | HH:MM | None |
| np-general *(Example)* | 24 | 1 per zone | HH:MM | HH:MM | PDB wait on search-replica, 6 min |
| np-memory *(Example)* | 6 | 1 per zone | HH:MM | HH:MM | None |

## 11. Verification

<!-- Guidance: Prove the cluster and its services are healthy at the target version before closing the window. Pitfall: checking only that nodes are Ready, not that traffic, jobs and alerts work. -->

| ID | Check | Method | Expected | Result |
|---|---|---|---|---|
| KUR-VER-01 | Control plane and all nodes at target | `kubectl get nodes` | All target, except deferred pools | [Pass / Fail] |
| KUR-VER-02 | System add-ons healthy | Pods in system namespaces | All ready, no restarts | [Pass / Fail] |
| KUR-VER-03 | No pending or crash-looping pods | Pod status query | None new since baseline | [Pass / Fail] |
| KUR-VER-04 | DNS resolution in and out of cluster | Test pod | Success | [Pass / Fail] |
| KUR-VER-05 | Ingress and Gateway routes | Synthetic checks | All green | [Pass / Fail] |
| KUR-VER-06 | Autoscaling | Scale test deployment | New node joins at target | [Pass / Fail] |
| KUR-VER-07 | Storage | Create and attach a test volume | Bound and mounted | [Pass / Fail] |
| KUR-VER-08 | Admission policies enforce | Apply a non-compliant test pod | Rejected | [Pass / Fail] |
| KUR-VER-09 | SLOs | SLO dashboards for [60] minutes | No burn alert | [Pass / Fail] |
| KUR-VER-10 | Backup job after upgrade | Run backup | Succeeds | [Pass / Fail] |
| KUR-VER-11 | Benchmark scan | Hardening scan | No new deviations | [Pass / Fail] |

## 12. Rollback and recovery

<!-- Guidance: Define in advance what triggers rollback at each stage and what rollback means there. Pitfall: debating rollback during an outage instead of following preset triggers. -->

| Stage | Trigger | Action | Decision by |
|---|---|---|---|
| Before control plane step | Any go or no-go item fails | Cancel; reschedule | Upgrade lead |
| Add-on upgrade fails | Add-on not ready after [10] minutes | Roll back add-on release | Upgrade lead |
| Control plane upgrade fails (managed) | Provider reports failure | Open provider case at highest priority; keep nodes at source version | Upgrade lead with provider |
| Control plane upgraded, services degraded | Gate 1 fails | Declare incident; fix forward where cause is known; otherwise restore etcd (self-managed) or fail over to the other region | Incident commander |
| Node pool upgrade degrades services | Gate 2 fails | Pause pool upgrade; recreate a pool at the source version (within skew) and move workloads | Upgrade lead |
| Data or object loss | Missing objects found | Restore namespaces from backup | Incident commander |

After any rollback, update the change record, keep the cluster in its current state until a review decides the next step, and record the cause in a postmortem using Postmortem (`dev/postmortem`).

## 13. Communication plan

<!-- Guidance: Tell the right people before, during and after the window. Pitfall: product teams discovering the upgrade through failed deployments during the freeze. -->

| When | Audience | Channel | Message | Owner |
|---|---|---|---|---|
| T minus 10 business days | Product teams | Engineering channel and email | Window, freeze, actions on deprecated APIs and PDBs | Upgrade lead |
| T minus 5 business days | Customers (if visible impact possible) | Maintenance notice | Window and expected impact | Communications |
| T minus 1 day | Approvers | Change meeting | Go or no-go result | Upgrade lead |
| Start of window | Engineering, support | Incident channel and status page if public | Upgrade started | Communications |
| Each gate | Engineering | Upgrade channel | Gate result | Second engineer |
| End of window | All above | Same channels | Completed or rolled back, next steps | Upgrade lead |

## 14. Worked example: 1.36 to 1.37

<!-- Guidance: Use this as a model of a filled record; every value is fictional. -->

*Example:* Corvid Commerce upgrades the prod-eu-1 cluster (managed control plane, four node pools) from 1.36.5 to 1.37.1. Kubernetes 1.37.0 was released upstream on 2026-08-26; on 2026-10-24 the upstream table lists 1.37, 1.36, 1.35 and 1.34 as maintained, with 1.34 reaching end of life on 2026-10-27.

| Step | Day and time *(Example)* | Detail | Result |
|---|---|---|---|
| Planning | Day minus 15 | Provider offers 1.37.1; staging upgraded day minus 12 | Staging stable 7 business days |
| API scan | Day minus 10 | Audit data for 30 days: one batch job used an API version deprecated (not removed) in 1.37 through an old client library; no client used a removed version | Library updated; logged as KUR-API-001 for the next minor |
| Add-on check | Day minus 8 | CNI requires a newer minor for 1.37; backup tool supported | CNI upgraded in staging, then production on day minus 3 |
| PDB check | Day minus 2 | search-replica budget allowed 0 disruptions | Changed to maxUnavailable 1 |
| Go or no-go | Day minus 1, 16:00 | 10 of 10 checks pass | Go |
| Backups | Day 0, 21:10 | Namespace and volume backup, Git hash recorded | Verified |
| Control plane | Day 0, 21:30 to 21:52 | Provider upgrade to 1.37.1 | Gate 1 passed at 22:07 |
| np-system | Day 0, 22:10 to 22:31 | 3 nodes, surge 1 per zone | Gate 2 passed |
| np-general | Day 0, 22:35 to 23:58 | 24 nodes in batches of 3 (one per zone): 8 batches at about 10 minutes each | One PDB wait of 6 minutes |
| np-memory | Day 1, 00:02 to 00:40 | 6 nodes | Passed |
| np-gpu | Deferred | Driver not yet validated; 1.36 kubelet within skew | New runbook KUR-002 within 30 days |
| Verification | Day 1, 00:45 to 01:45 | KUR-VER-01 to 11 | All pass |

Totals for the example: 33 nodes replaced (3 + 24 + 6), window used 4 hours 35 minutes of the 6 hours booked, no customer impact.

## 15. Outcome and closure

<!-- Guidance: Close the change with evidence and lessons, and schedule the next step. Pitfall: leaving deferred pools without a dated follow-up, so skew grows silently. -->

| Field | Value |
|---|---|
| Outcome | [Completed / Completed with deferrals / Rolled back] |
| Final versions | [Control plane x.y.z; pools and versions] |
| Deviations from runbook | [List or "None"] |
| Incidents raised | [Incident IDs or "None"] |
| Deferred items | [Pool or add-on, owner, due date] |
| Lessons learned | [Two or three points to change in the next runbook] |
| Change record closed | [CHG-0000, date] |
| Next minor upgrade planned | [Target version, environment, month] |

## 16. Glossary and references

<!-- Guidance: Keep terms short and list the pages checked with their dates. -->

### 16.1 Glossary

| Term | Meaning |
|---|---|
| Minor version | The second number in a Kubernetes version (1.37); upgrades move one at a time |
| Patch release | Bug and security fix release within a minor version (1.37.1) |
| Version skew | Allowed version difference between components and the API server |
| Deprecated API | API version still served but scheduled for removal in a later release |
| Drain | Cordoning a node and evicting its pods through the eviction API |
| Surge upgrade | Adding new nodes before removing old ones so capacity does not drop |
| PodDisruptionBudget | Limit on how many pods of a set may be voluntarily disrupted at once |
| Stored version | CRD version in which objects are persisted in etcd |

### 16.2 References

| Reference | Version or date checked | Use |
|---|---|---|
| Kubernetes Releases, kubernetes.io/releases | Checked 2026-10-24 | Supported versions and end of life dates |
| Kubernetes 1.37 release page, kubernetes.io/releases/1.37 | 1.37.0 released 2026-08-26 | Worked example |
| Kubernetes Version Skew Policy, kubernetes.io/releases/version-skew-policy | Checked 2026-10-24 | Skew table and upgrade order |
| Kubernetes Deprecated API Migration Guide, kubernetes.io/docs/reference/using-api/deprecation-guide | Check for the target release | Removed APIs |
| Kubernetes documentation: Upgrading kubeadm clusters, Safely Drain a Node, Specifying a Disruption Budget, Operating etcd clusters | Checked 2026-10-24 | Procedure concepts in original words |
| Provider upgrade documentation | [Provider, date] | Managed control plane steps |

## Standard and licence note

Based on the Kubernetes release, version skew policy and task documentation published at kubernetes.io by The Kubernetes Authors under the Creative Commons Attribution 4.0 International licence (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/). The skew limits are summarised in original words and no documentation text is copied. Kubernetes is a registered trademark of The Linux Foundation. Other product and service names are trademarks of their owners; this template is not affiliated with or endorsed by them. This template is original text written for this collection. Check the current release table, skew policy and release notes for the target version before use. Template licence: MIT.