Following the stable channel would have skipped two minor versions

This lab runs on a real k3s

There is one k3s v1.34.11+k3s1 running in the VM. You raise this cluster one step to 1.35 and actually do the preparation before the upgrade and the comparison after it. It takes about 2 minutes to come up the first time. The working directory is /root/upgrade.

Goal

While upgrading k3s by one minor version, you do in turn recording the starting version, checking deprecated APIs, backing up, and judging drain, and after the upgrade you prove by UID that the same node and the same workloads are alive. Finally, you write the next step as a system-upgrade-controller Plan.

Why it matters

Because running the install script again completes the upgrade, choosing the version is taken lightly. But a channel (stable) does not know what version the cluster is on now and goes to the latest recommended version. If you go from 1.34 to stable, on a day when stable is 1.36 you skip two minor versions, and the Kubernetes skew policy does not allow this even for a cluster with only one instance. No tool prevents skipping, so a person must pin the version and raise it one step at a time.

And "it went up" is not evidence. If you do not write down the version and UIDs before raising it, there is no way to confirm after raising it that the same cluster came up carrying the same workloads.

Steps

1. Save the API server version, kubelet version, node name, and node UID to /root/upgrade/before.json.
2. In the upg namespace, create a Deployment web (nginx 1.27-alpine, 2 replicas) and a PDB web (minAvailable 1), and write the UIDs to /root/upgrade/workload.json.
3. Copy the apiserver_requested_deprecated_apis metric to /root/upgrade/deprecated.txt and write removed_by_target=.
4. Back up the SQLite DB and the token to /root/upgrade/backup/ and write /root/upgrade/backup.json.
5. Try a drain, leave the output in /root/upgrade/drain.txt, and after uncordoning, append a decision= line.
6. Upgrade to v1.35.8+k3s1 and write /root/upgrade/upgrade.json.
7. Write the post-upgrade state and the skew allowed range to /root/upgrade/after.json.
8. Add the Plan CRD and apply a Plan k3s-server-next that pins the next step, as /root/upgrade/plan.yaml.
9. In /root/upgrade/report.md, write five lines and an explanation.

Notes

• The k3s configuration of this VM is in /etc/rancher/k3s/config.yaml (disable: traefik). Values given by environment variable at install time disappear if you do not give them again when you run it again, but the configuration file remains.
• Images use public.ecr.aws/docker/library/.... Docker Hub may hit pull limits.
• Common mistake 1: upgrading first and writing before.json later. The grader checks whether the record was written before the upgrade.
• Common mistake 2: forgetting to uncordon after a drain fails. The replacement Pod stays in Pending and all the Ready checks in later steps fail.
• The system-upgrade-controller controller and the rancher/k3s-upgrade image are on Docker Hub, so they are not run in this environment. The Plan is only stored, and no actual upgrade takes place.

Write down the starting version first

Save the API server version, kubelet version, node name, and node UID to /root/upgrade/before.json with the keys server_version, kubelet_version, node_name, and node_uid.

After the upgrade, you can no longer see the previous version anywhere. Look at serverVersion in kubectl version -o json and status.nodeInfo and metadata.uid of the node object. The UID is the value you use in a later step to compare 'whether the same node was upgraded.'

Bring up a workload with a PDB

In the upg namespace, create a Deployment web (image public.ecr.aws/docker/library/nginx:1.27-alpine, 2 replicas) and a PodDisruptionBudget web (minAvailable 1) that selects app=web, and save the Deployment UID to /root/upgrade/workload.json with the keys deployment_uid and pdb_min_available.

A PDB is a promise that 'at least this many stay during a voluntary disruption (eviction).' You can create them with kubectl create deployment and kubectl create pdb, and write the UID after both Pods are Ready. Do not use Docker Hub images.

Who is still calling old APIs

Copy the apiserver_requested_deprecated_apis lines from the API server metrics as they are to /root/upgrade/deprecated.txt, and among them write the number of those that are removed up to the target version 1.35 (removed_release is 1.35 or lower) as a removed_by_target=<수> line (replace the placeholder with that number).

You can see the API server metrics with kubectl get --raw /metrics. If the removed_release label is empty, it means the API is only deprecated and has no removal schedule. This cluster shows one line even if you do nothing.

Back up the DB and the token together

Back up the SQLite datastore to /root/upgrade/backup/state.db and the server token to /root/upgrade/backup/token, and write the backed-up version and the token hash to /root/upgrade/backup.json with the keys k3s_version and token_sha256.

The SQLite of k3s is in /var/lib/rancher/k3s/server/db/. For a DB in use, sqlite3's online backup command is safer than copying the file. For why the token is needed, see the backup section of the reading again.

Drain does not finish

Try emptying the node with kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=40s, save the entire output to /root/upgrade/drain.txt, and then uncordon the node. On the last line, write the decision as decision=upgrade-without-drain or decision=add-node-first.

Drain cordons and then sends Pods out through the eviction API, and eviction honors the PDB. If there is only one node, think about where the replacement Pod for an evicted Pod is supposed to go. Even if the drain ends in failure, you must leave the output in the file.

Raise only one minor step

Download the install script from the target version tag, upgrade with INSTALL_K3S_VERSION=v1.35.8+k3s1, and write the result to /root/upgrade/upgrade.json with the keys from, to, and node_uid.

Pin the version instead of a channel. The install script downloads the install.sh of that version tag from the k3s-io/k3s repository at raw.githubusercontent.com. This VM's configuration is in /etc/rancher/k3s/config.yaml, so you do not need to give it again. When it finishes, wait until the API responds and then write the values.

Compare whether the same things came up

Write the post-upgrade state to /root/upgrade/after.json with the keys server_version, deployment_uid, web_ready, web_restarts (the sum of the web Pod restarts), traefik_deployed, kubelet_allowed, and kubectl_allowed. The last two are the lists of minor versions ("1.xx" strings) that the skew policy allows for the current API server version.

Compare workloads by UID, and settings by whether traefik came back to life. For how many versions lower than the API server the kubelet may be and how many versions above and below kubectl may be under the skew policy, see the table in the reading.

Write the next step as a Plan

Apply the crd.yaml of system-upgrade-controller v0.20.1, and write and apply a Plan k3s-server-next in the system-upgrade namespace as /root/upgrade/plan.yaml. Pin the very next minor k3s version after the current one with version, select only server nodes, and have it cordon and upgrade one at a time.

You get the CRD from the pinned-version address of the GitHub release. The controller is not started, so the Plan is not executed and is only stored. The CRD accepts a Plan with neither version nor channel as it is, so that applying succeeded does not make it a correct Plan. Think about what to change the channel to in the server Plan example in the official k3s documentation.

How many steps it would have been if you had followed stable

In /root/upgrade/report.md, write five lines — initial=, current=, next_plan=, stable_channel= (the version the k3s stable channel points to now), and stable_minor_jump= (the number of minor steps from the starting version to that version) — and an explanation of at least 150 characters on why you pinned the version instead of the channel.

https://update.k3s.io/v1-release/channels gives the version a channel points to as JSON. The number of steps is the difference in the minor numbers. In the explanation, include the skew policy and what did not stop you from skipping.