Kubernetes - Automated K3s Upgrades with System Upgrade Controller

Table of Contents

This post is part of our ongoing series of posts for Kubernetes. In this post, we will upgrade our K3s cluster using System Upgrade Controller, a Kubernetes controller from Rancher that upgrades nodes based on declarative Plan resources.

In our previous post, Renovate pointed out how outdated our Helm charts were, and we held back some of the updates because they need a newer Kubernetes version. The cluster itself was even further behind: the server node was running v1.31.5+k3s1, the agent node v1.31.6+k3s1, while the current stable K3s release is v1.36.5+k3s1. Upgrading K3s manually means SSHing into every node, cordoning it, running the install script again with the new version and waiting for it to come back, one node and one minor version at a time. Doing this by hand is error-prone and time-consuming.

System Upgrade Controller addresses this challenge by watching Plan resources and running upgrade jobs on the matching nodes, handling cordoning and ordering for us. A Plan is just another manifest, so the K3s version becomes part of our GitOps repository along with everything else. Let’s dive into setting it up.

1. Plan the Upgrade Path

Before making any changes, let’s check which version our nodes are running:

1kubectl get nodes
2
3NAME           STATUS   ROLES                  AGE    VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KER
4192.168.1.19   Ready    control-plane,master   611d   v1.31.5+k3s1   192.168.1.19   <none>        openSUSE MicroOS   7.3
5192.168.1.20   Ready    <none>                 524d   v1.31.6+k3s1   192.168.1.20   <none>        openSUSE MicroOS   7.3

Kubernetes only supports upgrading one minor version at a time, so we can’t go from v1.31 straight to v1.36. K3s publishes the latest release of each minor version through its channel server, which gives us the path:

1curl -s https://update.k3s.io/v1-release/channels | jq -r '.data[] | "\(.id) \(.latest)"'
2
3stable v1.36.5+k3s1
4latest v1.37.1+k3s1
5...
6v1.31 v1.31.14+k3s1
7v1.32 v1.32.13+k3s1
8v1.33 v1.33.13+k3s2
9...

In this post, we’ll first move both nodes to the latest v1.31 patch release, v1.31.14+k3s1, and then to v1.32.13+k3s1. The rest of the minor versions follow the same process.

2. Setup System Upgrade Controller

System Upgrade Controller doesn’t ship a Helm chart, so instead of a HelmRelease we’ll commit the release manifests to our GitOps repository. Let’s download them for version v0.20.2 into cluster/default/system-upgrade/:

1mkdir -p cluster/default/system-upgrade && cd cluster/default/system-upgrade
2curl -sfLO https://github.com/rancher/system-upgrade-controller/releases/download/v0.20.2/crd.yaml
3curl -sfLO https://github.com/rancher/system-upgrade-controller/releases/download/v0.20.2/system-upgrade-controller.yaml

Since these manifests will run privileged jobs on our nodes, let’s make sure they match the checksums published with the release:

1curl -sL https://github.com/rancher/system-upgrade-controller/releases/download/v0.20.2/sha256sum-amd64.txt | grep -E "crd.yaml|controller.yaml"
2
31fb061e0d8bf9534da0145e614a9e03ca376cd5d1adcca17d13cfa12e211c48c  dist/artifacts/crd.yaml
4fd474b662d1f2b74ab150d95b6482869e1fd1f27ceee54b13e94b92292097969  dist/artifacts/system-upgrade-controller.yaml

1sha256sum crd.yaml system-upgrade-controller.yaml
2
31fb061e0d8bf9534da0145e614a9e03ca376cd5d1adcca17d13cfa12e211c48c  crd.yaml
4fd474b662d1f2b74ab150d95b6482869e1fd1f27ceee54b13e94b92292097969  system-upgrade-controller.yaml

Both checksums match. Next, add cluster/default/system-upgrade/kustomization.yaml so Flux applies both files from this directory:

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3resources:
4  - crd.yaml
5  - system-upgrade-controller.yaml

The controller manifest includes:

  • The system-upgrade namespace with the pod-security.kubernetes.io/enforce: privileged label, as upgrade jobs need host access
  • The system-upgrade service account and the RBAC rules used by the controller and its jobs
  • A default-controller-env ConfigMap with the controller settings, like SYSTEM_UPGRADE_PLAN_POLLING_INTERVAL: 15m and SYSTEM_UPGRADE_JOB_ACTIVE_DEADLINE_SECONDS: "900"
  • The controller Deployment running rancher/system-upgrade-controller:v0.20.2

Let’s commit and push these changes. Once Flux has reconciled them, verify that the controller is up and running:

1kubectl -n system-upgrade get pods
2
3NAME                                         READY   STATUS    RESTARTS   AGE
4system-upgrade-controller-759dcbc44c-r7tn7   1/1     Running   0          20s

As shown above, System Upgrade Controller has been successfully deployed. It doesn’t do anything yet, as there are no plans to apply.

3. Create Upgrade Plans

Now we’ll describe what our upgrade looks like by creating cluster/default/system-upgrade/k3s-upgrade-plans.yaml with one plan for the server node and one for the agent nodes:

 1apiVersion: upgrade.cattle.io/v1
 2kind: Plan
 3metadata:
 4  name: server-plan
 5  namespace: system-upgrade
 6spec:
 7  concurrency: 1
 8  cordon: true
 9  nodeSelector:
10    matchExpressions:
11      - key: node-role.kubernetes.io/control-plane
12        operator: In
13        values:
14          - "true"
15  serviceAccountName: system-upgrade
16  upgrade:
17    image: rancher/k3s-upgrade
18  version: v1.31.14+k3s1
19---
20apiVersion: upgrade.cattle.io/v1
21kind: Plan
22metadata:
23  name: agent-plan
24  namespace: system-upgrade
25spec:
26  concurrency: 1
27  cordon: true
28  nodeSelector:
29    matchExpressions:
30      - key: node-role.kubernetes.io/control-plane
31        operator: DoesNotExist
32  prepare:
33    args:
34      - prepare
35      - server-plan
36    image: rancher/k3s-upgrade
37  serviceAccountName: system-upgrade
38  upgrade:
39    image: rancher/k3s-upgrade
40  version: v1.31.14+k3s1

This configuration:

  • Selects nodes by the node-role.kubernetes.io/control-plane label, so the server plan only runs on our control plane node and the agent plan on all other nodes
  • Upgrades one node at a time per plan with concurrency: 1
  • Cordons each node before upgrading it with cordon: true. We could use drain instead to evict pods first, but on a two-node cluster there aren’t many places to move pods to, and K3s keeps containers running while it restarts
  • Adds a prepare step to the agent plan that waits until server-plan is complete, so agents never run a newer version than the server
  • Runs the rancher/k3s-upgrade image, which replaces the K3s binary on the node and restarts K3s. Our /etc/rancher/k3s/config.yaml stays as it is, so the settings from Kubernetes - Installing Cilium CNI are kept
  • Pins an exact version. A plan can also follow a channel like https://update.k3s.io/v1-release/channels/stable and upgrade automatically, but we want each upgrade to be a commit we can review

Finally, add the plans to the kustomization.yaml:

1 resources:
2   - crd.yaml
3   - system-upgrade-controller.yaml
4+  - k3s-upgrade-plans.yaml

4. Upgrade to the Latest Patch Release

Let’s commit and push the plans, and trigger the reconciliation instead of waiting for the polling interval:

1flux reconcile source git flux-system
2flux reconcile kustomization flux-system

The controller sees both plans immediately and creates a job for each node. The server node is cordoned first, while the agent job waits in its prepare init container:

 1kubectl -n system-upgrade get plans
 2kubectl -n system-upgrade get jobs,pods
 3kubectl get nodes
 4
 5NAME          IMAGE                 CHANNEL   VERSION         COMPLETE   MESSAGE
 6agent-plan    rancher/k3s-upgrade             v1.31.14+k3s1   False
 7server-plan   rancher/k3s-upgrade             v1.31.14+k3s1   False
 8...
 9NAME                                                                  READY   STATUS            RESTARTS   AGE
10pod/apply-agent-plan-on-192-with-c1647c036547a825c7e3c09cfb62-sdnjg   0/1     Init:0/2          0          10s
11pod/apply-server-plan-on-192-with-c1647c036547a825c7e3c09cfb6-scvhx   0/1     PodInitializing   0          10s
12pod/system-upgrade-controller-759dcbc44c-r7tn7                        1/1     Running           0          54s
13
14NAME           STATUS                     ROLES                  AGE    VERSION
15192.168.1.19   Ready,SchedulingDisabled   control-plane,master   611d   v1.31.5+k3s1
16192.168.1.20   Ready                      <none>                 524d   v1.31.6+k3s1

For a few seconds, we can’t reach the API server while K3s restarts on the server node, as our cluster has a single control plane node:

1kubectl get nodes
2
3The connection to the server 192.168.1.19:6443 was refused - did you specify the right host or port?

Once it is back, the server node already reports the new version. The job pod that was running when K3s restarted shows as Unknown, so the job spins up a new pod to finish the plan:

 1NAME          IMAGE                 CHANNEL   VERSION         COMPLETE   MESSAGE
 2agent-plan    rancher/k3s-upgrade             v1.31.14+k3s1   False
 3server-plan   rancher/k3s-upgrade             v1.31.14+k3s1   True
 4...
 5NAME                                                                  READY   STATUS      RESTARTS   AGE
 6pod/apply-agent-plan-on-192-with-c1647c036547a825c7e3c09cfb62-sdnjg   0/1     Init:0/2    0          56s
 7pod/apply-server-plan-on-192-with-c1647c036547a825c7e3c09cfb6-9lkqn   0/1     Completed   0          16s
 8pod/apply-server-plan-on-192-with-c1647c036547a825c7e3c09cfb6-scvhx   0/1     Unknown     0          56s
 9...
10
11NAME           STATUS   ROLES                  AGE    VERSION
12192.168.1.19   Ready    control-plane,master   611d   v1.31.14+k3s1
13192.168.1.20   Ready    <none>                 524d   v1.31.6+k3s1

With server-plan complete, the agent job continues and the same happens on the agent node. About two minutes after the push, both plans are complete:

 1kubectl -n system-upgrade get plans
 2kubectl get nodes
 3
 4NAME          IMAGE                 CHANNEL   VERSION         COMPLETE   MESSAGE
 5agent-plan    rancher/k3s-upgrade             v1.31.14+k3s1   True
 6server-plan   rancher/k3s-upgrade             v1.31.14+k3s1   True
 7
 8NAME           STATUS   ROLES                  AGE    VERSION
 9192.168.1.19   Ready    control-plane,master   611d   v1.31.14+k3s1
10192.168.1.20   Ready    <none>                 524d   v1.31.14+k3s1

Both nodes are now on the same version again, and all HelmReleases in the cluster still report READY as True.

5. Upgrade to the Next Minor Version

The next step on our path is v1.32.13+k3s1. With the plans in place, upgrading is just a version bump in Git:

1   upgrade:
2     image: rancher/k3s-upgrade
3-  version: v1.31.14+k3s1
4+  version: v1.32.13+k3s1

After committing and pushing the change for both plans, the controller sees the new version and creates new jobs. This time the API server was briefly unavailable while starting up:

1kubectl get nodes
2
3Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes)

About a minute and a half later, the upgrade is done:

 1kubectl -n system-upgrade get plans
 2kubectl get nodes
 3
 4NAME          IMAGE                 CHANNEL   VERSION         COMPLETE   MESSAGE
 5agent-plan    rancher/k3s-upgrade             v1.32.13+k3s1   True
 6server-plan   rancher/k3s-upgrade             v1.32.13+k3s1   True
 7
 8NAME           STATUS   ROLES                  AGE    VERSION
 9192.168.1.19   Ready    control-plane,master   611d   v1.32.13+k3s1
10192.168.1.20   Ready    <none>                 524d   v1.32.13+k3s1

The results confirm the upgrade was successful. Our cluster went from two mismatched v1.31 versions to v1.32.13+k3s1 with two commits, and without logging in to any of the nodes.

What next ?

Future posts will keep bringing the cluster up to date, including:

  • Continuing the K3s upgrades towards the current stable release
  • Rolling out the Flux and Helm chart updates held back in the previous post
  • Migrating from ingress-nginx to Gateway API

Stay tuned for each of these topics.

References