Automated K3s Upgrades with System Upgrade Controller

Overview

The K3s System Upgrade Controller (SUC) automates cluster upgrades by applying Plan resources that define target versions, node selectors, and upgrade strategies. It performs rolling upgrades — draining and cordoning nodes one at a time to maintain availability.

Why This Matters

• -Zero-downtime upgrades — rolling strategy keeps workloads running
• -Declarative — upgrade plans are Kubernetes resources managed via GitOps
• -Safe — drain and cordon prevent scheduling during upgrades
• -Automated — no SSH-ing into nodes to run upgrade scripts

How It Works

Step 1: Install the System Upgrade Controller

# Apply the SUC manifest
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

# Verify it is running
kubectl -n system-upgrade get pods

Step 2: Create Server Upgrade Plan

# server-plan.yaml
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/master
        operator: In
        values: ["true"]
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.29.2+k3s1

Step 3: Create Agent Upgrade Plan

# agent-plan.yaml
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/master
        operator: DoesNotExist
  prepare:
    args: ["prepare", "server-plan"]
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  version: v1.29.2+k3s1

Step 4: Apply and Monitor

# Apply upgrade plans
kubectl apply -f server-plan.yaml
kubectl apply -f agent-plan.yaml

# Monitor upgrade progress
watch kubectl get nodes -o wide
kubectl -n system-upgrade get plans
kubectl -n system-upgrade get jobs

Best Practices

• -Always upgrade server nodes before agent nodes (use prepare step)
• -Set concurrency to 1 for safe rolling upgrades
• -Test target version on a staging cluster first
• -Create an etcd snapshot before starting the upgrade
• -Use the channel field instead of version for automatic latest stable

Common Mistakes

• -Upgrading agents before servers (version skew issues)
• -Setting concurrency too high (multiple nodes down simultaneously)
• -Not testing the target version in staging first
• -Forgetting to snapshot etcd before upgrade (no rollback point)
• -Skipping minor versions (always upgrade incrementally)