{"id":13572735,"url":"https://github.com/rancher/system-upgrade-controller","last_synced_at":"2026-04-10T01:19:47.222Z","repository":{"id":40557648,"uuid":"229781511","full_name":"rancher/system-upgrade-controller","owner":"rancher","description":"In your Kubernetes, upgrading your nodes","archived":false,"fork":false,"pushed_at":"2025-04-16T22:46:22.000Z","size":22126,"stargazers_count":828,"open_issues_count":40,"forks_count":95,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-04-24T01:59:20.201Z","etag":null,"topics":["crd-controller","custom-resource-definition","kubernetes","upgrades"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rancher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-23T15:50:38.000Z","updated_at":"2025-04-24T00:28:58.000Z","dependencies_parsed_at":"2023-09-29T07:45:19.334Z","dependency_job_id":"e4fa798f-e87a-4532-9af9-2c7a93374493","html_url":"https://github.com/rancher/system-upgrade-controller","commit_stats":null,"previous_names":[],"tags_count":61,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rancher%2Fsystem-upgrade-controller","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rancher%2Fsystem-upgrade-controller/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rancher%2Fsystem-upgrade-controller/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rancher%2Fsystem-upgrade-controller/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rancher","download_url":"https://codeload.github.com/rancher/system-upgrade-controller/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253990468,"owners_count":21995774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crd-controller","custom-resource-definition","kubernetes","upgrades"],"created_at":"2024-08-01T15:00:18.637Z","updated_at":"2026-02-18T15:02:36.059Z","avatar_url":"https://github.com/rancher.png","language":"Go","funding_links":[],"categories":["👋 Introduction","Go","kubernetes"],"sub_categories":[],"readme":"# System Upgrade Controller\n\n## Introduction\n\nThis project aims to provide a general-purpose, Kubernetes-native upgrade controller (for nodes).\nIt introduces a new CRD, the **Plan**, for defining any and all of your upgrade policies/requirements.\nA **Plan** is an outstanding intent to mutate nodes in your cluster.\nFor up-to-date details on defining a plan please review [v1/types.go](pkg/apis/upgrade.cattle.io/v1/types.go).\n\n![diagram](doc/architecture.png \"The Controller manages Plans by selecting Nodes to run upgrade Jobs on.\n A Plan defines which Nodes are eligible for upgrade by specifying a label selector.\n When a Job has run to completion successfully the Controller will label the Node\n on which it ran according to the Plan that was applied by the Job.\")\n\n### Presentations and Recordings\n\n#### April 14, 2020\n[CNCF Member Webinar: Declarative Host Upgrades From Within Kubernetes](https://www.cncf.io/webinars/declarative-host-upgrades-from-within-kubernetes/)\n- [Slides](https://www.cncf.io/wp-content/uploads/2020/08/CNCF-Webinar-System-Upgrade-Controller-1.pdf)\n- [Video](https://www.youtube.com/watch?v=uHF6C0GKjlA)\n\n#### March 4, 2020\n[Rancher Online Meetup: Automating K3s Cluster Upgrades](https://info.rancher.com/online-meetup-automating-k3s-cluster-upgrades)\n- [Video](https://www.youtube.com/watch?v=UsPV8cZX8BY)\n\n### Considerations\n\nPurporting to support general-purpose node upgrades (essentially, arbitrary mutations) this controller attempts\nminimal imposition of opinion. Our design constraints, such as they are:\n\n- content delivery via container image a.k.a. container command pattern\n- operator-overridable command(s)\n- a very privileged job/pod/container:\n  - host IPC, NET, and PID\n  - CAP_SYS_BOOT\n  - host root file-system mounted at `/host` (read/write)\n- optional opt-in/opt-out via node labels\n- optional cordon/drain a la `kubectl`\n\n_Additionally, one should take care when defining upgrades by ensuring that such are idempotent--**there be dragons**._\n\n## Deploying\n\nThe most up-to-date manifest is usually [manifests/system-upgrade-controller.yaml](manifests/system-upgrade-controller.yaml)\nbut since release v0.4.0 a manifest specific to the release has been created and uploaded to the release artifacts page.\nSee [releases/download/v0.4.0/system-upgrade-controller.yaml](https://github.com/rancher/system-upgrade-controller/releases/download/v0.4.0/system-upgrade-controller.yaml)\n\nBut in the time-honored tradition of `curl ${script} | sudo sh -` here is a nice one-liner:\n\n```shell script\n# Y.O.L.O.\nkubectl apply -k github.com/rancher/system-upgrade-controller\n```\n\n## API Documentation\n\nAutogenerated API docs for `upgrade.cattle.io/v1 Plan` are available at [doc/plan.md](doc/plan.md#Plan)\n\n### Example Plans\n\n- [examples/k3s-upgrade.yaml](examples/k3s-upgrade.yaml)\n  - Demonstrates upgrading k3s itself.\n- [examples/ubuntu/bionic.yaml](examples/ubuntu/bionic.yaml)\n  - Demonstrates upgrading, apt-get style, arbitrary packages at pinned versions.\n- [examples/ubuntu/bionic/linux-kernel-aws.yaml](examples/ubuntu/bionic/linux-kernel-aws.yaml)\n  - Demonstrates upgrading the kernel on Ubuntu 18.04 EC2 instances on AWS.\n- [examples/ubuntu/bionic/linux-kernel-virtual-hwe-18.04.yaml](examples/ubuntu/bionic/linux-kernel-virtual-hwe-18.04.yaml)\n  - Demonstrates upgrading the kernel on Ubuntu 18.04 (to the HWE version) on generic virtual machines.\n\n\nBelow is an example Plan developed for [k3OS](https://github.com/rancher/k3os) that implements something like an\n`rsync` of content from the container image to the host, preceded by a remount if necessary, immediately followed by a reboot.\n\n```\n---\napiVersion: upgrade.cattle.io/v1\nkind: Plan\n\nmetadata:\n  # This `name` should be short but descriptive.\n  name: k3os-latest\n\n  # The same `namespace` as is used for the system-upgrade-controller Deployment.\n  namespace: k3os-system\n\nspec:\n  # The maximum number of concurrent nodes to apply this update on.\n  concurrency: 1\n\n  # The value for `channel` is assumed to be a URL that returns HTTP 302 with the last path element of the value\n  # returned in the Location header assumed to be an image tag (after munging \"+\" to \"-\").\n  channel: https://github.com/rancher/k3os/releases/latest\n\n  # Providing a value for `version` will prevent polling/resolution of the `channel` if specified.\n  version: v0.10.0\n\n  # Select which nodes this plan can be applied to.\n  nodeSelector:\n    matchExpressions:\n      # This limits application of this upgrade only to nodes that have opted in by applying this label.\n      # Additionally, a value of `disabled` for this label on a node will cause the controller to skip over the node.\n      # NOTICE THAT THE NAME PORTION OF THIS LABEL MATCHES THE PLAN NAME. This is related to the fact that the\n      # system-upgrade-controller will tag the node with this very label having the value of the applied plan.status.latestHash.\n      - {key: plan.upgrade.cattle.io/k3os-latest, operator: Exists}\n      # This label is set by k3OS, therefore a node without it should not apply this upgrade.\n      - {key: k3os.io/mode, operator: Exists}\n      # Additionally, do not attempt to upgrade nodes booted from \"live\" CDROM.\n      - {key: k3os.io/mode, operator: NotIn, values: [\"live\"]}\n\n  # The service account for the pod to use. As with normal pods, if not specified the `default` service account from the namespace will be assigned.\n  # See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/\n  serviceAccountName: k3os-upgrade\n\n  # Specify which node taints should be tolerated by pods applying the upgrade.\n  # Anything specified here is appended to the default of:\n  # - {key: node.kubernetes.io/unschedulable, effect: NoSchedule, operator: Exists}\n  tolerations:\n  - {key: kubernetes.io/arch, effect: NoSchedule, operator: Equal, value: amd64}\n  - {key: kubernetes.io/arch, effect: NoSchedule, operator: Equal, value: arm64}\n  - {key: kubernetes.io/arch, effect: NoSchedule, operator: Equal, value: s390x}\n\n  # The prepare init container, if specified, is run before cordon/drain which is run before the upgrade container.\n  # Shares the same format as the `upgrade` container.\n  prepare:\n    # If not present, the tag portion of the image will be the value from `.status.latestVersion` a.k.a. the resolved version for this plan.\n    image: alpine:3.18\n    command: [sh, -c]\n    args: [\"echo '### ENV ###'; env | sort; echo '### RUN ###'; find /run/system-upgrade | sort\"]\n\n  # If left unspecified, no drain will be performed.\n  # See:\n  # - https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/\n  # - https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain\n  drain:\n    # deleteLocalData: true  # default\n    # ignoreDaemonSets: true # default\n    force: true\n    # Use `disableEviction == true` and/or `skipWaitForDeleteTimeout \u003e 0` to prevent upgrades from hanging on small clusters.\n    # disableEviction: false # default, only available with kubectl \u003e= 1.18\n    # skipWaitForDeleteTimeout: 0 # default, only available with kubectl \u003e= 1.18\n\n  # If `drain` is specified, the value for `cordon` is ignored.\n  # If neither `drain` nor `cordon` are specified and the node is marked as `schedulable=false` it will not be marked as `schedulable=true` when the apply job completes.\n  cordon: true\n\n  upgrade:\n    # If not present, the tag portion of the image will be the value from `.status.latestVersion` a.k.a. the resolved version for this plan.\n    image: rancher/k3os\n    command: [k3os, --debug]\n    # It is safe to specify `--kernel` on overlay installations as the destination path will not exist and so the\n    # upgrade of the kernel component will be skipped (with a warning in the log).\n    args:\n      - upgrade\n      - --kernel\n      - --rootfs\n      - --remount\n      - --sync\n      - --reboot\n      - --lock-file=/host/run/k3os/upgrade.lock\n      - --source=/k3os/system\n      - --destination=/host/k3os/system\n```\n\n## Building\n\n```shell script\nmake\n```\n\n## Running\n\nUse `./bin/system-upgrade-controller`.\n\nAlso see [`manifests/system-upgrade-controller.yaml`](manifests/system-upgrade-controller.yaml) that spells out what a\n\"typical\" deployment might look like with default environment variables that parameterize various operational aspects\nof the controller and the resources spawned by it.\n\n## Testing\n\nIntegration tests are bundled as a [Sonobuoy plugin](https://sonobuoy.io/docs/v0.19.0/plugins/) that expects to be run within a pod.\nTo verify locally:\n\n```shell script\nmake e2e\n```\n\nThis will, via Dapper, stand up a local cluster (using docker-compose) and then run the Sonobuoy plugin against/within it.\nThe Sonobuoy results are parsed and a `Status: passed` results in a clean exit, whereas `Status: failed` exits non-zero.\n\nAlternatively, if you have a working cluster and Sonobuoy installation, provided you've pushed the images (consider building with\nsomething like `make REPO=dweomer TAG=dev`), then you can run the e2e tests thusly:\n\n```shell script\nsonobuoy run --plugin dist/artifacts/system-upgrade-controller-e2e-tests.yaml --wait\nsonobuoy results $(sonobuoy retrieve)\n```\n\n## License\nCopyright (c) 2019-2022 [Rancher Labs, Inc.](http://rancher.com)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Francher%2Fsystem-upgrade-controller","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Francher%2Fsystem-upgrade-controller","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Francher%2Fsystem-upgrade-controller/lists"}