{"id":15222289,"url":"https://github.com/googlecloudplatform/gke-rolling-updates-demo","last_synced_at":"2025-10-20T01:30:59.133Z","repository":{"id":54321142,"uuid":"142069768","full_name":"GoogleCloudPlatform/gke-rolling-updates-demo","owner":"GoogleCloudPlatform","description":"This project demonstrates a different upgrade procedures best suited for clusters containing stateless and stateful workloads. You will perform the upgrades in two stages. First, the control plane is updated, then node pools are upgraded.","archived":false,"fork":false,"pushed_at":"2023-12-14T22:54:31.000Z","size":3921,"stargazers_count":52,"open_issues_count":10,"forks_count":34,"subscribers_count":29,"default_branch":"master","last_synced_at":"2024-12-18T08:40:51.795Z","etag":null,"topics":["gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","updates","upgrades"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-07-23T21:10:19.000Z","updated_at":"2024-01-10T09:20:52.000Z","dependencies_parsed_at":"2023-01-18T22:15:37.082Z","dependency_job_id":"1961caac-918e-4bc4-beb0-ceb6e4dce096","html_url":"https://github.com/GoogleCloudPlatform/gke-rolling-updates-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-rolling-updates-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-rolling-updates-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-rolling-updates-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-rolling-updates-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/gke-rolling-updates-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237243005,"owners_count":19278060,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","updates","upgrades"],"created_at":"2024-09-28T15:11:26.735Z","updated_at":"2025-10-20T01:30:53.265Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kubernetes Engine Rolling Upgrades\n\nKubernetes Engine is a managed service that provides fully automated\nupgrades to keep clusters up to date with the latest Kubernetes versions and\nfeatures.  This managed service includes the control plane - API Server,\nWorkload Controllers, and etcd storage back-end - at no cost to the user.\n\nWorker nodes are organized in \"Node Pools\" which can take automated or manual\nversion upgrades.  When you choose manual upgrades of Node Pools, you\nhave several choices for upgrade methodologies.  This repository illustrates\nthree different upgrade strategies, discusses their trade-offs, and provides\ndemos of each.\n\n## Table of Contents\n\n\u003c!--ts--\u003e\n  * [Versions](#versions)\n  * [Upgrade Paths](#upgrade-paths)\n  * [Automated vs Manual Upgrades](#automated-vs-manual-upgrades)\n    * [Automated Upgrades](#automated-upgrades)\n    * [Manual Upgrades](#manual-upgrades)\n  * [Regional vs Multi-Zone Clusters](#regional-vs-multi-zone-clusters)\n  * [Highly Available Application Architecture](#highly-available-application-architecture)\n  * [Upgrade Strategies](#upgrade-strategies)\n    * [In-Place Rolling Upgrade](#in-place-rolling-upgrade)\n      * [\u003ca href=\"./in-place-rolling-upgrade\"\u003eIn Place Rolling Upgrade\u003c/a\u003e Example](#in-place-rolling-upgrade-example)\n    * [Expand and Contract Upgrade](#expand-and-contract-upgrade)\n      * [\u003ca href=\"./expand-contract-upgrade\"\u003eExpand and Contract Upgrade\u003c/a\u003e Example](#expand-and-contract-upgrade-example)\n    * [Blue/Green Upgrade](#bluegreen-upgrade)\n      * [\u003ca href=\"./blue-green-upgrade\"\u003eBlue/Green Upgrade\u003c/a\u003e Example](#bluegreen-upgrade-example)\n  * [Downgrades](#downgrades)\n  * [Kubernetes Engine and Change Control](#kubernetes-engine-and-change-control)\n  * [Relevant Material](#relevant-material)\n\u003c!--te--\u003e\n\n## Versions\n\nA Kubernetes Engine version number represents an immutable collection of\nKubernetes components (api server, workload controllers, kube-proxy, kubelet,\netc), the container runtime, and the host VM operating system.\n\nThe version numbers are based on the Open Source Kubernetes [Semantic Versioning](https://semver.org)\nformat.  A Kubernetes Engine version will take the form of `X.Y.Z-gke.N`, where:\n*   `X` - Kubernetes major version\n*   `Y` - Kubernetes minor version\n*   `Z` - Kubernetes patch version\n*   `N` - Kubernetes Engine patch version\n\nThe `X.Y.Z` portion corresponds directly with the Open Source Kubernetes\nversion of the same number and is fully compatible.  The various Kubernetes\nEngine patch versions (`N`) can contain bug fixes and security patches for the\nKubernetes components, the container runtime, and/or the VM Host OS.\n\nNot all Kubernetes versions are available in Kubernetes Engine.  Generally,\nmultiple patch\nversions of the 3 most recent minor versions are available.  Within minor\nversions, patch versions are turned down and are no longer available for new\nclusters or upgrades as newer versions are introduced.  Kubernetes Engine\nversions are also removed when bugs and/or security vulnerabilities are\ndiscovered.\n\nWith this command, you can find the currently available Kubernetes Engine\nversions new clusters and upgrades:\n\n```\ngcloud container get-server-config [--region \u003cyour-region\u003e] [--zone \u003cyour-zone\u003e]\n```\n\n## Upgrade Paths\n\nKubernetes Engine supports upgrading from one minor version to the next and any\npatch version within a minor version.  Some example version upgrades:\n```\n1.8.8-gke.0 -\u003e 1.8.12-gke.1\n1.8.8-gke.0 -\u003e 1.9.7-gke.3\n1.9.7-gke.3 -\u003e 1.10.4-gke.2\n```\nInvalid version upgrades:\n```\n1.8.8-gke.0 -\u003e 1.10.4-gke.0\n1.8.12-gke.1 -\u003e 1.10.4-gke.2\n```\nAn error will result if you try to make an invalid upgrade.\n\n## Automated vs Manual Upgrades\n\n##### Automated Upgrades\n\nKubernetes Engine will automatically upgrade the control plane, you can not opt-\nout.  You may opt-in to automatic upgrades for Node Pools as well but this\nfunctionality is disabled by default.  When a Node Pool is configured for\nautomatic updates, Kubernetes Engine will upgrade the Node Pool each time it\ndetects a difference between the control plane and Node Pool versions.\n\nThe control plane upgrade schedule is posted on the [Kubernetes Engine Release\nNotes](https://cloud.google.com/kubernetes-engine/release-notes) page.  You can also designate a weekly maintenance window to gain\npredictability around the timing of automated upgrades.  Kubernetes Engine,\nhowever, may still make upgrades outside the maintenance window for unplanned,\nemergency upgrades.\n\nThough the patch version upgrades will proceed fairly frequently, upgrades to\nminor versions will only occur when a specific minor version is being turned\ndown, or going out of support.  At that time, the control plane will be\nupgraded to the next minor version.\n\nWhen configured for automatic upgrades, you may still manually initiate an\nupgrade at any time, provided:\n*   A current upgrade is not already in progress\n*   A Node Pool version will not be advanced beyond the control plane version\n\n##### Manual Upgrades\n\nThough the control plane is upgraded automatically, you may still take a manual\napproach to keeping it up to date.  Automated control plane upgrades are\nannounced on the [Kubernetes Engine Release Notes](https://cloud.google.com/kubernetes-engine/release-notes) page.  You\ncan pre-empt this schedule by manually initiating the upgrade prior to the\nscheduled automated upgrade.\n\nAs most changes in the Kubernetes API functionality happen between minor\nversions, many operators will have less concern about patch version upgrades and\nallow them to proceed automatically but make planned and deliberate upgrades to\nnew minor versions.\n\nFor Node Pools, manual upgrades are the default configuration but the pools will\nstill be upgraded when the specific minor version is turned down and removed\nfrom Kubernetes Engine.\n\nIf you choose to adopt a manual approach to node pool upgrades, you must keep\nan eye on minor version deprecations published in the [Kubernetes Engine Release\nNotes](https://cloud.google.com/kubernetes-engine/release-notes) to ensure that you initiate upgrades before the scheduled upgrade.\n\n## Regional vs Multi-Zone Clusters\n\nRegional cluster has multi-node highly available control plane.  This allows\nzero-downtime upgrades for control plane and five 9s of availability.  Three\ndifferent zones are selected automatically at cluster creation time and one\ncontrol plane node is run in each.  The Node Pools are also spread out among\nthe three zones.\n\nMulti-zone clusters, also called \"zonal clusters\", have a single control plane\nnode located in one zone.  The api server will become unavailable during version\nupgrades resulting in Kubernetes API unavailability during that time.  The\nNode Pool will default to the same single zone but can be expanded to multiple\nzones in the same region.\n\nIn both cases, the cost for running the control plane is the same for the end\nuser - $0.\n\n## Highly Available Application Architecture\n\nHighly Available (HA) applications can continue running during planned and\nunplanned interruptions.  Specific knowledge of how your application operates\nand performs is necessary to ensure your application will be highly available\nwhile running within Kubernetes Engine.\n\nThere are specific Kubernetes features to assist in building HA applications.\nThey include but are not limited to:\n*   Having enough replicas of an application to allow continued functioning when\n    one or more instances are lost, restarted, or rescheduled on a different\n    node.\n*   Using `anti-affinity` annotations to avoid multiple replicas of the same\n    application being scheduled on the same node.\n*   Using `PodDisruptionBudgets` to make sure the appropriate number of\n    application replicas are left running when pods are being rescheduled.\n*   Using `readinessProbes` to ensure that after rescheduled application\n    instances are started they are fully initialized and operational before\n    being marked \"healthy\".\n\n## Upgrade Strategies\n\nThere are several strategies available when upgrading Kubernetes Engine\nNode Pools.  They range from fast and fully automated, to slow and deliberate.\n\nThe types of applications you have running in the cluster should dictate your\nchoice of upgrade strategy.  The root of this repository has a common properties\nfile called `env`.  Make a copy and update with values appropriate for your\ntesting.\n```console\ncp env .env\n```\n\n### In-Place Rolling Upgrade\n\nThis is the same procedure used when automated Node Pool upgrades is enabled.\nEach node is cordoned, drained, terminated, and finally replaced with a new\nnode running the new version.  This works well in large clusters that tend\nto have significant resource head room.\n\n**[In Place Rolling Upgrade](in-place-rolling-upgrade) Example**\n\n### Expand and Contract Upgrade\n\nThe Expand and Contract Upgrade is a further refinement of the Rolling Upgrade.\nMore capacity is added to the Node Pool prior to the rolling upgrade and then\nremoved at the end of the upgrade.  This is a good choice for smaller clusters\nthat don't have significant headroom, or clusters with Stateful workloads using\nzone specific disks.\n\n**[Expand and Contract Upgrade](expand-contract-upgrade) Example**\n\n### Blue/Green Upgrade\n\nThe Blue/Green, or \"Lift and Shift\", Upgrade gives the operator the most\ncontrol over workload migrations.  A duplicate Node Pool running the new version\nis run simultaneously with the Node Pool running the old version.  The nodes\nin the old pool are cordoned then the operator can migrate pods to the nodes\nrunning the new version in the order and timing deemed appropriate.\n\n**[Blue/Green Upgrade](blue-green-upgrade) Example**\n\n## Downgrades\n\n**Downgrading the control plane is not possible.**  If you find a unexpected\nbehaviors during a Node Pool upgrade, you can abort and roll back the Node Pool\nto the previous version.\n\n## Kubernetes Engine and Change Control\n\nChange Control procedures generally require making changes during specified\nmaintenance windows and articulating back-out plans.  Kubernetes Engine can\nhelp organizations achieve these requirements with specific configurations and\nplanning.\n\n*   Plan and execute Control Plane upgrades prior to the automated upgrade\n    schedule\n*   Do not configure Node Pools for automatic upgrades and instead plan and\n    execute Node Pool upgrades when necessary.\n*   Since control plane downgrades are not possible, create/maintain a\n    development or staging cluster with identical application workload types to\n    test fully upgrades prior to applying them in your production environments.\n\n## Manual upgrade process\n\nManual upgrades require different considerations and planning depending on\nthe type of upgrade.  The following decision tree illustrates a high-level\noverview of a manual upgrade of a Kubernetes Engine cluster.\n\n![manual-upgrade](images/manual-upgrade.png)\n\n## Relevant Material\n\n*   [Kubernetes Engine Release Notes](https://cloud.google.com/kubernetes-engine/release-notes)\n*   [Kubernetes Engine Maintenance Window](https://cloud.google.com/kubernetes-engine/docs/how-to/maintenance-window)\n*   [Multi-Zone and Regional Clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/multi-zone-and-regional-clusters)\n*   [Affinity and Anti-Affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity)\n*   [readinessProbes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes)\n\n\n**This is not an officially supported Google product**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-rolling-updates-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fgke-rolling-updates-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-rolling-updates-demo/lists"}