{"id":16513906,"url":"https://github.com/twin/aws-eks-asg-rolling-update-handler","last_synced_at":"2025-05-08T19:50:40.184Z","repository":{"id":38848999,"uuid":"260529912","full_name":"TwiN/aws-eks-asg-rolling-update-handler","owner":"TwiN","description":"Handles rolling upgrades for AWS ASGs on EKS","archived":false,"fork":false,"pushed_at":"2024-10-28T04:16:26.000Z","size":15917,"stargazers_count":44,"open_issues_count":4,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-09T18:51:53.866Z","etag":null,"topics":["aws","controller","eks","go","golang","handler","kubernetes","launch-template","rolling-update","rolling-upgrade"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TwiN.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["TwiN"]}},"created_at":"2020-05-01T18:25:25.000Z","updated_at":"2024-10-28T04:16:28.000Z","dependencies_parsed_at":"2023-12-26T05:28:32.289Z","dependency_job_id":"9b0423ec-f0df-4b7a-9d8d-7969210ab610","html_url":"https://github.com/TwiN/aws-eks-asg-rolling-update-handler","commit_stats":{"total_commits":264,"total_committers":9,"mean_commits":"29.333333333333332","dds":0.5681818181818181,"last_synced_commit":"5a8c952372cb8717310ef8a3d13603f81c247ca0"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TwiN%2Faws-eks-asg-rolling-update-handler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TwiN%2Faws-eks-asg-rolling-update-handler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TwiN%2Faws-eks-asg-rolling-update-handler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TwiN%2Faws-eks-asg-rolling-update-handler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TwiN","download_url":"https://codeload.github.com/TwiN/aws-eks-asg-rolling-update-handler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225110575,"owners_count":17422411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","controller","eks","go","golang","handler","kubernetes","launch-template","rolling-update","rolling-upgrade"],"created_at":"2024-10-11T16:10:49.152Z","updated_at":"2025-05-08T19:50:40.145Z","avatar_url":"https://github.com/TwiN.png","language":"Go","funding_links":["https://github.com/sponsors/TwiN"],"categories":[],"sub_categories":[],"readme":"# aws-eks-asg-rolling-update-handler\n\n![test](https://github.com/TwiN/aws-eks-asg-rolling-update-handler/actions/workflows/test.yml/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/TwiN/aws-eks-asg-rolling-update-handler)](https://goreportcard.com/report/github.com/TwiN/aws-eks-asg-rolling-update-handler)\n[![Docker pulls](https://img.shields.io/docker/pulls/twinproduction/aws-eks-asg-rolling-update-handler.svg)](https://cloud.docker.com/repository/docker/twinproduction/aws-eks-asg-rolling-update-handler)\n\nThis application handles rolling upgrades for AWS ASGs for EKS by replacing outdated nodes by new nodes.\nOutdated nodes are defined as nodes whose current configuration does not match its ASG's current launch \ntemplate version or launch configuration.\n\nInspired by aws-asg-roller, this application only has one purpose: Scale down outdated nodes gracefully.\n\nUnlike aws-asg-roller, it will not attempt to control the amount of nodes at all; it will scale up enough new nodes\nto move the pods from the old nodes to the new nodes, and then evict the old nodes. \n\nIt will not adjust the desired size back to its initial desired size like aws-asg-roller does, it will simply leave\neverything else up to cluster-autoscaler.\n\nNote that unlike other solutions, this application actually uses the resources to determine how many instances should \nbe spun up before draining the old nodes. This is much better, because simply using the initial number of instances is \ncompletely useless in the event that the ASG's update on the launch configuration/template is a change of instance type.\n\n\n## Behavior\n\nOn interval, this application:\n1. Iterates over each ASG discovered by the `CLUSTER_NAME`, `AUTODISCOVERY_TAGS` environment variables or the ones defined in the `AUTO_SCALING_GROUP_NAMES` environment variable, in that order.\n2. Iterates over each instance of each ASG\n3. Checks if there's any instance with an outdated launch template version\n4. **If ASG uses MixedInstancesPolicy**, checks if there's any instances with an instance type that isn't part of the list of instance type overrides\n5. Checks if there's any instance with an outdated launch configuration\n6. If any of the conditions defined in the step 3, 4 or 5 are met for any instance, begin the rolling update process for that instance\n\nThe steps of each action are persisted directly on the old nodes via annotations (i.e. when the old node starts rolling out, gets drained, and gets scheduled for termination).\nTherefore, this application will not run into any issues if it is restarted, rescheduled or stopped at any point in time.\n\n\n**NOTE**: Ensure that your PodDisruptionBudgets - if you have any - are properly configured. This usually means having at least 1 allowed disruption at all time (i.e. at least `minAvailable: 1` with at least 2 replicas OR `maxUnavailable: 1`)\n\n\n## Usage\n\n| Environment variable                 | Description                                                                                                                                                                                                                                                                  | Required | Default     |\n|:-------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------|:------------|\n| CLUSTER_NAME                         | Name of the eks-cluster, used in place of `AUTODISCOVERRY_TAGS` and `AUTO_SCALING_GROUP_NAMES`. Checks for `k8s.io/cluster-autoscaler/\u003cCLUSTER_NAME\u003e: owned` and `k8s.io/cluster-autoscaler/enabled: true` tags on ASG                                                       | yes      | `\"\"`        |\n| AUTODISCOVERY_TAGS                   | Comma separated key value string with tags to autodiscover ASGs, used in place of `CLUSTER_NAME` and `AUTO_SCALING_GROUP_NAMES`.                                                                                                                                             | yes      | `\"\"`        |\n| AUTO_SCALING_GROUP_NAMES             | Comma-separated list of ASGs, CLUSTER_NAME takes priority.                                                                                                                                                                                                                   | yes      | `\"\"`        |\n| IGNORE_DAEMON_SETS                   | Whether to ignore DaemonSets when draining the nodes                                                                                                                                                                                                                         | no       | `true`      |\n| DELETE_EMPTY_DIR_DATA                | Whether to delete empty dir data when draining the nodes                                                                                                                                                                                                                     | no       | `true`      |\n| AWS_REGION                           | Self-explanatory                                                                                                                                                                                                                                                             | no       | `us-west-2` |\n| ENVIRONMENT                          | If set to `dev`, will try to create the Kubernetes client using your local kubeconfig. Any other values will use the in-cluster configuration                                                                                                                                | no       | `\"\"`        |\n| EXECUTION_INTERVAL                   | Duration to sleep between each execution in seconds                                                                                                                                                                                                                          | no       | `20`        |\n| EXECUTION_TIMEOUT                    | Maximum execution duration before timing out in seconds                                                                                                                                                                                                                      | no       | `900`       |\n| POD_TERMINATION_GRACE_PERIOD         | How long to wait for a pod to terminate in seconds; 0 means \"delete immediately\"; set to a negative value to use the pod's terminationGracePeriodSeconds.                                                                                                                    | no       | `-1`        |\n| METRICS_PORT                         | Port to bind metrics server to                                                                                                                                                                                                                                               | no       | `8080`      |\n| METRICS                              | Expose metrics in Prometheus format at `:${METRICS_PORT}/metrics`                                                                                                                                                                                                            | no       | `\"\"`        | \n| SLOW_MODE                            | If enabled, every time a node is terminated during an execution, the current execution will stop rather than continuing to the next ASG                                                                                                                                      | no       | `false`     |\n| EAGER_CORDONING                      | If enabled, all outdated nodes will get cordoned before any rolling update action. The default mode is to cordon a node just before draining it. See [#41](https://github.com/TwiN/aws-eks-asg-rolling-update-handler/issues/41) for possible consequences of enabling this. | no       | `false`     |\n| EXCLUDE_FROM_EXTERNAL_LOAD_BALANCERS | If enabled, node label `node.kubernetes.io/exclude-from-external-load-balancers=true` will be added to nodes before draining. See [#131](https://github.com/TwiN/aws-eks-asg-rolling-update-handler/pull/131) for more information                                           | no       | `false`     |\n\n**NOTE:** Only one of `CLUSTER_NAME`, `AUTODISCOVERY_TAGS` or `AUTO_SCALING_GROUP_NAMES` must be set.\n\n\n## Metrics\n\n| Metric name                                | Metric type | Labels       | Description                           |\n|--------------------------------------------|-------------|--------------|---------------------------------------|\n| rolling_update_handler_node_groups         | Gauge       |              | Node groups managed by the handler    |\n| rolling_update_handler_outdated_nodes      | Gauge       | `node_group` | The number of outdated nodes          |\n| rolling_update_handler_updated_nodes       | Gauge       | `node_group` | The number of updated nodes           |\n| rolling_update_handler_scaled_up_nodes     | Counter     | `node_group` | The total number of nodes scaled up   |\n| rolling_update_handler_scaled_down_nodes   | Counter     | `node_group` | The total number of nodes scaled down |\n| rolling_update_handler_drained_nodes_total | Counter     | `node_group` | The total number of drained nodes     |\n| rolling_update_handler_errors              | Counter     |              | The total number of errors            |\n\n\n## Permissions\n\nTo function properly, this application requires the following permissions on AWS:\n- autoscaling:DescribeAutoScalingGroups\n- autoscaling:DescribeAutoScalingInstances\n- autoscaling:DescribeLaunchConfigurations\n- autoscaling:SetDesiredCapacity\n- autoscaling:TerminateInstanceInAutoScalingGroup\n- autoscaling:UpdateAutoScalingGroup\n- ec2:DescribeLaunchTemplates\n- ec2:DescribeInstances\n\n\n## Deploying on Kubernetes\n\n```yaml\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: aws-eks-asg-rolling-update-handler\n  namespace: kube-system\n  labels:\n    app: aws-eks-asg-rolling-update-handler\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: aws-eks-asg-rolling-update-handler\n  labels:\n    app: aws-eks-asg-rolling-update-handler\nrules:\n  - apiGroups:\n      - \"*\"\n    resources:\n      - \"*\"\n    verbs:\n      - get\n      - list\n      - watch\n  - apiGroups:\n      - \"*\"\n    resources:\n      - nodes\n    verbs:\n      - get\n      - list\n      - watch\n      - update\n      - patch\n  - apiGroups:\n      - \"*\"\n    resources:\n      - pods/eviction\n    verbs:\n      - get\n      - list\n      - create\n  - apiGroups:\n      - \"*\"\n    resources:\n      - pods\n    verbs:\n      - get\n      - list\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: aws-eks-asg-rolling-update-handler\n  labels:\n    app: aws-eks-asg-rolling-update-handler\nroleRef:\n  kind: ClusterRole\n  name: aws-eks-asg-rolling-update-handler\n  apiGroup: rbac.authorization.k8s.io\nsubjects:\n  - kind: ServiceAccount\n    name: aws-eks-asg-rolling-update-handler\n    namespace: kube-system\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: aws-eks-asg-rolling-update-handler\n  namespace: kube-system\n  labels:\n    app: aws-eks-asg-rolling-update-handler\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: aws-eks-asg-rolling-update-handler\n  template:\n    metadata:\n      labels:\n        app: aws-eks-asg-rolling-update-handler\n    spec:\n      automountServiceAccountToken: true\n      serviceAccountName: aws-eks-asg-rolling-update-handler\n      restartPolicy: Always\n      dnsPolicy: Default\n      containers:\n        - name: aws-eks-asg-rolling-update-handler\n          image: twinproduction/aws-eks-asg-rolling-update-handler\n          imagePullPolicy: Always\n          env:\n            - name: AUTO_SCALING_GROUP_NAMES\n              value: \"asg-1,asg-2,asg-3\" # REPLACE THESE VALUES FOR THE NAMES OF THE ASGs\n```\n\n\n## Deploying with Helm\n\nFor the chart associated to this project, see [TwiN/helm-charts](https://github.com/TwiN/helm-charts):\n```sh\nhelm repo add twin https://twin.github.io/helm-charts\nhelm repo update\nhelm install aws-eks-asg-rolling-update-handler twin/aws-eks-asg-rolling-update-handler\n```\n\n\n## Developing\n\nTo run the application locally, make sure your local kubeconfig file is configured properly (i.e. you can use kubectl).\n\nOnce you've done that, set the local environment variable `ENVIRONMENT` to `dev` and `AUTO_SCALING_GROUP_NAMES` \nto a comma-separated list of auto scaling group names.\n\nYour local aws credentials must also be valid (i.e. you can use `awscli`)\n\n\n## Special thanks\n\nI had originally worked on [deitch/aws-asg-roller](https://github.com/deitch/aws-asg-roller), but due to the numerous conflicts it had with cluster-autoscaler, \nI decided to make a project that heavily relies on cluster-autoscaler rather than simply coexist with it, with a much bigger emphasis on maintaining \nhigh availability during rolling upgrades.\n\nIn any case, this project was inspired by aws-asg-roller and the code for comparing launch template versions also comes from there, hence why this special thanks section exists.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwin%2Faws-eks-asg-rolling-update-handler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwin%2Faws-eks-asg-rolling-update-handler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwin%2Faws-eks-asg-rolling-update-handler/lists"}