{"id":13618406,"url":"https://github.com/stefanprodan/eks-hpa-profile","last_synced_at":"2025-08-12T13:42:13.764Z","repository":{"id":50141126,"uuid":"226285343","full_name":"stefanprodan/eks-hpa-profile","owner":"stefanprodan","description":"An eksctl gitops profile for autoscaling with Prometheus metrics on Amazon EKS on AWS Fargate","archived":false,"fork":false,"pushed_at":"2021-06-02T19:29:38.000Z","size":68,"stargazers_count":36,"open_issues_count":5,"forks_count":16,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T08:43:46.394Z","etag":null,"topics":["auto-scaling","aws-appmesh","eks","fargate","gitops","kubernetes","metrics-server","prometheus"],"latest_commit_sha":null,"homepage":"https://aws.amazon.com/blogs/containers/autoscaling-eks-on-fargate-with-custom-metrics/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanprodan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-06T08:50:46.000Z","updated_at":"2024-09-13T04:13:13.000Z","dependencies_parsed_at":"2022-08-30T07:50:16.666Z","dependency_job_id":null,"html_url":"https://github.com/stefanprodan/eks-hpa-profile","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Feks-hpa-profile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Feks-hpa-profile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Feks-hpa-profile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanprodan%2Feks-hpa-profile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanprodan","download_url":"https://codeload.github.com/stefanprodan/eks-hpa-profile/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252852544,"owners_count":21814377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-scaling","aws-appmesh","eks","fargate","gitops","kubernetes","metrics-server","prometheus"],"created_at":"2024-08-01T20:02:00.741Z","updated_at":"2025-05-07T09:42:15.465Z","avatar_url":"https://github.com/stefanprodan.png","language":null,"funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# eks-hpa-profile\n\nThis repo is an [eksctl GitOps profile](https://eksctl.io/usage/experimental/gitops-flux/)\nfor configuring HPA with metrics provided by Prometheus to automatically scale pods running on EKS on Fargate. \n\n![](docs/fargate-eks-hpa.png)\n\n### Create an EKS cluster\n\nCreate an EKS cluster with two EC2 managed nodes and a Fargate profile:\n\n```sh\ncat \u003c\u003c EOF | eksctl create cluster -f -\napiVersion: eksctl.io/v1alpha5\nkind: ClusterConfig\nmetadata:\n  name: eks-fargate-hpa\n  region: eu-west-1\n\nmanagedNodeGroups:\n  - name: default\n    instanceType: m5.large\n    desiredCapacity: 2\n    volumeSize: 120\n    iam:\n      withAddonPolicies:\n        appMesh: true\n        albIngress: true\n\nfargateProfiles:\n  - name: default\n    selectors:\n      - namespace: demo\n        labels:\n          scheduler: fargate\nEOF\n```\n\nYou'll use the managed nodes for cluster add-ons(CoreDNS, KubeProxy) and for the HPA metrics add-ons:\n* [Prometheus](https://github.com/stefanprodan/eks-hpa-profile/blob/master/monitoring-system/prometheus.yaml) - scrapes pods and stores metrics\n* [Prometheus metrics adapter](https://github.com/stefanprodan/eks-hpa-profile/blob/master/monitoring-system/prometheus-adapter.yaml) - queries Prometheus and exposes metrics for the Kubernetes custom metrics API\n* [Metrics server](https://github.com/stefanprodan/eks-hpa-profile/blob/master/monitoring-system/metrics-server.yaml) - collects pods CPU and memory usage and exposes metrics for the Kubernetes resource metrics API\n\nYou'll use Fargate for the demo application [podinfo](https://github.com/stefanprodan/eks-hpa-profile/tree/master/demo/podinfo),\nnote that only the pods deployed in the `demo` namespace with a `scheduler: fargate` label will be running on Fargate.\n\n### Create a GitHub repository\n\nTo configure HPA for Fargate you'll be using an eksctl GitOps [profile](https://eksctl.io/gitops-quickstart/setup-gitops/). \nA Profile allows you to create a specific Kubernetes application platform tailored for a specific use case.\n\nCreate a GitHub [repository](https://github.com/new) and clone it locally.\nReplace `GH_USER/GH_REPO` value with your GitHub username and new repo.\nUse these variables to clone your repo and setup GitOps for your cluster.\n\n```sh\nexport GH_USER=YOUR_GITHUB_USERNAME\nexport GH_REPO=YOUR_GITHUB_REPOSITORY\n\ngit clone https://github.com/${GH_USER}/${GH_REPO}\ncd ${GH_REPO}\n```\n\nRun the eksctl repo command:\n\n```sh\nexport EKSCTL_EXPERIMENTAL=true\n\neksctl enable repo \\\n--cluster=eks-fargate-hpa \\\n--region=eu-west-1 \\\n--git-url=git@github.com:${GH_USER}/${GH_REPO} \\\n--git-user=fluxcd \\\n--git-email=${GH_USER}@users.noreply.github.com\n```\n\nThe command `eksctl enable repo` takes an existing EKS cluster and an empty repository \nand sets up a GitOps pipeline.\n\nAfter the command finishes installing [FluxCD](https://github.com/fluxcd/flux) and [Helm Operator](https://github.com/fluxcd/flux),\nyou will be asked to add Flux's deploy key to your GitHub repository.\n\nCopy the public key and create a deploy key with write access on your GitHub repository.\nGo to `Settings \u003e Deploy keys` click on `Add deploy key`, check `Allow write access`,\npaste the Flux public key and click `Add key`.\n\nOnce that is done, Flux will be able to pick up changes in the repository and deploy them to the cluster.\n\n### Install the metrics add-ons\n\nRun the eksctl profile command:\n\n```sh\neksctl enable profile \\\n--profile-source=https://github.com/stefanprodan/eks-hpa-profile \\\n--cluster=eks-fargate-hpa \\\n--region=eu-west-1 \\\n--git-url=git@github.com:${GH_USER}/${GH_REPO} \\\n--git-user=fluxcd \\\n--git-email=${GH_USER}@users.noreply.github.com\n```\n\nThe command `eksctl enable profile` adds the HPA metrics add-ons and\nthe demo app manifests to the configured repository.\n\nSync your local repository with:\n\n```sh\ngit pull origin master\n```\n\nInstall [fluxctl](https://github.com/fluxcd/flux/releases):\n\n```sh\ncurl -sL https://fluxcd.io/install | sh\nexport PATH=\"$PATH:$HOME/.fluxcd/bin\"\n```\n\nRun the fluxctl sync command to apply the manifests on your cluster:\n\n```sh\nfluxctl sync --k8s-fwd-ns flux\n```\n\nFlux does a git-cluster reconciliation every five minutes,\nthe above command can be used to speed up the synchronization.\n\nList the installed components:\n\n```\n$ kubectl -n monitoring-system get helmreleases\n\nNAME                 RELEASE              STATUS\nmetrics-server       metrics-server       DEPLOYED\nprometheus           prometheus           DEPLOYED\nprometheus-adapter   prometheus-adapter   DEPLOYED\n```\n\n### Install podinfo\n\nYou'll use a Go web app named [podinfo](https://github.com/stefanprodan/podinfo) to try out HPA.\nThe app is instrumented with Prometheus and exposes the `http_requests_total` [counter](https://prometheus.io/docs/concepts/metric_types/#counter).\nThe HPA controller will scale the pods running on Fargate based on the number of requests per second.\n\nInstall podinfo by setting `fluxcd.io/ignore` to `false` in base/demo/namespace.yaml:\n\n```sh\ncat \u003c\u003c EOF | tee base/demo/namespace.yaml\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: demo\n  annotations:\n    fluxcd.io/ignore: \"false\"\nEOF\n```\n\nApply changes via git:\n\n```sh\ngit add -A \u0026\u0026 \\\ngit commit -m \"init demo\" \u0026\u0026 \\\ngit push origin master \u0026\u0026 \\\nfluxctl sync --k8s-fwd-ns flux\n```\n\nWait for Fargate to schedule and start podinfo:\n\n```sh\nwatch kubectl -n demo get po -l scheduler=fargate\n```\n\nWhen podinfo starts, Prometheus will scrape the metrics endpoint and the Prometheus adapter will export the HTTP \nrequests per second metrics to the Kubernetes custom metrics API:\n\n```\n$ watch kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .\n\n{\n  \"kind\": \"APIResourceList\",\n  \"apiVersion\": \"v1\",\n  \"groupVersion\": \"custom.metrics.k8s.io/v1beta1\",\n  \"resources\": [\n    {\n      \"name\": \"namespaces/http_requests_per_second\",\n      \"singularName\": \"\",\n      \"namespaced\": false,\n      \"kind\": \"MetricValueList\",\n      \"verbs\": [\n        \"get\"\n      ]\n    },\n    {\n      \"name\": \"pods/http_requests_per_second\",\n      \"singularName\": \"\",\n      \"namespaced\": true,\n      \"kind\": \"MetricValueList\",\n      \"verbs\": [\n        \"get\"\n      ]\n    }\n  ]\n}\n``` \n\n### Configure autoscaling based on HTTP traffic\n\nTo configure auto-scaling you can set up a HPA definition that uses the `http_requests_per_second` metric. \nThe HPA manifest is in `base/demo/podinfo.hpa.yaml` and it's set to scale up podinfo\nwhen the average req/sec per pod goes over 10:\n\n```yaml\napiVersion: autoscaling/v2beta1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: podinfo\n  namespace: demo\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: podinfo\n  minReplicas: 1\n  maxReplicas: 10\n  metrics:\n    - type: Pods\n      pods:\n        metricName: http_requests_per_second\n        targetAverageValue: 10\n```\n\nNote that the podinfo [deployment manifest](https://github.com/stefanprodan/eks-hpa-profile/blob/master/demo/podinfo/deployment.yaml)\nhas no replicas defined in `deployment.spec.replicas`, the HPA controller updates the number of replicas based on the metric average value.\n\nOnce the metric is available to the metrics API, the HPA controller will display the current value:\n\n```\n$ watch kubectl -n demo get hpa\n\nNAME      REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE\npodinfo   Deployment/podinfo   200m/10   1         10        1          8m58s\n```\n\nThe m in `200m` represents milli-units, 200m means 0.2 req/sec.\nThe traffic is generated by Prometheus that scrapes the `/metrics` endpoint every five seconds.\n\nExec into the loadtester pod with:\n\n```sh\nkubectl -n demo exec -it loadtester-xxxx-xxxx\n```\n\nGenerate traffic with `hey`:\n\n```sh\nhey -z 10m -c 5 -q 5 -disable-keepalive http://podinfo.demo\n```\n\nAfter a few minutes the HPA begins to scale up the deployment:\n\n```\n$ kubectl -n demo describe hpa podinfo\n\nEvents:\n  Type    Reason             Age   From                       Message\n  ----    ------             ----  ----                       -------\n  Normal  SuccessfulRescale  2m    horizontal-pod-autoscaler  New size: 3; reason: pods metric http_requests_per_second above target\n```\n\nAfter the load tests finishes, the HPA down scales the deployment to it's initial replicas:\n\n```\nEvents:\n  Type    Reason             Age   From                       Message\n  ----    ------             ----  ----                       -------\n  Normal  SuccessfulRescale  21s   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target\n```\n\nYou may have noticed that the autoscaler doesn't react immediately to usage spikes.\nBy default the metrics sync happens once every 30 seconds and scaling up/down can only happen\nif there was no rescaling within the last 3-5 minutes.\nIn this way, the HPA prevents rapid execution of conflicting decisions.\n\n### Configure autoscaling based on CPU usage\n\nFor workloads that aren't instrumented with Prometheus, you can use the Kubernetes\n[metrics server](https://github.com/stefanprodan/eks-hpa-profile/blob/master/monitoring-system/metrics-server.yaml)\nand configure auto-scaling based on CPU and/or memory usage.\n\nUpdate the HPA manifest by replacing the HTTP metric with CPU average utilization:\n\n```sh\ncat \u003c\u003c EOF | tee base/demo/podinfo/hpa.yaml\napiVersion: autoscaling/v2beta1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: podinfo\n  namespace: demo\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: podinfo\n  minReplicas: 1\n  maxReplicas: 10\n  metrics:\n    - type: Resource\n        resource:\n          name: cpu\n          targetAverageUtilization: 90\nEOF\n```\n\nApply changes via git:\n\n```sh\ngit add -A \u0026\u0026 \\\ngit commit -m \"cpu hpa\" \u0026\u0026 \\\ngit push origin master \u0026\u0026 \\\nfluxctl sync --k8s-fwd-ns flux\n```\n\nRun a load test to increase CPU usage above 90% to trigger the HPA:\n\n```sh\nhey -z 10m -c 5 -q 5 -m POST -d 'test' -disable-keepalive http://podinfo.demo/token\n```\n\nThe Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data,\nit collects CPU and memory usage for nodes and pods by pooling data from the `kubernetes.summary_api`.\nThe summary API is a memory-efficient API for passing data from Kubelet to the metrics server.\n\n### Configure autoscaling based on App Mesh traffic\n\nOne of the advantages of using a service mesh like App Mesh is the builtin monitoring capability.\nYou don’t have to instrument your web apps in order to monitor the L7 traffic.\n\nThe Envoy sidecar used by App Mesh exposes a counter `envoy_cluster_upstream_rq`, you can configure the\n[Prometheus adapter](https://github.com/stefanprodan/eks-hpa-profile/blob/master/monitoring-system/prometheus-adapter.yaml)\nto transform this metric into req/sec rate with:\n\n```yaml\napiVersion: helm.fluxcd.io/v1\nkind: HelmRelease\nmetadata:\n  name: prometheus-adapter\n  namespace: monitoring-system\nspec:\n  releaseName: prometheus-adapter\n  chart:\n    repository: https://kubernetes-charts.storage.googleapis.com/\n    name: prometheus-adapter\n    version: 1.4.0\n  values:\n    prometheus:\n      url: http://prometheus.monitoring-system\n      port: 9090\n    rules:\n      default: false\n      custom:\n        - seriesQuery: 'envoy_cluster_upstream_rq{kubernetes_namespace!=\"\",kubernetes_pod_name!=\"\"}'\n          resources:\n            overrides:\n              kubernetes_namespace: {resource: \"namespace\"}\n              kubernetes_pod_name: {resource: \"pod\"}\n          name:\n            matches: \"envoy_cluster_upstream_rq\"\n            as: \"appmesh_requests_per_second\"\n          metricsQuery: 'sum(rate(\u003c\u003c.Series\u003e\u003e{\u003c\u003c.LabelMatchers\u003e\u003e}[1m])) by (\u003c\u003c.GroupBy\u003e\u003e)'\n```\n\nNow you can use the `appmesh_requests_per_second` metric in the HPA definition:\n\n```yaml\napiVersion: autoscaling/v2beta1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: podinfo\n  namespace: demo\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: podinfo\n  minReplicas: 1\n  maxReplicas: 10\n  metrics:\n    - type: Pods\n      pods:\n        metricName: appmesh_requests_per_second\n        targetAverageValue: 10\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Feks-hpa-profile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefanprodan%2Feks-hpa-profile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefanprodan%2Feks-hpa-profile/lists"}