{"id":20612862,"url":"https://github.com/k-yomo/elastic-cloud-autoscaler","last_synced_at":"2025-04-15T07:09:29.183Z","repository":{"id":65127307,"uuid":"575199135","full_name":"k-yomo/elastic-cloud-autoscaler","owner":"k-yomo","description":"Elastic Cloud Autoscaler based on CPU util or cron schedules","archived":false,"fork":false,"pushed_at":"2023-11-06T17:13:33.000Z","size":200,"stargazers_count":9,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T07:09:21.760Z","etag":null,"topics":["autoscaler","autoscaling","elasticcloud","elasticsearch","go"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/k-yomo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-12-07T01:09:39.000Z","updated_at":"2024-09-29T01:45:40.000Z","dependencies_parsed_at":"2023-11-07T01:13:10.670Z","dependency_job_id":null,"html_url":"https://github.com/k-yomo/elastic-cloud-autoscaler","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-yomo%2Felastic-cloud-autoscaler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-yomo%2Felastic-cloud-autoscaler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-yomo%2Felastic-cloud-autoscaler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k-yomo%2Felastic-cloud-autoscaler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/k-yomo","download_url":"https://codeload.github.com/k-yomo/elastic-cloud-autoscaler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249023723,"owners_count":21199960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaler","autoscaling","elasticcloud","elasticsearch","go"],"created_at":"2024-11-16T11:08:08.664Z","updated_at":"2025-04-15T07:09:29.166Z","avatar_url":"https://github.com/k-yomo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# elastic-cloud-autoscaler\n\n![License: Apache-2.0](https://img.shields.io/badge/License-MIT-blue.svg)\n[![Test](https://github.com/k-yomo/elastic-cloud-autoscaler/actions/workflows/test.yml/badge.svg)](https://github.com/k-yomo/elastic-cloud-autoscaler/actions/workflows/test.yml)\n[![Codecov](https://codecov.io/gh/k-yomo/elastic-cloud-autoscaler/branch/main/graph/badge.svg?token=P3pNbMGbeN)](https://codecov.io/gh/k-yomo/elastic-cloud-autoscaler)\n[![Go Report Card](https://goreportcard.com/badge/k-yomo/elastic-cloud-autoscaler)](https://goreportcard.com/report/k-yomo/elastic-cloud-autoscaler)\n\nElastic Cloud Autoscaler based on CPU util or cron schedules inspired by [es-operator](https://github.com/zalando-incubator/es-operator).\n\n**⚠️ This library is still experimental, please use at your own risk if you use.**\nI also highly recommend using with `DryRun: true` at first.\n\n## Compatibility\n- Elasticsearch \u003e= 8.x\n\n## Features\nThe autoscaler supports following ways of auto-scaling.\n- CPU utilization based auto-scaling.\n - Autoscaler tries to scale-out/scale-in when average CPU util is higher/lower than the desired CPU utilization throughout the threshold duration\n- Cron schedule based auto-scaling.\n - You can override min/max node num for configured duration with the cron format schedule.\n\n## Configuration\n| Config properties | Type | Required | Description |\n|-----------------------------------------|-----------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| index | string | true | Index to update replicas when scaling out/in |\n| shardsPerNode | int | true | Desired shard count per 1 node. Autoscaler won't scale-in / scale-out to the node count that can't meet this ratio. |\n| defaultMinMemoryGBPerZone | int | true | Default memory min size per zone. Available number is only 64,...(64xN node) |\n| defaultMaxMemoryGBPerZone | int | true | Default memory max size per zone. Available number is only 64,...(64xN node) |\n| autoScaling | object | | |\n| autoScaling.desiredCPUUtilPercent | int | true (in autoScaling) | Desired CPU utilization percent. Autoscaler will change nodes to make CPU utilization closer to the desired CPU utilization. |\n| autoScaling.scaleOutThresholdDuration | time.Duration | | Threshold duration for scale-out. When CPU util is higher than desiredCPUUtilPercent throughout the threshold duration scale-out may happen. |\n| autoScaling.scaleOutCoolDownDuration | time.Duration | | Cool down period for scale-out after the last scaling operation. |\n| autoScaling.scaleInThresholdDuration | time.Duration | | Threshold duration for scale-in. When CPU util is lower than desiredCPUUtilPercent throughout the threshold duration scale-in may happen. |\n| autoScaling.scaleInCoolDownDuration | time.Duration | | Cool down period for scale-in after the last scaling operation |\n| []scheduledScalings | array of object | | |\n| scheduledScalings[i].startCronSchedule | string | true (in scheduledScaling) | Cron format schedule to start the specified min/max size. Default timezone is machine local timezone. If you want to specify, set TZ= prefix (e.g. `TZ=UTC 0 0 0 0 0`). |\n| scheduledScalings[i].duration | time.Duration | true (in scheduledScaling) | Duration to apply above min/max size from startCronSchedule |\n| scheduledScalings[i].minMemoryGBPerZone | int | true (in scheduledScaling) | Min memory size during the specified period. |\n| scheduledScalings[i].maxMemoryGBPerZone | int | true (in scheduledScaling) | Max memory size during the specified period. |\n\n### Example YAML Config\n```yaml\nindex: test\nshardsPerNode: 1\ndefaultMinMemoryGBPerZone: 64\ndefaultMaxMemoryGBPerZone: 256\nautoScaling:\n desiredCPUUtilPercent: 50\n scaleOutThresholdDuration: 5m\n scaleInThresholdDuration: 10m\nscheduledScalings:\n - startCronSchedule: TZ=UTC 0 0 * * *\n duration: 1h\n minMemoryGBPerZone: 128\n maxMemoryGBPerZone: 256\n```\n\n## How it scales\nAutoscaler tries to scale-out/scale-in within min/max range keeping configured `shardsPerNode`.\n\nIf CPU based auto-scaling is configured, Autoscaler tries to increase/decrease the number of nodes and replicas so that they are closer to the target utilization\nwhen the CPU utilization stays above/below the target CPU utilization for a certain period of time.\n\nIf it can't meet the `shardsPerNode`, Autoscaler won't apply scaling operation.\n\n### Example\n#### - scale-out when CPU utilization is higher then the target\n##### Premise\n```yaml\ncluster:\n memoryGBPerZone: 384 (64g * 6)\n zoneCount: 2\n averageCPUUtil: 60 (keeping 60 for 5 minutes)\nindex:\n numberOfShards: 2\n numberOfReplicas: 5\n```\n\n##### Config\n```yaml\nindex: test\nshardsPerNode: 1\ndefaultMinMemoryGBPerZone: 284 (64g * 6)\ndefaultMaxMemoryGBPerZone: 768 (64g * 12)\nautoScaling:\n desiredCPUUtilPercent: 45\n scaleOutThresholdDuration: 5m\n scaleInThresholdDuration: 10m\n```\n\n##### Result\n\nScaling-out to 8 nodes per zone to reduce CPU utilization. (`60% * 12 nodes / 16 nodes =\u003e 45%`).\n```yaml\ncluster:\n memoryGBPerZone: 256 (64g * 6) =\u003e 512 (64g * 8)\n zoneCount: 2\n averageCPUUtil: 80\nindex:\n numberOfShards: 2\n numberOfReplicas: 5 =\u003e 6\n```\n\n#### - not scaling due to `shardsPerNode` violation\n##### Premise\n```yaml\ncluster:\n memoryGBPerZone: 192 (64g * 3)\n zoneCount: 2\nindex:\n numberOfShards: 3\n numberOfReplicas: 1\n```\n\n##### Config\n```yaml\nindex: test\nshardsPerNode: 1\ndefaultMinMemoryGBPerZone: 256 (64g * 4)\ndefaultMaxMemoryGBPerZone: 256 (64g * 4)\n```\nIn the above case, Autoscaler won't apply scaling-out to 4 nodes x 2 zones even though defaultMinMemoryGBPerZone is 4 nodes.\nsince either 1 replica (6 shards in total) or 2 replicas (9 shards in total) can't be 8 which is required by `shardsPerNode: 1`.\n\n## Usage\nElastic Cloud Autoscaler can be used as library.\nExample is in [./examples/main.go](./examples/main.go).\n\nAlso handy docker image is provided. See [kyomo/elastic-cloud-autoscaler](https://hub.docker.com/r/kyomo/elastic-cloud-autoscaler) for more image details.\n\n## Constraints\n- Monitoring deployment must be enabled to use this library.\n https://www.elastic.co/guide/en/cloud/current/ec-enable-logging-and-monitoring.html#ec-enable-logging-and-monitoring-steps\n- This library only support `hot_content` topology and greater than or equal to `64g` memory size for now.\n- Scaling-out from 5 nodes or less to 6 nodes or more is not possible since from 6 data nodes dedicated master nodes are required.\n- `auto_expand_replicas` won't be used. Autoscaler will manually expand / drop replicas before or after node scaling.\n\n## Demo\nYou can easily test this library with the below repository.\nhttps://github.com/k-yomo/elastic-cloud-autoscaler-demo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-yomo%2Felastic-cloud-autoscaler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fk-yomo%2Felastic-cloud-autoscaler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk-yomo%2Felastic-cloud-autoscaler/lists"}