{"id":13618392,"url":"https://github.com/structurely/ecs-autoscale","last_synced_at":"2025-04-14T10:31:51.049Z","repository":{"id":197064460,"uuid":"117281233","full_name":"structurely/ecs-autoscale","owner":"structurely","description":"A framework that runs on AWS Lambda for autoscaling ECS clusters and services","archived":false,"fork":false,"pushed_at":"2020-05-11T13:33:14.000Z","size":620,"stargazers_count":72,"open_issues_count":4,"forks_count":9,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-08T03:37:04.974Z","etag":null,"topics":["autoscaling","aws","aws-ec2","aws-ecs","aws-lambda","ec2","ecs","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/structurely.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-01-12T19:43:13.000Z","updated_at":"2024-10-26T11:34:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"29e768da-bb7e-4723-be1c-e171b9142e4c","html_url":"https://github.com/structurely/ecs-autoscale","commit_stats":null,"previous_names":["structurely/ecs-autoscale"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structurely%2Fecs-autoscale","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structurely%2Fecs-autoscale/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structurely%2Fecs-autoscale/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/structurely%2Fecs-autoscale/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/structurely","download_url":"https://codeload.github.com/structurely/ecs-autoscale/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248862699,"owners_count":21173861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaling","aws","aws-ec2","aws-ecs","aws-lambda","ec2","ecs","python"],"created_at":"2024-08-01T20:02:00.288Z","updated_at":"2025-04-14T10:31:50.470Z","avatar_url":"https://github.com/structurely.png","language":"Python","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/structurely/ecs-autoscale/blob/master/docs/img/logo.png\"/\u003e\n\u003c/p\u003e\n\n# ecs-autoscale\n\n[![Build Status](https://travis-ci.org/structurely/ecs-autoscale.svg?branch=master)](https://travis-ci.org/structurely/ecs-autoscale)\n\nThis is a Lambda function that allows you to automatically\nscale EC2 instances and services within an ECS cluster simultaneously based on \narbitrary metrics from sources not limited to CloudWatch.\n\n## Table of contents\n\n- [Requirements](https://github.com/structurely/ecs-autoscale#requirements)\n- [Quick start](https://github.com/structurely/ecs-autoscale#quick-start)\n- [Deploying updates](https://github.com/structurely/ecs-autoscale#deploying-updates)\n- [Scaling details](https://github.com/structurely/ecs-autoscale#scaling-details)\n- [Metrics](https://github.com/structurely/ecs-autoscale#metrics)\n  - [Sources](https://github.com/structurely/ecs-autoscale#sources)\n  - [Metric arithmetic](https://github.com/structurely/ecs-autoscale#metric-arithmetic)\n- [Logging](https://github.com/structurely/ecs-autoscale#logging)\n- [Contributing](https://github.com/structurely/ecs-autoscale#contributing)\n\n## Requirements\n\nThe only requirement is an AWS account with programmatic access and Docker.\n\n## Quick start\n\nSuppose we want to set up autoscaling for a cluster on ECS called `my_cluster`\nwith two services running: `backend` and `worker`. Suppose `backend` is just a simple\nweb server and `worker` is a [celery](http://www.celeryproject.org) worker \nfor handling long-running tasks for the web server with a RabbitMQ instance as the broker.\n\nIn this case we want to scale the web server based on CPU utilization and scale the\ncelery worker based on the number of waiting tasks (which is given by the number of `ready` \nmessages on the RabbitMQ instance).\n\nWe can get the CPU utilization of the web server directly from CloudWatch,\nbut in order to get the number of queued messages in the RabbitMQ instance,\nwe will need to make an HTTP GET request to the api of the queue.\n\n\u003e To learn more about the RabbitMQ API, see [https://cdn.rawgit.com/rabbitmq/rabbitmq-management/v3.7.2/priv/www/api/index.html](https://cdn.rawgit.com/rabbitmq/rabbitmq-management/v3.7.2/priv/www/api/index.html).\n\n**Step 1: Define the cluster scaling requirements**\n\nCreate a YAML file `./lambda/clusters/my_cluster.yml`.\n\n\u003e NOTE: The name of the YAML file sans extension must exactly match the name of the cluster on ECS.\n\nOur cluster definition will look like this:\n\n```yaml\n# Exact name of the autoscaling group.\nautoscale_group: EC2ContainerService-my_cluster-EcsInstanceAsg-AAAAA\n\n# Set to false to ignore this cluster when autoscaling.\nenabled: true\n\n# Buffer room: you can think of this as an empty service / task.\ncpu_buffer: 0  # Size of buffer in CPU units.\nmem_buffer: 0  # Size of buffer in memory.\n\n# Optionally specify the minimum and maximum number of instances for the cluster's\n# autoscaling group from here.\nmin: 1\nmax: 4\n\n# Defines scaling for individual services.\nservices:\n  # Here we specify scaling for the \"worker\" service.\n  worker:  # This should be the exact name of the service as on ECS.\n    # Set to false to ignore service when autoscaling.\n    enabled: true\n\n    min: 1  # Min number of tasks.\n    max: 3  # Max number of tasks.\n\n    metric_sources:\n      # Data sources needed for gathering metrics. Currently only `third_party` and \n      # `cloudwatch` are supported. Only one statistic from one source is needed.\n      # For more information on the metrics available, see below under \"Metrics\".\n      third_party:\n        - url: https://username:password@my_rabbitmq_host.com/api/queues/celery\n          method: GET  # Either GET or POST\n          payload: null  # Optional JSON paylaod to include with the request\n          statistics:\n            - name: messages_ready\n              alias: queue_length\n          # In this case it is assumed that we will make a GET request to the url\n          # given, and that request will return a JSON object that contains\n          # the field `messages_ready`.\n\n    # Autoscaling events which determine when to scale up or down.\n    events:\n      - metric: queue_length  # Name of metric to use.\n        action: 1  # Scale up by one.\n        # Conditions of the event:\n        min: 5\n        max: null\n      - metric: queue_length\n        action: -1  # Scale down by one.\n        min: null\n        max: 3\n\n  # Here we specify scaling for the \"backend\" service.\n  backend:\n    enabled: true\n    min: 1\n    max: 3\n    metric_sources:\n      # We only need metrics from CloudWatch this time.\n      cloudwatch:\n        - namespace: AWS/ECS\n          metric_name: CPUUtilization\n          dimensions:\n            - name: ClusterName\n              value: my_cluster\n            - name: ServiceName\n              value: backend\n          period: 300\n          statistics:\n            - name: Average\n              alias: cpu_usage\n\n    events:\n      - metric: cpu_usage\n        action: 1  # Scale up by 1\n        min: 10\n        max: null\n      - metric: cpu_usage\n        action: -1  # Scale down by 1\n        min: null\n        max: 1\n```\n\n\u003e NOTE: You may not want to store sensitive information in your cluster definition,\nsuch as the username and password in the RabbitMQ URL above. In this case you could store\nthose values in environment variables and pass them to the cluster definition\nusing our special syntax: `%(VARIABLE_NAME)`. So, for example, suppose\nwe have the environment variables `USERNAME` and `PASSWORD`. Then the line above\nwith the url for RabbitMQ would become `url: https://%(USERNAME):%(PASSWORD)@my_rabbitmq_host.com/api/queues/celery`.\n\n\n**Step 2: Build the docker image**\n\nThe docker image is used to test the Lambda function locally, as well as to set \nup the Lambda environment on AWS and upload deployment packages.\n\nYou can build the image with\n\n```\ndocker build -t epwalsh/ecs-autoscale .\n```\n\n**Step 3: Test the function locally**\n\nWe can use the Docker image created in step 2 to test the Lamda function.\nIn order to start the container, create a file called `access.txt`\nthat looks like this:\n\n```\nAWS_DEFAULT_REGION=***\nAWS_ACCESS_KEY_ID=***\nAWS_SECRET_ACCESS_KEY=***\n```\n\nIf your cluster definitions requires other environment variabes, you can also\nput them in there.\n\nThen run:\n\n```bash\ndocker run --env-file=./access.txt --rm epwalsh/ecs-autoscale make test-run\n```\n\n**Step 4: Setup and deployment**\n\nWe can now create the Lambda function on AWS.\n\nAll you have to do is run the Docker container again with the `bootstrap.sh`\nscript as the command:\n\n```bash\ndocker run --env-file=./access.txt --rm epwalsh/ecs-autoscale ./bootstrap.sh\n```\n\nThis will:\n\n- Create an IAM policy that gives access to the resources the lambda function will need.\n- Create a role for the Lambda function to use, and attach the policy to that role.\n- Build a deployment package.\n- Create a Lambda function on AWS with the role attached and upload the deployment package.\n\n\n**Step 5: Create a trigger to execute your function**\n\nIn this example we will create a simple CloudWatch rule that triggers our Lambda function to run\nevery 5 minutes.\n\nTo do this, first login to the AWS Console and the go to the CloudWatch service. On the left side menu,\nclick on \"Rules\". You should see a page that looks like this:\n\n![step1](https://github.com/structurely/ecs-autoscale/blob/master/docs/img/step1.png)\n\nThen click \"Create rule\" by the top. You should now see a page that looks like this:\n\n![step2](https://github.com/structurely/ecs-autoscale/blob/master/docs/img/step2.png)\n\nMake sure you check \"Schedule\" instead of \"Event Pattern\", and then set it to a fixed\nrate of 5 minutes. Then on the right side click \"Add target\" and choose \"ecs-autoscale\"\nfrom the drop down.\n\nNext click \"Configure details\", give your rule a name, and then click \"Create rule\".\n\nYou're all set! After 5 minutes your function should run.\n\n\n## Deploying updates\n\nIf you update your cluster definitions, you can easily redeploy **ecs-autoscale**\nby running:\n\n```bash\ndocker run --env-file=./access.txt --rm epwalsh/ecs-autoscale make deploy\n```\n\n## Scaling details\n\n### Scaling individual services\n\nIndividual services can be scaled up or down according to arbitrary metrics, as\nlong as those metrics can be gathered through a simple HTTP request. For example,\ncelery workers can be scaled according to the number of queued messages.\n\n### Scaling up the cluster\n\nA cluster is triggered to scale up by one instance when both of the following two conditions are met:\n\n- the desired capacity of the corresponding autoscaling group is less than the maximum capacity, and\n- the additional tasks for services that need to scale up cannot fit on the existing \ninstances with room left over for the predefined CPU and memory buffers.\n\n### Scaling down the cluster\n\nA cluster is triggered to scale down by one instance when both of the following two conditions are met:\n\n- the desired capacity of the corresponding autoscaling group is greater than the minimum capacity, and\n- all of the tasks on the EC2 instance in the cluster with either the smallest amount of \nreserved CPU units or memory could fit entirely on another instance in the cluster, and \nso that the other instances could still support all additional tasks for services that need\nto scale up with room left over for the predefined CPU and memory buffers.\n\n## Metrics\n\n### Sources\n\nIn order to use a metric, you need to define the source of the metric under\n`metric_sources` in the YAML definition. In the example above, we defined a third party\nmetric like this:\n\n```yaml\nthird_party:\n  - url: https://username:password@my_rabbitmq_host.com/api/queues/celery\n    method: GET\n    payload: null\n    statistics:\n      - name: messages_ready\n        alias: queue_length\n```\n\nThis created a metric called `queue_length` based on `messages_ready`.\nThe alias `queue_length` was arbitrary, and is the name used to reference this\nmetric when defining events like in the above example:\n\n```yaml\nevents:\n  - metric: queue_length\n    action: 1\n    min: 5\n    max: null\n  - metric: queue_length\n    action: -1\n    min: null\n    max: 3\n```\n\nIn general, third party metrics are gathered by making an HTTP request to the\nurl given. It is then assumed that the request will return a JSON object with\na field name corresponding to the `name` of the metric. To retreive a nested field\nin the JSON object, you can use dot notation.\n\nDefining metrics from CloudWatch are pretty straight forward as well, like in our example:\n\n```yaml\nmetric_sources:\n  cloudwatch:\n    - namespace: AWS/ECS\n      metric_name: CPUUtilization\n      dimensions:\n        - name: ClusterName\n          value: my_cluster\n        - name: ServiceName\n          value: backend\n      period: 300\n      statistics:\n        - name: Average\n          alias: cpu_usage\n```\n\nOne thing to watch out for is how you define the `statistics` field above.\nThe `name` part has to match exactly with a statistic used by CloudWatch,\nand the `alias` part is an arbitrary name you use to reference this metric when\ndefining events.\n\n\n### Metric arithmetic\n\nYou can easily combine metrics with arbitrary arithmetic operations.\nFor example, suppose we create two metrics with aliases `cpu_usage` and `mem_usage`.\nWe could create an event based on the product of these two metrics like this:\n\n```yaml\nevents:\n  - metric: cpu_usage * mem_usage * 100\n    action: 1\n    min: 50\n    max: 100\n```\n\nYou could even go crazy for no reason:\n\n```yaml\nevents:\n  - metric: cpu_usage ** 2 - (mem_usage - 2000 + 1) * mem_usage + mem_usage * 0\n    action: 1\n    min: 50\n    max: 100\n```\n\nIn fact, metric arithmetic is interpreted directly as a Python statement, so you can even \nuse functions like `min` and `max`:\n\n```yaml\nevents:\n  - metric: max([cpu_usage, mem_usage])\n    action: 1\n    min: 0.5\n    max: 1.0\n```\n\n## Logging\n\nLogs from the Lambda function will be sent to a CloudWatch logstream `/aws/lambda/ecs-autoscale`.\nYou can also set the log level easily by setting the environment variable `LOG_LEVEL`, which can\nbe set to\n\n- `debug`\n- `info`\n- `warning`\n- `error`\n\n## Contributing\n\nThis project is in its very early stages and we encourage developer contributions.\nPlease read [CONTRIBUTING.md](https://github.com/structurely/ecs-autoscale/blob/master/CONTRIBUTING.md) before submitting a PR.\n\n## Bugs\n\nTo report a bug, submit an issue at [https://github.com/structurely/ecs-autoscale/issues/new](https://github.com/structurely/ecs-autoscale/issues/new).\n\n\n## Credit where credit is due\n\nThis project was inspired by the following articles and projects:\n\n- [http://garbe.io/blog/2017/04/12/a-better-solution-to-ecs-autoscaling/](http://garbe.io/blog/2017/04/12/a-better-solution-to-ecs-autoscaling/)\n- [https://medium.com/@omerxx/how-to-scale-in-ecs-hosts-2d0906d2ba](https://medium.com/@omerxx/how-to-scale-in-ecs-hosts-2d0906d2ba)\n- [https://github.com/omerxx/ecscale/blob/master/ecscale.py](https://github.com/omerxx/ecscale/blob/master/ecscale.py)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstructurely%2Fecs-autoscale","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstructurely%2Fecs-autoscale","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstructurely%2Fecs-autoscale/lists"}