{"id":19124321,"url":"https://github.com/buildkite/buildkite-agent-scaler","last_synced_at":"2025-04-06T10:09:38.328Z","repository":{"id":28291137,"uuid":"113275102","full_name":"buildkite/buildkite-agent-scaler","owner":"buildkite","description":"📈A lambda for scaling an AutoScalingGroup based on Buildkite metrics","archived":false,"fork":false,"pushed_at":"2025-03-27T13:30:44.000Z","size":5424,"stargazers_count":61,"open_issues_count":12,"forks_count":31,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-30T09:05:18.087Z","etag":null,"topics":["aws","buildkite","lambda"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/buildkite.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-06T05:55:16.000Z","updated_at":"2025-02-17T17:07:06.000Z","dependencies_parsed_at":"2024-04-17T05:32:25.645Z","dependency_job_id":"ee804ca3-af28-4313-8ebd-d61bdc5eee3c","html_url":"https://github.com/buildkite/buildkite-agent-scaler","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Fbuildkite-agent-scaler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Fbuildkite-agent-scaler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Fbuildkite-agent-scaler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Fbuildkite-agent-scaler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/buildkite","download_url":"https://codeload.github.com/buildkite/buildkite-agent-scaler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247464185,"owners_count":20942966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","buildkite","lambda"],"created_at":"2024-11-09T05:28:44.984Z","updated_at":"2025-04-06T10:09:38.304Z","avatar_url":"https://github.com/buildkite.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Buildkite Agent Scaler\n\nAn AWS lambda function that handles the scaling of an\n[Amazon Autoscaling Group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html)\n(ASG) based on metrics provided by the Buildkite Agent Metrics API.\n\nIn practice, we've seen 300% faster initial scale-ups with this lambda vs native AutoScaling rules.\n🚀\n\n## Why?\n\nThe [Elastic CI Stack][] depends on being able to scale up quickly from zero instances in response\nto scheduled Buildkite jobs. Amazon's AutoScaling primatives have a number of limitations that we\nwanted more granular control over:\n\n* The median time for a scaling event to be triggered was 2 minutes, due to needing two samples with\n  a minimum period of 60 seconds between.\n* Scaling can either be by a fixed rate, a fixed step size or tracking, but tracking doesn't work\n  well with custom metrics like we use.\n\n## How does it work?\n\nThe lambda (or cli version) polls the Buildkite Metrics API every 10 seconds, and based on the\nresults sets the `DesiredCount` to exactly what is needed. This allows much faster scale up.\n\n## Gracefully scaling in\n\nWhilst the lambda does support scaling in via setting `DesiredCount`, Amazon ASGs appear to not send\n[Lifecycle Hooks][] before terminating instances, so jobs in progress are interrupted.\n\nInstead, in the [Elastic CI Stack][] we run the scaler with scale-in disabled (`DISABLE_SCALE_IN`)\nand rely on the\n[recent addition in buildkite-agent v3.10.0](https://github.com/buildkite/agent/releases/tag/v3.10.0)\nof `--disconnect-after-idle-timeout` in the Agent combined with a\n[systemd PostStop script](https://github.com/buildkite/elastic-ci-stack-for-aws/blob/00c45ab47160b1d1d44c0b3bea8456456444c60e/packer/linux/conf/bin/bk-install-elastic-stack.sh#L136-L143)\nto terminate the instance and atomically decrease the `DesiredCount` after the agent has been idle\nfor a time period. We've found it to work really well, and is less complicated than relying on\n[lifecycled] and [Lifecycle Hooks][].\n\nSee the [forum post](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425) for more details.\n\n## Publishing Cloudwatch Metrics\n\nThe scaler collects it's own metrics and doesn't require [buildkite-agent-metrics][]. It supports\noptionally publishing the metrics it collects back to Cloudwatch, although it only supports a subset\nof the metrics that the [buildkite-agent-metrics][] binary collects:\n\n* Buildkite \u003e (Org, Queue) \u003e `ScheduledJobsCount`\n* Buildkite \u003e (Org, Queue) \u003e `RunningJobCount`\n\n## Running as an AWS Lambda\n\nAn AWS Lambda bundle is created and published as part of the build process. The lambda will require\nthe following IAM permissions:\n\n- `cloudwatch:PutMetricData`\n- `autoscaling:DescribeAutoScalingGroups`\n- `autoscaling:DescribeScalingActivities`\n- `autoscaling:SetDesiredCapacity`\n\nIts handler is `bootstrap`, it uses a `provided.al2` runtime and requires the following env vars:\n\n- `BUILDKITE_AGENT_TOKEN` or `BUILDKITE_AGENT_TOKEN_SSM_KEY`\n- `BUILDKITE_QUEUE`\n- `AGENTS_PER_INSTANCE`\n- `ASG_NAME`\n\nIf `BUILDKITE_AGENT_TOKEN_SSM_KEY` is set, the token will be read from\n[AWS Systems Manager Parameter Store GetParameter](https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameter.html)\nwhich [can also read from AWS Secrets Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html).\n\n```bash\naws lambda create-function \\\n  --function-name buildkite-agent-scaler \\\n  --memory 128 \\\n  --role arn:aws:iam::account-id:role/execution_role \\\n  --runtime provided.al2 \\\n  --zip-file fileb://handler.zip \\\n  --handler bootstrap\n```\n\n## Running locally for development\n\n```\n$ aws-vault exec my-profile -- go run . \\\n  --asg-name elastic-runners-AgentAutoScaleGroup-XXXXX\n  --agent-token \"$BUILDKITE_AGENT_TOKEN\"\n```\n\n## Using Clusters\n\nThe `BUILDKITE_AGENT_TOKEN` is scoped to a specific cluster. It's best to create a unique token for\nthe cluster being targeted by the scaler.\n\nThe scaler is set up automatically by the [Elastic CI Stack][]'s CloudFormation templates, which\nreference the agent token and a queue name. A Lambda function running the scaler is then generated\nusing these references (e.g., `BUILDKITE_AGENT_TOKEN_SSM_KEY` and `BUILDKITE_QUEUE`).\n\n\n## Copyright\n\nCopyright (c) 2014-2019 Buildkite Pty Ltd. See [LICENSE](./LICENSE.txt) for details.\n\n[Elastic CI Stack]: https://github.com/buildkite/elastic-ci-stack-for-aws\n[buildkite-agent-metrics]: https://github.com/buildkite/buildkite-agent-metrics\n[Lifecycle Hooks]: https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html\n[lifecycled]: https://github.com/buildkite/lifecycled\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuildkite%2Fbuildkite-agent-scaler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuildkite%2Fbuildkite-agent-scaler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuildkite%2Fbuildkite-agent-scaler/lists"}