{"id":13797575,"url":"https://github.com/spotify/bigtable-autoscaler","last_synced_at":"2026-01-14T02:46:06.179Z","repository":{"id":33952103,"uuid":"147797791","full_name":"spotify/bigtable-autoscaler","owner":"spotify","description":"A service that autoscales Bigtable clusters based on CPU load","archived":true,"fork":false,"pushed_at":"2022-02-16T00:55:46.000Z","size":1913,"stargazers_count":71,"open_issues_count":17,"forks_count":22,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-12-26T16:51:04.482Z","etag":null,"topics":["bigtable","google-cloud"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spotify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2018-09-07T08:53:48.000Z","updated_at":"2025-12-18T02:11:29.000Z","dependencies_parsed_at":"2022-08-07T23:30:56.719Z","dependency_job_id":null,"html_url":"https://github.com/spotify/bigtable-autoscaler","commit_stats":null,"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"purl":"pkg:github/spotify/bigtable-autoscaler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbigtable-autoscaler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbigtable-autoscaler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbigtable-autoscaler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbigtable-autoscaler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spotify","download_url":"https://codeload.github.com/spotify/bigtable-autoscaler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fbigtable-autoscaler/sbom","scorecard":{"id":842159,"data":{"date":"2025-08-11","repo":{"name":"github.com/spotify/bigtable-autoscaler","commit":"6df20a8f67862cd6c2c8a977b4d751b2307ba61f"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":6,"reason":"Found 9/14 approved changesets -- score normalized to 6","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating openjdk:11-jre to openjdk:11-jre@sha256:356949c3125c4fa8104745e7ea92bd995da4567634e6599b470d2f972d13e0e2","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENCE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENCE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 25 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"77 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-vmq6-5m68-f53m","Warn: Project is vulnerable to: GHSA-668q-qrv7-99fm","Warn: Project is vulnerable to: GHSA-6v67-2wr5-gvf4","Warn: Project is vulnerable to: GHSA-pr98-23f8-jwxv","Warn: Project is vulnerable to: GHSA-h46c-h94j-95f3","Warn: Project is vulnerable to: GHSA-wf8f-6423-gfxg","Warn: Project is vulnerable to: GHSA-288c-cq4h-88gq","Warn: Project is vulnerable to: GHSA-3x8x-79m2-3w2w","Warn: Project is vulnerable to: GHSA-57j2-w4cx-62h2","Warn: Project is vulnerable to: GHSA-jjjh-jjxp-wpff","Warn: Project is vulnerable to: GHSA-rgv9-q543-rqg4","Warn: Project is vulnerable to: GHSA-4jrv-ppp4-jm57","Warn: Project is vulnerable to: GHSA-5mg8-w23w-74h3","Warn: Project is vulnerable to: GHSA-7g45-4rm6-3mm3","Warn: Project is vulnerable to: GHSA-f263-c949-w85g","Warn: Project is vulnerable to: GHSA-hw42-3568-wj87","Warn: Project is vulnerable to: GHSA-4gg5-vx3j-xwc7","Warn: Project is vulnerable to: GHSA-735f-pc8j-v9w8","Warn: Project is vulnerable to: GHSA-77rm-9x9h-xj3g","Warn: Project is vulnerable to: GHSA-g5ww-5jh7-63cx","Warn: Project is vulnerable to: GHSA-h4h5-3hr4-j3g2","Warn: Project is vulnerable to: GHSA-wrvw-hg22-4m67","Warn: Project is vulnerable to: GHSA-w33c-445m-f8w7","Warn: Project is vulnerable to: GHSA-cghx-9gcr-r42x","Warn: Project is vulnerable to: GHSA-m8wh-mqgf-rr8g","Warn: Project is vulnerable to: GHSA-2xxh-f8r3-hvvr","Warn: Project is vulnerable to: GHSA-4vrv-ch96-6h42","Warn: Project is vulnerable to: GHSA-cjcf-wm2p-59h5","Warn: Project is vulnerable to: GHSA-g76j-4cxx-23h9","Warn: Project is vulnerable to: GHSA-jcq3-cprp-m333","Warn: Project is vulnerable to: GHSA-m6vm-37g8-gqvh","Warn: Project is vulnerable to: GHSA-pwh7-92h3-mqr6","Warn: Project is vulnerable to: GHSA-4g9r-vxhx-9pgx","Warn: Project is vulnerable to: GHSA-7hfm-57qf-j43q","Warn: Project is vulnerable to: GHSA-crv7-7245-f45f","Warn: Project is vulnerable to: GHSA-mc84-pj99-q6hh","Warn: Project is vulnerable to: GHSA-xqfj-vm6h-2x34","Warn: Project is vulnerable to: GHSA-j288-q9x7-2f5v","Warn: Project is vulnerable to: GHSA-7r82-7xv7-xcpj","Warn: Project is vulnerable to: GHSA-6qvw-249j-h44c","Warn: Project is vulnerable to: GHSA-7g24-qg88-p43q","Warn: Project is vulnerable to: GHSA-jgvc-jfgh-rjvv","Warn: Project is vulnerable to: GHSA-4cx2-fc23-5wg6","Warn: Project is vulnerable to: GHSA-6xx3-rg99-gc3p","Warn: Project is vulnerable to: GHSA-hr8g-6v94-x4m9","Warn: Project is vulnerable to: GHSA-wjxj-5m7g-mg7q","Warn: Project is vulnerable to: GHSA-8xfc-gm6g-vgpv","Warn: Project is vulnerable to: GHSA-v435-xc8x-wvr9","Warn: Project is vulnerable to: GHSA-cj7v-27pg-wf7q","Warn: Project is vulnerable to: GHSA-hmr7-m48g-48f6","Warn: Project is vulnerable to: GHSA-qh8g-58pp-2wxh","Warn: Project is vulnerable to: GHSA-26vr-8j45-3r4w","Warn: Project is vulnerable to: GHSA-86wm-rrjm-8wh8","Warn: Project is vulnerable to: GHSA-g8m5-722r-8whq","Warn: Project is vulnerable to: GHSA-m394-8rww-3jr7","Warn: Project is vulnerable to: GHSA-m6cp-vxjx-65j6","Warn: Project is vulnerable to: GHSA-p26g-97m4-6q7c","Warn: Project is vulnerable to: GHSA-q4rv-gq96-w7c5","Warn: Project is vulnerable to: GHSA-qw69-rqj8-6qw8","Warn: Project is vulnerable to: GHSA-24rp-q3w6-vc56","Warn: Project is vulnerable to: GHSA-562r-vg33-8x8h","Warn: Project is vulnerable to: GHSA-673j-qm5f-xpv8","Warn: Project is vulnerable to: GHSA-727h-hrw8-jg8q","Warn: Project is vulnerable to: GHSA-88cc-g835-76rp","Warn: Project is vulnerable to: GHSA-r38f-c4h4-hqq2","Warn: Project is vulnerable to: GHSA-v7wg-cpwc-24m4","Warn: Project is vulnerable to: GHSA-36p3-wjmg-h94x","Warn: Project is vulnerable to: GHSA-hh26-6xwr-ggv7","Warn: Project is vulnerable to: GHSA-ffvq-7w96-97p7","Warn: Project is vulnerable to: GHSA-3mc7-4q67-w48m","Warn: Project is vulnerable to: GHSA-98wm-3w3q-mw94","Warn: Project is vulnerable to: GHSA-9w3m-gqgf-c4p9","Warn: Project is vulnerable to: GHSA-c4r9-r8fh-9vj2","Warn: Project is vulnerable to: GHSA-hhhw-99gj-p3c3","Warn: Project is vulnerable to: GHSA-mjmj-j48q-9wg2","Warn: Project is vulnerable to: GHSA-rvwf-54qp-4r6v","Warn: Project is vulnerable to: GHSA-w37g-rhq8-7m4j"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T20:43:04.204Z","repository_id":33952103,"created_at":"2025-08-23T20:43:04.204Z","updated_at":"2025-08-23T20:43:04.204Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408737,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigtable","google-cloud"],"created_at":"2024-08-04T00:00:24.365Z","updated_at":"2026-01-14T02:46:06.141Z","avatar_url":"https://github.com/spotify.png","language":"Java","funding_links":[],"categories":["Cool Stuff"],"sub_categories":["Interesting Projects"],"readme":"# DEPRECATION NOTICE\n\n**This repo is no longer actively maintained. While it should continue to work and there are no major known bugs, we will not be improving bigtable autoscaler or releasing new versions.**\n\n# bigtable-autoscaler\n\n[![CircleCI](https://circleci.com/gh/spotify/bigtable-autoscaler.svg?style=svg)](https://circleci.com/gh/spotify/bigtable-autoscaler)\n\nIf you have a Bigtable cluster and you would like to optimize its cost-efficiency by using the\nright number of nodes at any given time you should consider using this Bigtable\nautoscaler service! The Bigtable autoscaler lets you do that\nwith no manual intervention.\n\n## Getting started\n\n### Prerequisites\n\n* A production Bigtable cluster (or several) to autoscale\n* Service account JSON key that has relevant access to the Bigtable clusters to autoscale. See [Google's documentation](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) on how to create a key.\n    * If the autoscaler is running in the same GCP project as all the Bigtable clusters, the Compute Engine Default Service Account is sufficient.\n    * The minimum permissions are:\n        * Role **Bigtable Administrator**, in particular the permissions\n            * bigtable.clusters.get\n            * bigtable.clusters.update\n        * Role **Monitoring Viewer**, in particular the permissions\n            * monitoring.timeSeries.list\n* Docker\n* Java 11 and maven\n* (Optional) **PostgreSQL database** for production use. In this quickstart session we're using a postgres docker image\n* (Optional) We have a [make-file](https://www.gnu.org/software/make) with local development helper methods.\n\n### Building\n\nRun this command to build the project and create a docker image:\n\n    mvn package\n\n### Running\n\nFirst review and edit [.env](.env) with your Google cloud credentials.\nStart the service with docker-compose using a dockerized local postgres:\n\n    # source your environment\n    . ./.env\n    # start the service with docker compose\n    make up\n\n    # see service logs\n    make logs\n\nRegister the Bigtable cluster that should be autoscaled in the service:\n\n```\nPROJECT_ID=\u003cYOUR GCP PROJECT ID\u003e\nINSTANCE_ID=\u003cYOUR INSTANCE ID\u003e\nCLUSTER_ID=\u003cYOUR CLUSTER ID\u003e\n\ncurl -v -X POST \"http://localhost:8080/clusters?projectId=$PROJECT_ID\u0026instanceId=$INSTANCE_ID\u0026clusterId=$CLUSTER_ID\u0026minNodes=4\u0026maxNodes=6\u0026cpuTarget=0.8\"\n```\n\nIf the cluster was at 3 nodes, this will immediately rescale the cluster to 4 nodes as that's the\nminimum threshold. If you generate some significant load to the cluster, it may scale up to 6 nodes.\n\nStop docker-compose:\n\n    make down\n\n### Using a Cloud SQL Postgres database as persistent storage\n\nIf you want to run this in production, consider using a Cloud SQL postgres database to store the\nstate. We recommend connecting using the [JDBC socket factory](https://cloud.google.com/sql/docs/postgres/connect-external-app#java).\n\nJust update [.env](.env) with your postgres url, user and password and then run:\n\n    # source your environment\n    . ./.env\n    # start the service with docker compose\n    make run\n\nThis runs the same bigtable-autoscaler image, doesn't run postgres, and points bigtable-autoscaler to the postgresql you provided.\n\nIn the same way you can see service logs (make logs) and then to stop the service:\n\n    make stop\n\n## Registering Jersey Resources and Providers Dynamically\nYou can register any additional JAX-RS resource, JAX-RS or Jersey contract provider or JAX-RS feature by editing the\n[config file](/src/main/resources/bigtable-autoscaler.conf).\nYou can either\n* add a package to `additionalPackages` for any resource to be discovered. For this to work, resources to be discovered should be annotated.\n* add a fully qualified class name to `additionalClasses` (semicolon separated).\n\n## How does it work?\n\nThe Bigtable autoscaler is a backend service that periodically sends\nresize commands to Bigtable clusters. It is backed by a PostgreSQL database for\nkeeping its state, like for example:\n\n* number of nodes min/max boundaries\n* target CPU utilization\n* last resize event\n\nThe autoscaler checks the database every 30 seconds and decides if it should\ndo something or not (there are time thresholds to not resize clusters too often).\nIn case it's time to check a cluster, it fetches the current CPU utilization\nfrom the Bigtable API. If that is different from the target CPU utilization\n(also here there are thresholds) it calculates the adequate number of nodes\nand then it sends a resize request.\n\nThe autoscaler also provides an HTTP API to insert, update and delete Bigtable\nclusters from being autoscaled.\n\n## Development Status\n\n**Beta**: We are using Bigtable Autoscaler in production clusters at Spotify, and we are actively developing it.\n\n## FAQ\n\n### Does it handle sudden load spikes, for instance Dataflow jobs reading/writing batch data?\n\nNot on its own. In order to not overwhelm Bigtable, you can PUT to the `/clusers/override-min-nodes/` endpoint, passing it a number that basically overrides the min nodes count that the autoscaler must immediately respect. The [official Google documentation](https://cloud.google.com/bigtable/docs/scaling) states that if you are doing big batch jobs, you should rescale in advance and wait up to 20 minutes before starting the actual job.\n\nAdditionally, when you decrease the number of nodes in a cluster to scale down after the job is complete, try not to reduce the cluster size by more than 10% in a 10-minute period. Scaling down too quickly can cause performance problems, such as increased latency, if the remaining nodes in the cluster become temporarily overwhelmed.\n\nWe realize that this can be inconvenient and welcome any ideas on how to approach this problem better.\n\n### Does it enforce storage constraints?\n\nYes.\n\nSince July 1st 2018 Google enforces storage limits on Bigtable nodes. In particular each Bigtable node will be able to handle at most 8Tb on HDD clusters and 2.5Tb on SSD clusters (for more info take a look [here](https://cloud.google.com/bigtable/quotas#storage-per-node)). Writes will fail until these conditions are not satisfied. The autoscaler will make sure that these constraints are respected and prefer those to the CPU target in that situation.\n\n### Does it take project quotas into account?\n\nNo!\n\nA resize command may fail if you don't have enough quota in the GCP project. This will be logged\nas an error.\n\n### Can I add an additional logic to resize the number of nodes?\n\nYes!\n\nWe increased the project's modularity, so you can create your custom strategy in your project,\nwhich uses the Bigtable Autoscaler as a dependency, and implement the class \"Algorithm\".\nIf you add the class path of your new custom strategy in the column `extra_enabled_algorithms`, it\nwill be considered for upscaling the cluster.\n\nNote that the recommended number of nodes will be the higher between the strategies in this\nproject (CPU + Storage constraints), and your custom strategies.\n\n## API\n\nSee the [API doc](api.md)\n\n## Code of conduct\n\nThis project adheres to the\n[Open Code of Conduct](https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md).\nBy participating, you are expected to honor this code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fbigtable-autoscaler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fbigtable-autoscaler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fbigtable-autoscaler/lists"}