{"id":15292921,"url":"https://github.com/hammady/scaltainer","last_synced_at":"2025-04-13T12:14:11.425Z","repository":{"id":26575230,"uuid":"108380057","full_name":"hammady/scaltainer","owner":"hammady","description":"Autoscale Kubernetes controllers or Docker Swarm services based on application metrics and more","archived":false,"fork":false,"pushed_at":"2023-09-15T21:27:17.000Z","size":96,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T03:22:26.277Z","etag":null,"topics":["autoscaling","docker","docker-swarm-mode","heroku","hirefire","k8s","k8s-cluster","kubernetes","kubernetes-cluster","metrics","newrelic","ruby-gem","worker-service"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hammady.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-26T08:00:36.000Z","updated_at":"2024-05-22T22:58:43.000Z","dependencies_parsed_at":"2022-08-07T12:00:43.623Z","dependency_job_id":null,"html_url":"https://github.com/hammady/scaltainer","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hammady%2Fscaltainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hammady%2Fscaltainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hammady%2Fscaltainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hammady%2Fscaltainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hammady","download_url":"https://codeload.github.com/hammady/scaltainer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248594138,"owners_count":21130312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaling","docker","docker-swarm-mode","heroku","hirefire","k8s","k8s-cluster","kubernetes","kubernetes-cluster","metrics","newrelic","ruby-gem","worker-service"],"created_at":"2024-09-30T16:32:01.251Z","updated_at":"2025-04-13T12:14:11.402Z","avatar_url":"https://github.com/hammady.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Build Status](https://github.com/hammady/scaltainer/actions/workflows/tests.yml/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/hammady/scaltainer/badge.svg?service=github\u0026branch=master)](https://coveralls.io/github/hammady/scaltainer?branch=master)\n[![Gem Version](https://badge.fury.io/rb/scaltainer.svg)](https://badge.fury.io/rb/scaltainer)\n\n# Scaltainer\n\nA Ruby gem to monitor Docker Swarm mode services and Kubernetes resources\nand auto-scale them based on user configuration.\nIt can be used to monitor web services and worker services. The web services type has metrics like response time using [New Relic](https://newrelic.com/). The worker services type metrics are basically the queue size for each.\nThis gem is inspired by [HireFire](https://manager.hirefire.io/) and was indeed motivated by the migration\nfrom [Heroku](https://www.heroku.com/) to Docker.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\nInstall using rubygems:\n\n    $ gem install scaltainer\n\n## Usage\n\nFor Docker swarm:\n\n    bundle exec scaltainer -o swarm\n\nOr simply:\n\n    bundle exec scaltainer\n\nFor Kubernetes:\n\n    bundle exec scaltainer -o kubernetes\n\n\nThis will do a one-time check on the running docker service replicas\nor Kubernetes replication controllers, replica sets, or deployments.\nThen it sends scaling out/in commands to the cluster as appropriate.\nConfiguration is read from `scaltainer.yml` by default. If you want to read from another file add `-f yourconfig.yml`:\n\n    bundle exec scaltainer -f yourconfig.yml\n\nNote that after each run a new file is created (`yourconfig.yml.state`) which stores the state of the previous run.\nThis is because there are some configuration parameters (like sensitivity) need to\nremember previous runs.\nIf you want to specify a different location for the state file, add the `--state-file` parameter.\nExample:\n\n    bundle exec scaltainer -f /path/to/configuration/file.yml --state-file /path/to/different/state/file.yml\n\nTypically one would want to repeatedly call scaltainer every minute or so. To do this\nspecify the wait time between repetitions using the `-w` parameter in seconds:\n\n    bundle exec scaltainer -w 60\n\nThis will repeatedly call scaltainer every 60 seconds, sleeping in-between.\n\nIf you would like to monitor the changes in scaling out and in. You can install\nPrometheus and add a configuration parameter pointing to its Push Gateway:\n\n    bundle exec scaltainer -g prometheus-pushgateway.monitoring.svc.cluster.local:9091\n\nWhere `prometheus-pushgateway.monitoring.svc.cluster.local:9091` is the address\nof the push gateway. For Kubernetes environments the above denotes the gateway service\nname (`prometheus-pushgateway`), where it is installed in the namespace called\n`monitoring`. Scaltainer will report the following metrics to Prometheus:\n\n- `scaltainer_web_replicas_total`: number of web replicas scaled (or untouched thereof).\nThis is labeled by the namespace and controller name, both matching the scaltainer\nconfiguration file.\n- `scaltainer_worker_replicas_total`: Same as above, but for workers\n- `scaltainer_web_response_time_seconds`: response times as reported by the web services\n- `scaltainer_worker_queue_size_total`: queue sizes as reported by the worker services\n- `scaltainer_ticks_total`: iterations scaltainer has performed (if `-w` is used)\n\nIf you prefer to use New Relic monitoring, replace the `-g` parameter with `--enable-newrelic-reporting`.\nIf enabled, must set the environment variables `NEW_RELIC_LICENSE_KEY` and `NEW_RELIC_APP_NAME` (see below).\nOnce enabled, the below will be reported:\n\n- `Custom/WebReplicas/service`\n- `Custom/WorkerReplicas/service`\n- `Custom/WebMetric/service`\n- `Custom/WorkerMetric/service`\n- `Custom/Scaltainer/ticks`\n\nWhere `service` is a placeholder for each service defined in `yourconfig.yml`.\n\nHere is an example NRQL to query the metrics:\n\n```\nFROM Metric\nSELECT max(newrelic.timeslice.value)\nWHERE appName = 'YOUR APP NAME'\nWITH METRIC_FORMAT 'Custom/WebReplicas/{web}'\nFACET web\nSINCE 1 day ago TIMESERIES MAX\n```\n\n## Configuration\n\n### Environment variables\n\n#### Docker swarm options\n\n- `DOCKER_URL`: Should point to the docker engine URL.\nIf not set, it defaults to local unix socket.\n\n#### Kubernetes options\n\n- `KUBECONFIG`: set to Kubernetes config\n(default: `$HOME/.kube/config`) if you want to connect\nto the current configured cluster.\n\n- `KUBERNETES_API_SERVER`: overrides option in `KUBECONFIG`\nand defaults to `https://kubernetes.default:443`.\n\n- `KUBERNETES_SKIP_SSL_VERIFY`: `KUBECONFIG` option overrides\nthis, set to any value to skip SSL verification.\n\n- `KUBERNETES_API_ENDPOINT`: defaults to `/api`.\n\n- `KUBERNETES_API_VERSION`: overrides option in `KUBECONFIG`\nand defaults to `v1`.\n\n- `KUBERNETES_CONTROLLER_KIND`: controller kind to scale,\nallowed values: `deployment` (default),\n`replication_controller`, or `replica_set`.\n\nMake sure the `KUBERNETES_CONTROLLER_KIND` you specify is\npart of the api specified using `KUBERNETES_API_ENDPOINT`\nand `KUBERNETES_API_VERSION`.\n\n#### General options\n\n- `HIREFIRE_TOKEN`: If your application is configured the\n[hirefire](https://help.hirefire.io/guides/hirefire/job-queue-any-programming-language) way, you need to\nset `HIREFIRE_TOKEN` environment variable before invoking\n`scaltainer`. This is used when probing your application\nendpoint (see below) to get the number of jobs per queue\nfor each worker.\n\n- `NEW_RELIC_API_KEY`: New Relic API key. Currently New Relic\nis used to retrieve average response time metric for web services.\nMore monitoring services can be added in the future.\n\n- `RESPONSE_TIME_WINDOW`: Time window in minutes to measure\naverage response time till the moment. For example 3 means\nmeasure average response time in the past 3 minutes. Default value is 5.\n\n- `LOG_LEVEL`: Accepted values here are: `DEBUG`, `INFO` (default), `WARN`, `ERROR`, `FATAL`.\nLog output goes to stdout.\n\n- `DOCKER_SECRETS_PATH_GLOB`: Path glob containing environment files to load.\nThis is useful if running from a docker swarm mode environment where one or more of the above\nenvironment variables are set using `docker config` or `docker secret`.\nThese files should be in the form `VARIABLE=value`.\nA typical value of this variable would be: `{/run/secrets/*,/config1,/config2}`\n\n- `NEW_RELIC_LICENSE_KEY`: New Relic license key, required if `--enable_newrelic_reporting`\nis used.\n- `NEW_RELIC_APP_NAME`: New Relic application name, required if `--enable_newrelic_reporting`\nis used.\n### Configuration file\n\nThe configuration file (determined by `-f FILE` command line parameter) should be in the following form:\n\n    # to get worker metrics\n    endpoint: https://your-app.com/hirefire/$HIREFIRE_TOKEN/info\n    # optional docker swarm stack name or kubernetes namespace (useful if having push gateway)\n    namespace: mynamespace\n    # list of web services to monitor\n    web_services:\n      # each service name should match docker service name\n      web:\n        # New Relic application id (required)\n        newrelic_app_id: \u003capp_id\u003e\n        # minimum replicas to maintain (default: 0)\n        min: 1\n        # maximum replicas to maintain (default: unlimited)\n        max: 5\n        # maximum response time above which to scale up (required)\n        max_response_time: 300\n        # minimum response time below which to scale down (required)\n        min_response_time: 100\n        # replica quantitiy to scale up at a time (default: 1)\n        upscale_quantity: 2\n        # replica quantitiy to scale down at a time (default: 1)\n        downscale_quantity: 1\n        # number of breaches to wait for before scaling up (default: 1)\n        upscale_sensitivity: 1\n        # number of breaches to wait for before scaling down (default: 1)\n        downscale_sensitivity: 1\n      webapi:\n        ...\n    worker_services:\n      worker1:\n        min: 1\n        max: 10\n        # number of jobs each worker replica should process (required)\n        # the bigger the ratio, the less number of workers scaled out\n        ratio: 3\n        upscale_sensitivity: 1\n        downscale_sensitivity: 1\n      worker2:\n        ...\n\nMore details about configuration parameters can be found in [HireFire docs](https://help.hirefire.io/guides).\n\n## Docker Swarm usage\n\nA service definition for scaltainer is typically something like this:\n\n    version: '3.3'\n    services:\n      scaltainer:\n        image: hammady/scaltainer:latest\n        command: -f /scaltainer.yml --state-file /tmp/scaltainer-state.yml -w 60\n        volumes:\n          - /var/run/docker.sock:/var/run/docker.sock\n        environment:\n          - DOCKER_URL=unix:///var/run/docker.sock\n          - DOCKER_SECRETS_PATH_GLOB={/run/secrets/*}\n          - RESPONSE_TIME_WINDOW=3\n        configs:\n          - source: scaltainer\n            target: /scaltainer.yml\n        secrets:\n          - scaltainer\n        deploy:\n          replicas: 1\n          placement:\n            constraints:\n              - node.role == manager\n    configs:\n      scaltainer:\n        file: scaltainer.yml\n    secrets:\n      scaltainer:\n        file: scaltainer.env\n\nWhere `scaltainer.env` is a file containing HireFire and NewRelic secrets:\n\n    HIREFIRE_TOKEN=\n    NEW_RELIC_API_KEY=\n\nAnd `scaltainer.yml` is the scaltainer configuration file.\n\n## Kubernetes usage\n\n### Create a ConfigMap\n\n    kubectl create configmap scaltainer --from-file=scaltainer.yaml=/path/to/your/scaltainer.yml\n\nWhere `/path/to/your/scaltainer.yml` is the scaltainer configuration file.\n\n### Create a Secret\n\n    kubectl create secret generic scaltainer --from-env-file=/path/to/scaltainer.env\n\nWhere `/path/to/scaltainer.env` is a file containing HireFire and NewRelic secrets:\n\n    HIREFIRE_TOKEN=\n    NEW_RELIC_API_KEY=\n\n### Create a Deployment:\n\n    kubectl apply -f scaltainer-kube.yaml\n\nWhere scaltainer-kube.yaml has the following content:\n\n    apiVersion: extensions/v1beta1\n    kind: Deployment\n    metadata:\n      labels:\n        app: scaltainer\n      name: scaltainer\n    spec:\n      replicas: 1\n      template:\n        metadata:\n          labels:\n            app: scaltainer\n        spec:\n          containers:\n          - image: hammady/scaltainer:latest\n            name: scaltainer\n            args:\n              - -o\n              - kubernetes\n              - -f\n              - /etc/config/scaltainer.yaml\n              - --state-file\n              - /tmp/scaltainer-state.yaml\n              - -w\n              - \"60\"\n            env:\n            - name: KUBERNETES_SKIP_SSL_VERIFY\n              value: \"yes\"\n            - name: KUBERNETES_API_ENDPOINT\n              value: /apis/extensions\n            - name: KUBERNETES_API_VERSION\n              value: v1beta1\n            - name: KUBERNETES_CONTROLLER_KIND\n              value: deployment\n            envFrom:\n            - secretRef:\n                name: scaltainer\n            volumeMounts:\n            - name: scaltainer-config\n              mountPath: \"/etc/config\"\n          volumes:\n          - name: scaltainer-config\n            configMap:\n              name: scaltainer\n\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/hammady/scaltainer.\n\n## Testing\n\n    rake\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhammady%2Fscaltainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhammady%2Fscaltainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhammady%2Fscaltainer/lists"}