{"id":18644117,"url":"https://github.com/indix/vamana","last_synced_at":"2025-04-11T12:31:00.486Z","repository":{"id":49963580,"uuid":"44617563","full_name":"indix/vamana","owner":"indix","description":"Autoscaling toolkit based on custom Application Metrics","archived":false,"fork":false,"pushed_at":"2021-06-07T18:07:27.000Z","size":151,"stargazers_count":9,"open_issues_count":4,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T13:39:26.765Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/indix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-20T15:55:06.000Z","updated_at":"2020-06-04T20:31:12.000Z","dependencies_parsed_at":"2022-09-13T15:32:02.936Z","dependency_job_id":null,"html_url":"https://github.com/indix/vamana","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indix%2Fvamana","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indix%2Fvamana/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indix%2Fvamana/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indix%2Fvamana/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/indix","download_url":"https://codeload.github.com/indix/vamana/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248401936,"owners_count":21097328,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T06:10:04.153Z","updated_at":"2025-04-11T12:31:00.110Z","avatar_url":"https://github.com/indix.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://snap-ci.com/ashwanthkumar/vamana2/branch/master/build_image)](https://snap-ci.com/ashwanthkumar/vamana2/branch/master)\n\n# Vamana\n[Vamana](https://en.wikipedia.org/wiki/Vamana) is an Autoscalar Abstraction that provides ability to scale clusters on the cloud. Vamana by itself doesn't do autoscaling (not yet), but uses the Autoscalar available on various cloud platforms.  Some autoscalars on the cloud support scaling up / down based on application metrics collected on their platform. If you're already using a metric collection system you need not move away or send the same metrics to two different places instead Vamana can read metrics from any metrics store and perform autoscaling of your application. \n\n## Motivation\nI'm managing quite a number of Hadoop Cluster (across environments) whose TTs are backed by Auto Scaling Groups (ASG). \nEach cluster has its own usage patterns. Certain clusters run 24x7 while certain other clusters need to be up only during certain duration (when we have jobs running) and not always.\n- We were forced to add Scale Up and Scale Down stages on the beginning and end of our job pipelines.\n- Though using something like Anisble's ASG plugin made it trivial it was still a pain to add this everytime some one creates a new pipeline.\n- It became a problem when we've more than 1 job pipelines sharing the same cluster, one's scale down shouldn't affect the other's runtime.\n- Overnight / weekend failures makes the cluster remain idle.\n\n## Usage\nVamana is expected to be run on a Cron (with reasonable duration).\n```bash\n$ git clone https://github.com/ashwanthkumar/vamana2.git\n$ cd vamana2 \u0026\u0026 mvn clean package\n$ java -cp target/vamana2-1.0.0-SNAPSHOT.jar in.ashwanthkumar.vamana2.Vamana path/to/clusters.conf\n```\n\n## Architecture\n![Vamana2 Architecture](https://raw.githubusercontent.com/ashwanthkumar/vamana2/master/docs/vaman-architecture.png)\n\n## Configuration\nSample configuration would be\n```\nvamana {\n  clusters = [{\n    # Name of the cluster\n    name = \"Hadoop1 Staging Cluster\"\n\n    # Identifier used by AutoScalar when resizing the cluster\n    as-id = \"as-hadoop-staging-spot\"\n\n    # Maximum number of nodes the cluster can scale upto\n    max-nodes = 5\n\n    # Minimum number of nodes in the cluster\n    # We throw an RuntimeException if the Scalar returns less than this value\n    min-nodes = 1\n\n    metrics {\n      # Metrics that represent your demand\n      demand = [\"map_count_demand\", \"reduce_count_demand\"]\n\n      # Metrics that represent your supply\n      supply = [\"map_count_supply\", \"reduce_count_supply\"]\n\n      # Namespace for your metrics (Optional)\n      # Useful when using Amazon CloudWatch\n      namespace = \"Hadoop\"\n\n      # Dimension for your metrics (Optional)\n      # Useful when using Amazon CloudWatch\n      dimensions {\n        name1 = \"value1\"\n        name2 = \"value2\"\n      }\n\n      # Range of metrics to retrieve using collector\n      range = \"10m\" # Range of metrics to retrieve\n    }\n\n    # Collector Implementation to use\n    collector = \"in.ashwanthkumar.vamana2.aws.CloudWatchCollector\"\n\n    # Autoscalar Implementation to use\n    autoscalar = \"in.ashwanthkumar.vamana2.aws.AutoScalingGroups\"\n\n    # Scalar Implementation to use\n    scalar = \"in.ashwanthkumar.vamana2.apps.HadoopScalar\" # Use YARNScalar for YARN clusters\n  }]\n}\n```\n\n## Features\n- [x] Pluggable Metric Collector\n  - [x] Amazon CloudWatch\n- [x] Pluggable Scalar\n  - [x] Hadoop1 (works best with [hadoop-as-publisher](https://github.com/ashwanthkumar/hadoop-as-publisher))\n  - [x] YARN (works best with [hadoop-as-publisher](https://github.com/ashwanthkumar/hadoop-as-publisher))\n- [x] Pluggable AutoScalar\n  - [x] AutoScaling on AWS\n  - [ ] SpotFleet on AWS\n  - [ ] AutoScaler on GCE\n\nPresentation [\"Introducing Vamana\"](http://j.mp/to-vamana) shared with AWS Chennai Meetup October, 2015.\n\n## References / Inspirations\n- http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scaling.html - Closed source\n- http://www.qubole.com/blog/product/industrys-first-auto-scaling-hadoop-clusters/ - Paid service\n\n## Status\nThis project has been used in Production at Indix for more than 3 years for managing multiple Hadoop1 clusters with very good success. YARN Support has been added recently.\n\nIf you've any feature requests, please feel free to send a PR or create an issue.\n\n## License\nLicensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findix%2Fvamana","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Findix%2Fvamana","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findix%2Fvamana/lists"}