{"id":13839352,"url":"https://github.com/uber/peloton","last_synced_at":"2025-06-11T17:40:16.270Z","repository":{"id":44430897,"uuid":"85815209","full_name":"uber/peloton","owner":"uber","description":"Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.","archived":false,"fork":false,"pushed_at":"2023-05-20T20:09:21.000Z","size":44996,"stargazers_count":644,"open_issues_count":20,"forks_count":65,"subscribers_count":848,"default_branch":"master","last_synced_at":"2024-11-21T04:31:03.925Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-03-22T10:33:00.000Z","updated_at":"2024-11-18T10:51:23.000Z","dependencies_parsed_at":"2024-01-07T18:09:27.134Z","dependency_job_id":null,"html_url":"https://github.com/uber/peloton","commit_stats":{"total_commits":705,"total_committers":29,"mean_commits":"24.310344827586206","dds":0.8354609929078014,"last_synced_commit":"a69e935fa47cd4350abb4e2635da3309949fa047"},"previous_names":[],"tags_count":145,"template":false,"template_full_name":null,"purl":"pkg:github/uber/peloton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fpeloton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fpeloton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fpeloton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fpeloton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber","download_url":"https://codeload.github.com/uber/peloton/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Fpeloton/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259308162,"owners_count":22837974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T17:00:20.014Z","updated_at":"2025-06-11T17:40:16.195Z","avatar_url":"https://github.com/uber.png","language":"Go","readme":"# Peloton\n\nAs compute clusters scale, making efficient use of cluster resources\nbecomes very important. Peloton is a *Unified Resource Scheduler* to\nco-schedule mixed types of workloads such as batch, stateless and\nstateful jobs in a single cluster for better resource\nutilization. Peloton is designed for web-scale companies like Uber\nwith millions of containers and tens of thousands of nodes. Peloton\nfeatures advanced resource management capabilities such as elastic\nresource sharing, hierarchical max-min fairness, resource overcommit,\nworkload preemption, etc. Peloton is also Cloud agnostic and can be run\nin on-premise datacenters or in the Cloud.\n\nFor more details, please see the Peloton [Blog\nPost](https://eng.uber.com/peloton/) and\n[Documentation](https://peloton.readthedocs.io/en/latest/).\n\n## Features\n\n- **Elastic Resource Sharing**: Support hierachical resource pools to\n  elastically share resources among different teams.\n\n- **Resource Overcommit and Task Preemption**: Improve cluster\n  utilization by scheduling workloads using slack resources and\n  preempting best effort workloads.\n\n- **Optimized for Big Data and Machine Learning**: Support GPU and Gang\n  scheduling for Tensorflow. Also support advanced Spark features such\n  as dynamic resource allocation.\n\n- **High Scalability**: Scale to millions of containers and tens of\n  thousands of nodes.\n\n- **Protobuf/gRPC based API**: Support most of the language bindings\n  such as golang, java, python, node.js etc.\n\n- **Co-scheduling Mixed Workloads**: Support mixed workloads such as\n  batch, stateless and stateful jobs in a single cluster.\n\n\n## Getting Started\n\nSee the [Tutorial](docs/tutorial.md) for step-by-step\ninstructions to start a local minicluster and submit a HelloWorld job\nto Peloton.\n\n## Architecture\nTo achieve high-availability and scalability, Peloton uses an\nactive-active architecture with four separate daemon types: job\nmanager, resource manager, placement engine, and host manager. The\ninteractions among those daemons are designed so that the dependencies\nare minimized and only occur in one direction. All four daemons depend\non Zookeeper for service discovery and leader election.\n\nFigure , below, shows the high-level architecture of Peloton built on\ntop of Mesos, Zookeeper, and Cassandra:\n![image](docs/figures/architecture.png)\n\n\n### Components:\nPeloton consists of the following components:\n\n- **Peloton UI**: is web-based UI for managing jobs, tasks, volumes, and\n  resource pools in Peloton.\n- **Peloton CLI**: is command-line interface for Peloton with similar\n  functionality to the web-based interface.\n- **Peloton API**: uses Protocol Buffers as the interface definition\n  language and YARPC as its RPC runtime. Peloton UI, Peloton CLI, and\n  other Peloton extensions are all built on top of the same Peloton\n  API.\n- **Host Manager**: abstracts away Mesos details from other Peloton\n  components. It registers with Mesos via Mesos HTTP API.\n- **Resource Manager**: maintains the resource pool hierarchy and\n  periodically calculates the resource entitlement of each resource\n  pool, which is then used to schedule or preempt tasks\n  correspondingly.\n- **Placement Engine**: finds the placement (i.e., task to host mapping)\n  by taking into consideration the job and task constraints as well as\n  host attributes. Placement engines could be pluggable for different\n  job types such as stateful services and batch jobs.\n- **Job Manager**: handles the lifecycle of jobs, tasks, and volumes. It\n  also supports rolling upgrades of tasks in a job for long-running\n  services.\n- **Storage Gateway**: provides an abstraction layer on top of different\n  storage backends so that we can migrate from one storage backend to\n  another without significant change in Peloton itself. We have a\n  default backend for Cassandra built-in, but can extend it to other\n  backends.\n- **Group Membership**: manages the set of Peloton master instances and\n  elects a leader to both register to Mesos as a framework and\n  instantiate a resource manager.\n\n## References\n\n### User Guide\nSee the [User Guide](docs/user-guide.md) for more detailed\ninformation on how to use Peloton.\n\n### Peloton CLI\nPeloton CLI is a command line interface for interacting with Peloton\nclusters, such as creating jobs, check job status etc. For detailed\nPeloton CLI commands and arguments, see [CLI Reference](docs/cli-reference.md).\n\n\n### Peloton API\nPeloton defines the APIs using Protobuf as the IDL and the clients can\naccess Peloton API via gRPC. Peloton supports three client bindings by\ndefault including Python, Golang and Java. Any other language bindings\nsupported by gRPC should work as well. \n\nSee the [API Guide](docs/api-guide.md) for examples of how to use\nPeloton clients to access the APIs. For detailed Peloton API\ndefinition, see the [API Reference](docs/api-reference.md).\n\n## Contributing\nSee the [Developer Guide](docs/developer-guide.md) on how to build Peloton\nfrom source code.\n\n\n## Resources\n\n### Documentation\n\n- [Peloton Documentation](https://peloton.readthedocs.io/en/latest/)\n\n### Blogs\n\n- [Peloton blog from Uber Engineering](https://eng.uber.com/peloton/)\n\n### Tech Talks\n\n- [KubeCon 2018, Seattle](https://sched.co/GrTx) ([Slides](https://schd.ws/hosted_files/kccna18/29/Peloton_%20KubeCon%202018.pdf), [Video](https://www.youtube.com/watch?v=USgbj87Ztlk))\n- [MesosCon 2018, San Francisco](https://sched.co/HYTn)\n- [MesosCon 2017, Los Angeles](https://sched.co/BYaD) ([Slides](http://events17.linuxfoundation.org/sites/events/files/slides/MesosCon_Distributed_Deep_Learning_v4.pdf), [Video](https://www.youtube.com/watch?v=Ktc3GjshHcc))\n\n## Contact\nTo contact us, please join our [Slack channel](https://join.slack.com/t/peloton-scheduler/shared_invite/enQtNTU5ODIzOTY4NDMyLTY2NmEwN2Y5ZGMzMzk4ZDE1NTNlYjE4M2E4Njg2OWY1MTk5M2U2NTgxZDUyMjAwYWE4ZjY5M2JlZGU1ZTc2MjI).\n\n## License\nPeloton is under the Apache 2.0 license. See the LICENSE file for details.\n","funding_links":[],"categories":["Scheduler","others","Platforms and microservice architectures","Packages"],"sub_categories":["Networking","Tools"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fpeloton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber%2Fpeloton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Fpeloton/lists"}