{"id":31527819,"url":"https://github.com/petuum/autodist","last_synced_at":"2025-10-03T22:47:36.597Z","repository":{"id":44192370,"uuid":"275910898","full_name":"petuum/autodist","owner":"petuum","description":"Simple Distributed Deep Learning on TensorFlow","archived":false,"fork":false,"pushed_at":"2025-06-17T03:41:09.000Z","size":16169,"stargazers_count":134,"open_issues_count":20,"forks_count":25,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-10-03T22:47:35.382Z","etag":null,"topics":["aws","cloud","deep-learning","distributed-computing","distributed-systems","machine-learning","on-premise","tensorflow"],"latest_commit_sha":null,"homepage":"https://petuum.github.io/autodist","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/petuum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-06-29T19:45:38.000Z","updated_at":"2025-09-29T11:04:49.000Z","dependencies_parsed_at":"2022-08-19T13:30:26.286Z","dependency_job_id":"f278cd3a-30b6-45d4-9757-f7da2696a1e7","html_url":"https://github.com/petuum/autodist","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/petuum/autodist","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fautodist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fautodist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fautodist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fautodist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/petuum","download_url":"https://codeload.github.com/petuum/autodist/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/petuum%2Fautodist/sbom","scorecard":{"id":729631,"data":{"date":"2025-08-11","repo":{"name":"github.com/petuum/autodist","commit":"9bf89c90cb41eafa82d5660b979e7d0797e3cdf5"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.5,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":7,"reason":"Found 21/30 approved changesets -- score normalized to 7","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/docs.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/docs.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/docs.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/docs.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/docs.yml:59: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/docs.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/release.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/release.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:53: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/release.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:60: update your workflow using https://app.stepsecurity.io/secureworkflow/petuum/autodist/release.yml/master?enable=pin","Warn: containerImage not pinned by hash: docker/Dockerfile.gpu:5","Warn: pipCommand not pinned by hash: docker/Dockerfile.gpu:25","Warn: pipCommand not pinned by hash: .github/workflows/docs.yml:38","Warn: pipCommand not pinned by hash: .github/workflows/docs.yml:39","Warn: goCommand not pinned by hash: .github/workflows/docs.yml:40","Warn: pipCommand not pinned by hash: .github/workflows/docs.yml:46","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:27","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:28","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   1 out of   7 pipCommand dependencies pinned","Info:   0 out of   1 goCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 23 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T14:02:38.056Z","repository_id":44192370,"created_at":"2025-08-22T14:02:38.057Z","updated_at":"2025-08-22T14:02:38.057Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278239971,"owners_count":25954096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-03T02:00:06.070Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud","deep-learning","distributed-computing","distributed-systems","machine-learning","on-premise","tensorflow"],"created_at":"2025-10-03T22:47:33.822Z","updated_at":"2025-10-03T22:47:36.592Z","avatar_url":"https://github.com/petuum.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/_static/img/logo.png\" width=400 /\u003e\u003c/p\u003e\n\n[![pipeline status](https://img.shields.io/badge/dynamic/json?url=https://jenkins.petuum.io/job/AutoDist/job/master/lastCompletedBuild/api/json\u0026label=build\u0026query=$.result\u0026color=informational)](https://jenkins.petuum.io/job/AutoDist/job/master/)\n[![coverage report](https://img.shields.io/badge/dynamic/json?url=https://jenkins.petuum.io/job/AutoDist/job/master/lastSuccessfulBuild/artifact/coverage-report/jenkinscovdata.json\u0026label=coverage\u0026query=$.total_coverage_pct\u0026color=green)](https://jenkins.petuum.io/job/AutoDist/job/master/lastSuccessfulBuild/artifact/)\n[![pypi version](https://img.shields.io/pypi/v/autodist?color=9cf)](https://pypi.org/project/autodist/)\n\n[Documentation](https://petuum.github.io/autodist) |\n[Examples](https://github.com/petuum/autodist/tree/master/examples/benchmark)\n\n**AutoDist** is a distributed deep learning training engine for TensorFlow. AutoDist provides a user-friendly interface to distribute the training of a wide variety deep learning models across many GPUs with scalability and minimal code change.\n\n## Introduction\nDifferent from specialized distributed ML systems, AutoDist is created to speed up a broad range of DL models with excellent all-round performance.\nAutoDist achieves this goal by:\n- **Compilation**: AutoDist expresses the parallelization of DL models as a standardized compilation process, optimizing multiple dimensions of ML\nparallelization including synchronization, partitioning, placement etc.\n- **Composable architecture**: AutoDist contains a flexible backend that can express various different ML parallelization techniques and\nallows for composing distribution strategies that blend different distributed ML system architectures.\n- **Model and resource awareness**: Based on the compilation process, AutoDist analyzes the model and generates more optimal distribution strategies that\nadapt to both the model properties and the cluster specification.\n\nBesides all these advanced features, AutoDist is designed to isolate the sophistication of distributed systems\nfrom ML prototyping and exposes a simple API that makes it easy to use and switch between different distributed ML techniques\nfor users of all levels.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/_static/img/Figure1.png\" width=400 /\u003e\u003cimg src=\"docs/_static/img/Figure2.png\" width=400 /\u003e\u003c/p\u003e\n\nFor a closer look at the performance, please refer to our [doc](https://petuum.github.io/autodist/usage/performance.html).\n\n## Using AutoDist\n\nInstallation:\n\n```bash\npip install autodist\n```\n\nModifying existing TensorFlow code to use AutoDist is easy:\n\n```python\nimport tensorflow as tf\nfrom autodist import AutoDist\n\nad = AutoDist(resource_spec_file=\"resource_spec.yml\")\n\nwith tf.Graph().as_default(), ad.scope():\n    ########################################################\n    # Build your (single-device) model here,\n    #   and train it distributedly.\n    ########################################################\n    sess = ad.create_distributed_session()\n    sess.run(...)\n```\nReady to try? Please refer to the examples in our [Getting Started page](https://petuum.github.io/autodist/usage/tutorials/getting-started.html).\n\n## References \u0026 Acknowledgements\n\nWe learned and borrowed insights from a few open source projects\nincluding\n[Horovod](https://github.com/horovod/horovod),\n[Parallax](https://github.com/snuspl/parallax),\nand [tf.distribute](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/distribute).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetuum%2Fautodist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpetuum%2Fautodist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpetuum%2Fautodist/lists"}