{"id":13586331,"url":"https://github.com/uber/fiber","last_synced_at":"2026-01-14T07:18:09.814Z","repository":{"id":40956539,"uuid":"232387380","full_name":"uber/fiber","owner":"uber","description":"Distributed Computing for AI Made Simple","archived":true,"fork":false,"pushed_at":"2023-03-19T22:55:22.000Z","size":8759,"stargazers_count":1047,"open_issues_count":26,"forks_count":110,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-09-27T07:34:32.440Z","etag":null,"topics":["distributed-computing","machine-learning","multiprocessing","python","sandbox"],"latest_commit_sha":null,"homepage":"https://uber.github.io/fiber/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uber.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-01-07T18:16:24.000Z","updated_at":"2025-09-11T12:37:48.000Z","dependencies_parsed_at":"2024-01-15T20:47:17.808Z","dependency_job_id":"f050e628-8328-40fe-a85d-e1805f5ce783","html_url":"https://github.com/uber/fiber","commit_stats":{"total_commits":65,"total_committers":5,"mean_commits":13.0,"dds":"0.16923076923076918","last_synced_commit":"ad6faf02b8e94dee498990e9fd9c588234666725"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/uber/fiber","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Ffiber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Ffiber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Ffiber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Ffiber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uber","download_url":"https://codeload.github.com/uber/fiber/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uber%2Ffiber/sbom","scorecard":{"id":906231,"data":{"date":"2025-08-11","repo":{"name":"github.com/uber/fiber","commit":"ad6faf02b8e94dee498990e9fd9c588234666725"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.8,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 2/23 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating python:3.6.8-stretch to python:3.6.8-stretch@sha256:c1b1294103262879af89ee2da9bbfbc1ded3bf6dae7dca499aaf16637916cb62","Warn: containerImage not pinned by hash: examples/gecco-2020/Dockerfile:1: pin your Docker image by updating python:3.6 to python:3.6@sha256:f8652afaf88c25f0d22354d547d892591067aa4026a7fa9a6819df9f300af6fc","Warn: pipCommand not pinned by hash: Dockerfile:2","Warn: pipCommand not pinned by hash: Dockerfile:4","Warn: pipCommand not pinned by hash: Dockerfile:5","Warn: pipCommand not pinned by hash: examples/gecco-2020/Dockerfile:2","Info:   0 out of   2 containerImage dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 21 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"50 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2022-42986 / GHSA-43fp-rhv2-5gv8","Warn: Project is vulnerable to: PYSEC-2023-135 / GHSA-xqr8-7jwr-rhp7","Warn: Project is vulnerable to: PYSEC-2022-42991 / GHSA-v3c5-jqr6-7qm8","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: PYSEC-2022-288 / GHSA-6hrg-qmvc-2xh8","Warn: Project is vulnerable to: PYSEC-2021-356 / GHSA-2ww3-fxvq-293j","Warn: Project is vulnerable to: PYSEC-2024-167 / GHSA-cgvx-9447-vcch","Warn: Project is vulnerable to: PYSEC-2021-859 / GHSA-f8m6-h2c7-8h9x","Warn: Project is vulnerable to: PYSEC-2022-5 / GHSA-rqjh-jp2r-59cj","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2022-42969","Warn: Project is vulnerable to: PYSEC-2023-117 / GHSA-mrwq-x4v8-fh7p","Warn: Project is vulnerable to: GHSA-jh85-wwv9-24hv","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: GHSA-753j-mpmx-qq6g","Warn: Project is vulnerable to: GHSA-7cx3-6m66-7c5m","Warn: Project is vulnerable to: GHSA-8w49-h785-mj3c","Warn: Project is vulnerable to: PYSEC-2023-75 / GHSA-hj3f-6gcp-jg8j","Warn: Project is vulnerable to: GHSA-qppv-j76h-2rpx","Warn: Project is vulnerable to: GHSA-w235-7p84-xx57","Warn: Project is vulnerable to: GHSA-g7vv-2v7x-gj9p","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2021-59 / GHSA-5phf-pp7p-vc2r","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2021-108 / GHSA-q2q7-5pp4-w6pg","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f","Warn: Project is vulnerable to: GHSA-jfmj-5v4g-7637","Warn: Project is vulnerable to: PYSEC-2018-34 / GHSA-2fc2-6r4j-p65h","Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: PYSEC-2019-108 / GHSA-9fq2-x9r6-wfmf","Warn: Project is vulnerable to: PYSEC-2018-33 / GHSA-cw6w-4rcx-xphc","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: PYSEC-2017-1 / GHSA-frgw-fgh6-9g52","Warn: Project is vulnerable to: PYSEC-2019-41 / GHSA-qfc5-mcwq-26q8","Warn: Project is vulnerable to: PYSEC-2014-14 / GHSA-652x-xj99-gmcc","Warn: Project is vulnerable to: PYSEC-2014-13 / GHSA-cfj3-7x9c-4p3h","Warn: Project is vulnerable to: PYSEC-2018-28 / GHSA-x84v-xcm2-53pg","Warn: Project is vulnerable to: PYSEC-2020-96 / GHSA-6757-jp84-gxfx","Warn: Project is vulnerable to: PYSEC-2021-142 / GHSA-8q59-q68h-6hv4","Warn: Project is vulnerable to: PYSEC-2020-99 / GHSA-537h-rv9q-vvph","Warn: Project is vulnerable to: PYSEC-2020-100 / GHSA-xrx6-fmxq-rjj2","Warn: Project is vulnerable to: PYSEC-2020-148 / GHSA-wqvq-5m8c-6g24"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-24T17:26:47.028Z","repository_id":40956539,"created_at":"2025-08-24T17:26:47.028Z","updated_at":"2025-08-24T17:26:47.028Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28412800,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","machine-learning","multiprocessing","python","sandbox"],"created_at":"2024-08-01T15:05:29.056Z","updated_at":"2026-01-14T07:18:09.775Z","avatar_url":"https://github.com/uber.png","language":"Python","readme":"\u003cp align=\"right\"\u003e\n  \u003ca href=\"https://travis-ci.com/uber/fiber\"\u003e\n      \u003cimg src=\"https://travis-ci.com/uber/fiber.svg?token=BxMzxQEDDtTBPG9151kk\u0026branch=master\" alt=\"build\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cimg src=\"docs/img/fiber_logo.png\" alt=\"drawing\" width=\"550\"/\u003e\n\n[**Project Home**](https://uber.github.io/fiber/) \u0026nbsp;\n[**Blog**](https://eng.uber.com/fiberdistributed/) \u0026nbsp;\n[**Documents**](https://uber.github.io/fiber/getting-started/) \u0026nbsp;\n[**Paper**](https://arxiv.org/abs/2003.11164) \u0026nbsp;\n[**Media Coverage**](https://venturebeat.com/2020/03/26/uber-details-fiber-a-framework-for-distributed-ai-model-training/)\n\n\u003cimg src=\"https://github.com/uber/fiber/raw/docs/email-list/docs/img/new-icon.png\"/\u003e Join Fiber users email list [fiber-users@googlegroups.com](https://groups.google.com/forum/#!forum/fiber-users)\n\n# Fiber\n\n### Distributed Computing for AI Made Simple\n\n*This project is experimental and the APIs are not considered stable.*\n\nFiber is a Python distributed computing library for modern computer clusters.\n\n* It is easy to use. Fiber allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.\n* It is easy to learn. Fiber provides the same API as Python's standard [multiprocessing](https://docs.python.org/3.6/library/multiprocessing.html) library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.\n* It is fast. Fiber's communication backbone is built on top of [Nanomsg](https://nanomsg.org/) which is a high-performance asynchronous messaging library to allow fast and reliable communication.\n* It doesn't need deployment. You run it as the same way as running a normal application on a computer cluster and Fiber handles the rest for you.\n* It it reliable. Fiber has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.\n\nOriginally, it was developed to power large scale parallel scientific computation projects like [POET](https://eng.uber.com/poet-open-ended-deep-learning/) and it has been used to power similar projects within Uber.\n\n\n## Installation\n\n```\npip install fiber\n```\n\nCheck [here](https://uber.github.io/fiber/installation/) for details.\n\n## Quick Start\n\n\n### Hello Fiber\nTo use Fiber, simply import it in your code and it works very similar to multiprocessing.\n\n```python\nimport fiber\n\nif __name__ == '__main__':\n    fiber.Process(target=print, args=('Hello, Fiber!',)).start()\n```\n\nNote that `if __name__ == '__main__':` is necessary because Fiber uses *spawn* method to start new processes. Check [here](https://stackoverflow.com/questions/50781216/in-python-multiprocessing-process-do-we-have-to-use-name-main) for details.\n\nLet's take look at another more complex example:\n\n### Estimating Pi\n\n\n```python\nimport fiber\nimport random\n\n@fiber.meta(cpu=1)\ndef inside(p):\n    x, y = random.random(), random.random()\n    return x * x + y * y \u003c 1\n\ndef main():\n    NUM_SAMPLES = int(1e6)\n    pool = fiber.Pool(processes=4)\n    count = sum(pool.map(inside, range(0, NUM_SAMPLES)))\n    print(\"Pi is roughly {}\".format(4.0 * count / NUM_SAMPLES))\n\nif __name__ == '__main__':\n    main()\n```\n\n\nFiber implements most of multiprocessing's API including `Process`, `SimpleQueue`, `Pool`, `Pipe`, `Manager` and it has its own extension to the multiprocessing's API to make it easy to compose large scale distributed applications. For the detailed API guild, check out [here](https://uber.github.io/fiber/process/).\n\n### Running on a Kubernetes cluster\n\nFiber also has native support for computer clusters. To run the above example on Kubernetes, fiber provided a convenient command line tool to manage the workflow.\n\nAssume you have a working docker environment locally and have finished configuring [Google Cloud SDK](https://cloud.google.com/sdk/docs/quickstarts). Both `gcloud` and `kubectl` are available locally. Then you can start by writing a Dockerfile which describes the running environment.  An example Dockerfile looks like this:\n\n```dockerfile\n# example.docker\nFROM python:3.6-buster\nADD examples/pi_estimation.py /root/pi_estimation.py\nRUN pip install fiber\n```\n**Build an image and launch your job**\n\n```\nfiber run -a python3 /root/pi_estimation.py\n```\n\nThis command will look for local Dockerfile and build a docker image and push it to your Google Container Registry . It then launches the main job which contains your code and runs the command `python3 /root/pi_estimation.py` inside your job. Once the main job is running, it will start 4 subsequent jobs on the cluster and each of them is a Pool worker.\n\n\n## Supported platforms\n\n* Operating system: Linux\n* Python: 3.6+\n* Supported cluster management systems:\n\t* Kubernetes (Tested with Google Kubernetes Engine on Google cloud)\n\nWe are interested in supporting other cluster management systems like [Slurm](https://slurm.schedmd.com/), if you want to contribute to it please let us know.\n\n\nCheck [here](https://uber.github.io/fiber/platforms/) for details.\n\n## Documentation\n\nThe documentation, including method/API references, can be found [here](https://uber.github.io/fiber/getting-started/).\n\n\n## Testing\n\nInstall test dependencies. You'll also need to make sure [docker](https://docs.docker.com/install/) is available on the testing machine.\n\n```bash\n$ pip install -e .[test]\n```\n\nRun tests\n\n```bash\n$ make test\n```\n\n## Contributing\nPlease read our [code of conduct](CODE_OF_CONDUCT.md) before you contribute! You can find details for submitting pull requests in the [CONTRIBUTING.md](CONTRIBUTING.md) file. Issue [template](https://help.github.com/articles/about-issue-and-pull-request-templates/).\n\n## Versioning\nWe document versions and changes in our changelog - see the [CHANGELOG.md](CHANGELOG.md) file for details.\n\n## License\nThis project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.\n\n## Cite Fiber\n\n```\n@misc{zhi2020fiber,\n    title={Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods},\n    author={Jiale Zhi and Rui Wang and Jeff Clune and Kenneth O. Stanley},\n    year={2020},\n    eprint={2003.11164},\n    archivePrefix={arXiv},\n    primaryClass={cs.LG}\n}\n```\n\n## Acknowledgments\n* Special thanks to Piero Molino for designing the logo for Fiber\n","funding_links":[],"categories":["Python","分布式机器学习","Computation Load Distribution"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Ffiber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuber%2Ffiber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuber%2Ffiber/lists"}