{"id":15663357,"url":"https://github.com/minimaxir/tensorflow-multiprocess-ray","last_synced_at":"2025-05-06T11:36:23.122Z","repository":{"id":65982455,"uuid":"188287917","full_name":"minimaxir/tensorflow-multiprocess-ray","owner":"minimaxir","description":"Proof of concept on how to use TensorFlow for prediction tasks in a multiprocess setting.","archived":false,"fork":false,"pushed_at":"2019-05-24T16:48:54.000Z","size":11,"stargazers_count":18,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-25T16:44:15.521Z","etag":null,"topics":["multiprocessing","prediction","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minimaxir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-23T18:38:40.000Z","updated_at":"2025-01-10T14:52:41.000Z","dependencies_parsed_at":"2023-02-19T18:46:13.274Z","dependency_job_id":null,"html_url":"https://github.com/minimaxir/tensorflow-multiprocess-ray","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minimaxir%2Ftensorflow-multiprocess-ray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minimaxir%2Ftensorflow-multiprocess-ray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minimaxir%2Ftensorflow-multiprocess-ray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minimaxir%2Ftensorflow-multiprocess-ray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minimaxir","download_url":"https://codeload.github.com/minimaxir/tensorflow-multiprocess-ray/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242789177,"owners_count":20185400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["multiprocessing","prediction","tensorflow"],"created_at":"2024-10-03T13:36:53.983Z","updated_at":"2025-03-10T04:30:49.722Z","avatar_url":"https://github.com/minimaxir.png","language":"Python","funding_links":["https://www.patreon.com/minimaxir"],"categories":[],"sub_categories":[],"readme":"# TensorFlow Prediction Multiprocess w/ Ray\n\nA proof of concept on how to use TensorFlow for prediction tasks in a multiprocess setting with a few important features:\n\n* Can run all processes in a single thread and single machine. (e.g. good for [Cloud Run](https://cloud.google.com/run/) which may be limited to 1 vCPU)\n* Can run worker tasks asynchronously via `asyncio`. (e.g. model prediction)\n* Creates a single TensorFlow Session/Model to be shared by the worker processes w/ between-graph replication. (especially important if using a GPU!)\n\nThis POC leverages [ray](https://github.com/ray-project/ray) for its multiprocessing/colocation capabilities and Distributed TensorFlow for its session management (this is essentially a port of Matthew Rahtz's [Distributed TensorFlow Hello World](https://github.com/mrahtz/distributed_tensorflow_gentle_introduction) w/ ray + a few Pythonic improvements). It is incredibly hacky, but it is necessary since `tf.Session()` does not play nice with multiple threads and/or processes.\n\nNote: will likely not work with Python wrappers (including Jupyter Notebooks, Anaconda, and Server Process Managers), and the async may not work with existing async apps.\n\n## Files\n\n* `ray_tensorflow.py`: Single script for running multiple TensorFlow workers on a single thread, but can be configured to use more threads for proportionate performance increase. The script adds the `worker_id` value on each `Worker` to a sum on the `ParameterServer`.\n* `ray_tensorflow_flask.py`: Simple Flask app which allows parallel predictioning using round-robin queuing (adds a user-provided `value` to a persistant sum). The app can handle concurrent requests in threaded mode, which admittingly reduces the impact of the implementation.\n* `ray_tensorflow_starlette.py`: Prototype of the Flask app which doesn't use async ray due to the event loop issues, and as a result is less effective.\n\n## Maintainer/Creator\n\nMax Woolf ([@minimaxir](https://minimaxir.com))\n\n*Max's open-source projects are supported by his [Patreon](https://www.patreon.com/minimaxir). If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.*\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminimaxir%2Ftensorflow-multiprocess-ray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminimaxir%2Ftensorflow-multiprocess-ray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminimaxir%2Ftensorflow-multiprocess-ray/lists"}