{"id":22018494,"url":"https://github.com/flavienbwk/grpc-multiprocessing-streaming","last_synced_at":"2025-05-07T03:27:01.889Z","repository":{"id":47314929,"uuid":"402070509","full_name":"flavienbwk/gRPC-multiprocessing-streaming","owner":"flavienbwk","description":"Python implementation of a multi-processed gRPC client/server with streaming capabilities.","archived":false,"fork":false,"pushed_at":"2021-09-03T09:08:48.000Z","size":157,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-31T05:51:11.213Z","etag":null,"topics":["docker","grpc","multi-processing","multi-threading","python","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flavienbwk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-01T13:26:48.000Z","updated_at":"2025-03-03T08:13:05.000Z","dependencies_parsed_at":"2022-09-08T18:01:02.903Z","dependency_job_id":null,"html_url":"https://github.com/flavienbwk/gRPC-multiprocessing-streaming","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FgRPC-multiprocessing-streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FgRPC-multiprocessing-streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FgRPC-multiprocessing-streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flavienbwk%2FgRPC-multiprocessing-streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flavienbwk","download_url":"https://codeload.github.com/flavienbwk/gRPC-multiprocessing-streaming/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252805805,"owners_count":21807083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","grpc","multi-processing","multi-threading","python","streaming"],"created_at":"2024-11-30T05:12:20.287Z","updated_at":"2025-05-07T03:27:01.869Z","avatar_url":"https://github.com/flavienbwk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gRPC multiprocessing with streaming\n\nThis repo is a boilerplate showing how you can use multiprocessing with gRPC with streaming capabilities in Python\n\nPayloads streaming allows **bypassing [gRPC's 2Gb limit](https://stackoverflow.com/questions/34128872/google-protobuf-maximum-size)** in its Python implementation.\n\n## Getting started\n\nThis project offers a minimum reproducible example of clients sending batches of videos to gRPC servers.\n\n- Input : video(s)\n- Output : thumbnail(s)\n\n`client.py` starts a pool of several gRPC channels. Each client process is attributed a video and cuts it into chunks that are sent to one of the gRPC servers parallely to other client processes.\n\nEach response including the video thumbnail gets returned and cut if exceeding 2Gb as well.\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"./flow.png\" /\u003e\u003c/div\u003e\n\n1. Adjust the `NUM_WORKERS` and `NUM_JOBS` variables in `docker-compose.yml` if desired\n\n2. Add your videos in the `input/` directory. Thumbnails will be saved in `output/`.\n\n  ```bash\n  # Download 2 videos (to test multiprocessing)\n  wget -O ./input/video.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4\n  wget -O ./input/video_2.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4\n  ```\n\n3. Start playing around by running :\n\n  ```bash\n  docker-compose build\n  docker-compose run client\n  ```\n\n## Re-compile proto\n\nYou must have `python3` and `pip` installed\n\n```bash\n# In project root directory\npython3 -m pip install grpcio==1.32.0 protobuf==3.12.2\npython3 -m grpc_tools.protoc --proto_path=./grpc_config --python_out=./grpc_config --grpc_python_out=./grpc_config ./grpc_config/video_thumbnail.proto\n```\n\n## Credits\n\n- This repo strongly builds on the great [fpaupier's gRPC-multiprocessing repo](https://github.com/fpaupier/gRPC-multiprocessing)\n- [Implementing gRPC in Python by Velotio](https://www.velotio.com/engineering-blog/grpc-implementation-using-python)\n- Video sample from [file-examples.com](https://file-examples.com/index.php/sample-video-files/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fgrpc-multiprocessing-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflavienbwk%2Fgrpc-multiprocessing-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflavienbwk%2Fgrpc-multiprocessing-streaming/lists"}