Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/flavienbwk/grpc-multiprocessing-streaming
Python implementation of a multi-processed gRPC client/server with streaming capabilities.
https://github.com/flavienbwk/grpc-multiprocessing-streaming
docker grpc multi-processing multi-threading python streaming
Last synced: 2 months ago
JSON representation
Python implementation of a multi-processed gRPC client/server with streaming capabilities.
- Host: GitHub
- URL: https://github.com/flavienbwk/grpc-multiprocessing-streaming
- Owner: flavienbwk
- License: mit
- Created: 2021-09-01T13:26:48.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-09-03T09:08:48.000Z (over 3 years ago)
- Last Synced: 2024-05-01T19:07:35.297Z (9 months ago)
- Topics: docker, grpc, multi-processing, multi-threading, python, streaming
- Language: Python
- Homepage:
- Size: 153 KB
- Stars: 5
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# gRPC multiprocessing with streaming
This repo is a boilerplate showing how you can use multiprocessing with gRPC with streaming capabilities in Python
Payloads streaming allows **bypassing [gRPC's 2Gb limit](https://stackoverflow.com/questions/34128872/google-protobuf-maximum-size)** in its Python implementation.
## Getting started
This project offers a minimum reproducible example of clients sending batches of videos to gRPC servers.
- Input : video(s)
- Output : thumbnail(s)
`client.py` starts a pool of several gRPC channels. Each client process is attributed a video and cuts it into chunks that are sent to one of the gRPC servers parallely to other client processes.
Each response including the video thumbnail gets returned and cut if exceeding 2Gb as well.
1. Adjust the `NUM_WORKERS` and `NUM_JOBS` variables in `docker-compose.yml` if desired
2. Add your videos in the `input/` directory. Thumbnails will be saved in `output/`.
```bash
# Download 2 videos (to test multiprocessing)
wget -O ./input/video.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
wget -O ./input/video_2.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
```
3. Start playing around by running :
```bash
docker-compose build
docker-compose run client
```
## Re-compile proto
You must have `python3` and `pip` installed
```bash
# In project root directory
python3 -m pip install grpcio==1.32.0 protobuf==3.12.2
python3 -m grpc_tools.protoc --proto_path=./grpc_config --python_out=./grpc_config --grpc_python_out=./grpc_config ./grpc_config/video_thumbnail.proto
```
## Credits
- This repo strongly builds on the great [fpaupier's gRPC-multiprocessing repo](https://github.com/fpaupier/gRPC-multiprocessing)
- [Implementing gRPC in Python by Velotio](https://www.velotio.com/engineering-blog/grpc-implementation-using-python)
- Video sample from [file-examples.com](https://file-examples.com/index.php/sample-video-files/)