{"id":13908583,"url":"https://github.com/mojtaba-arvin/video-service","last_synced_at":"2025-04-22T07:31:49.415Z","repository":{"id":53197333,"uuid":"340950715","full_name":"mojtaba-arvin/video-service","owner":"mojtaba-arvin","description":"Python Video Streaming microervice | supports HLS, MPEG-DASH and HLS with fmp4 segments (CMAF), this project uses gRPC protocol for communication and S3-compatible object storage. The multi-stage dockerfile of project uses Python3.9.2 and FFmpeg4.1","archived":false,"fork":false,"pushed_at":"2021-04-01T10:11:50.000Z","size":414,"stargazers_count":35,"open_issues_count":0,"forks_count":7,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2024-08-07T23:57:31.384Z","etag":null,"topics":["cloud-video-processing","docker-compose","ffmpeg","grpc","grpc-video-service","hls","http-live-streaming","mp4-to-hls-fmp4","mpeg-dash","object-storage","python","python-ffmpeg","python-ffmpeg-video-streaming","python-hls-service","python-video-streaming","transcoding","video","video-processing-service","video-streaming","vod"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mojtaba-arvin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-21T16:42:37.000Z","updated_at":"2024-06-13T20:30:11.000Z","dependencies_parsed_at":"2022-09-25T03:20:14.817Z","dependency_job_id":null,"html_url":"https://github.com/mojtaba-arvin/video-service","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mojtaba-arvin%2Fvideo-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mojtaba-arvin%2Fvideo-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mojtaba-arvin%2Fvideo-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mojtaba-arvin%2Fvideo-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mojtaba-arvin","download_url":"https://codeload.github.com/mojtaba-arvin/video-service/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223892218,"owners_count":17220729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-video-processing","docker-compose","ffmpeg","grpc","grpc-video-service","hls","http-live-streaming","mp4-to-hls-fmp4","mpeg-dash","object-storage","python","python-ffmpeg","python-ffmpeg-video-streaming","python-hls-service","python-video-streaming","transcoding","video","video-processing-service","video-streaming","vod"],"created_at":"2024-08-06T23:02:50.676Z","updated_at":"2024-11-09T22:32:16.981Z","avatar_url":"https://github.com/mojtaba-arvin.png","language":"Python","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# **[Python Video Streaming Microservice](https://github.com/mojtaba-arvin/video-service)**\n\n![video-service](https://user-images.githubusercontent.com/6056661/110987240-0d7ab780-8384-11eb-8e20-05a0a9144d35.png)\n\nPython video streaming microservice, allows you to change the number of qualities or formats,\nto reduce processing and storage space costs.\n\nFor example, instead of building both HLS and MPEG-DASH playlists, you can build a HLS with `fmp4`, that similar to `MPEG-DASH`, which reduces costs by 50%. \n\nThe multi-stage dockerfile of the project uses `Python3.9.2` and **[FFmpeg](https://ffmpeg.org)** 4.1 that supports `fmp4` HLS segment type.\n\nFeatures:\n* Supports multi outputs with custom encode formats, codecs and qualities\n* Supports HLS, MPEG-DASH and HLS with fmp4 segments (CMAF)\n* Supports any S3-compatible object storage like Minio object storage\n* Supports **[gRPC](https://grpc.io/)** protocol for low latency and high throughput communication\n* No use of web frameworks to avoid unnecessary abstractions and dependencies\n* Supports tracking of outputs status and returns progress details of each input or output, including **checking, processing and uploading progress**\n* Supports force stop a list of jobs, to kill processes and delete all local files of every job. or revoke one output of a job\n* All tasks are separate to manage and retry each part again in some exceptions\n* Supports webhook callback when all outputs uploaded\n* Returns input video file details to client (using FFprobe), before video processing starting. ( client can use this information to show to the end user)\n* Returns **CPU and memory usage** with spent time of every output for the financial purposes when a video is being processed to create a playlist \n* Supports to generate different thumbnails from the input video by list of times, to choose one of them by user as a player poster or other purpose such as screenshots for demo \n* Supports to add watermark\n\nTODO\n* adding optional argument to mapping playlist qualities to different video files\n* adding gRPC client sample and test cases\n* update document\n\n## Setup\n\n### 1. Requirements\n\n * a **[Redis](https://redis.io/)** service ( to cache result and as a backend for celery )\n * a message broker ( e.g. **[RabbitMQ](https://www.rabbitmq.com/)** or Redis )\n * an S3-compatible object storage ( e.g. **[Minio](https://min.io/)** )\n\n\nif you have not already, a Redis, a broker and an S3 object storage,\nyou can `clone` these repositories:\n\n* Redis: **[Simple Redis service](https://github.com/mojtaba-arvin/redis)**\n  ```\n  git clone https://github.com/mojtaba-arvin/redis.git\n  ```\n* RabbitMQ: **[Simple RabbitMQ service](https://github.com/mojtaba-arvin/rabbitmq)**\n  ```\n  git clone https://github.com/mojtaba-arvin/rabbitmq.git\n  ```\n* Minio: **[Simple S3-compatible object storage service using Minio](https://github.com/mojtaba-arvin/minio)**\n  ```\n  git clone https://github.com/mojtaba-arvin/minio.git\n  ```\n\n\n#### up workflow\n\n![build-workflow](https://user-images.githubusercontent.com/6056661/111037785-f64ad100-843a-11eb-9450-3197e05fc696.png)\n\n\n### 2. Setup video service project environments\n\nYou should create an `.env` file at `video-streaming/video_streaming`\n. There is a sample of required environments that you can use it:\n```\ncp video-streaming/video_streaming/.env.local video-streaming/video_streaming/.env\n```\nThe project uses `python-decouple` package, you can add other variables and cast them in `settings.py`. or anywhere in project using `RepositoryEnv` class\n\n|    | VARIABLE                         | DESCRIPTION                                                                 |\n|----|----------------------------------|-----------------------------------------------------------------------------|\n| 1  | CELERY_BROKER_URL                | Celery needs a message broker url, e.g. RabbitMQ url                        |\n| 2  | CELERY_RESULT_BACKEND            | To keep track of Celery tasks results, e.g. Redis url                       |\n| 3  | TASK_RETRY_BACKOFF_MAX           |                                                                             |\n| 4  | TASK_RETRY_FFMPEG_COMMAND_MAX    |                                                                             |\n| 5  | S3_ENDPOINT_URL                  |                                                                             |\n| 6  | S3_ACCESS_KEY_ID                 |                                                                             |\n| 7  | S3_SECRET_ACCESS_KEY             |                                                                             |\n| 8  | S3_REGION_NAME                   |                                                                             |\n| 9  | S3_IS_SECURE                     | Default is False but note that not all services support non-ssl connections.|\n| 10 | S3_DEFAULT_BUCKET                | Default bucket name                                                         |\n| 11 | S3_DEFAULT_INPUT_BUCKET_NAME     | Default bucket name of S3 storage to download videos                        |\n| 12 | S3_DEFAULT_OUTPUT_BUCKET_NAME    | Default bucket name of S3 storage to upload videos                          |\n| 13 | S3_TRANSFER_MULTIPART_THRESHOLD  |                                                                             |\n| 14 | S3_TRANSFER_MAX_CONCURRENCY      |                                                                             |\n| 15 | S3_TRANSFER_MULTIPART_CHUNKSIZE  |                                                                             |\n| 16 | S3_TRANSFER_NUM_DOWNLOAD_ATTEMPTS|                                                                             |\n| 17 | S3_TRANSFER_MAX_IO_QUEUE         |                                                                             |\n| 18 | S3_TRANSFER_IO_CHUNKSIZE         |                                                                             |\n| 19 | S3_TRANSFER_USE_THREADS          |                                                                             |\n| 20 | TMP_DOWNLOADED_DIR               | Directory for temporary downloaded videos (should be volume on compose file)|                                                                             |\n| 21 | TMP_TRANSCODED_DIR               | Directory for temporary transcoded videos (should be volume on compose file)|\n| 22 | FFPROBE_BIN_PATH                 |                                                                             |\n| 23 | FFMPEG_BIN_PATH                  |                                                                             |\n| 24 | REDIS_TIMEOUT_SECOND             |                                                                             |\n| 25 | REDIS_URL                        | Redis url                                                                   |\n\n\n### 3. Generate Certificates to use by gRPC\nTODO\n\n### 4. Config circus\n\nThis project uses `circusd`, to manage processes, \n\n`.circus.ini` file is git ignored, you need have a `.circus.ini` at `.docker-compose/video-streaming/circus/` directory. \n\nfor local development you can use the sample by following command:\n```\ncp .docker-compose/video-streaming/circus/circus.local.ini .docker-compose/video-streaming/circus/circus.ini\n```\nthere are some variables in the `[env]` section:\n\n* `WORKING_DIR`: The path on the container that main module is located.\n* `MODULE_NAME`: The name of main module of project.\n* `CELERY_APP`: The name of celery instance in the main module.\n* `BIN_PATH`: Python installed at `/usr/local/bin/` in the Python Docker Official Image.\n* `GRPC_PORT`: The gRPC port, if you change it, make sure it's exposed on your network.\n  \nafter any change in `.circus.ini` you need to build image again.\n\n### 5. Docker composes\n\nin the first time, before up using `docker-compose.join.*` files, create external networks by build **[Redis](https://github.com/mojtaba-arvin/redis)**, **[RabbitMQ](https://github.com/mojtaba-arvin/rabbitmq)** and **[Minio](https://github.com/mojtaba-arvin/minio)** projects.\n\n|    | Compose file                     | Description                                                                 |\n|----|----------------------------------|-----------------------------------------------------------------------------|\n| 1  | docker-compose.yml               | includes `video-streaming` service that has `WAIT_FOR` environment variable |\n| 2  | docker-compose.local.yml         | for local development, maps exposed gRPC port to `8081` as your host port   |\n| 3  | docker-compose.join.redis.yml    | to join the Redis service network as an external network. see : **[Redis service](https://github.com/mojtaba-arvin/redis)**|\n| 4  | docker-compose.join.rabbitmq.yml | to join the RabbitMQ service network as an external network. see : **[RabbitMQ service](https://github.com/mojtaba-arvin/rabbitmq)**|\n| 5  | docker-compose.join.minio.yml    | to join the Minio service network as an external network. see : **[Minio service](https://github.com/mojtaba-arvin/minio)**|\n\nthere are some `sh` scripts in the root directory of this repository, that you can use them:\n\n* Services are built once\n  ```\n  bash ./build.sh\n  ```\n* Builds, (re)creates, starts, and attaches to containers for a service.\n  ```\n  bash ./up.sh\n  ```\n* like `up.sh` but for local development, if you are not using a reverse proxy, you can use `up.local.sh`\n  that maps exposed grpc port to `8081` as your host port\n  ```\n  bash ./up.local.sh\n  ```\n* Shows services states\n  ```\n  bash ./ps.sh\n  ```\n* Displays log output from services.\n  ```\n  bash ./logs.sh\n  ```\n* To get an interactive prompt in `video-streaming` service\n  ```\n  bash ./exec.sh\n  ```\n* Stops containers and removes containers, networks, volumes, and images created by `up`.\n  ```\n  bash ./down.sh\n  ```\n\n### 6. Generate gRPC modules\n\ngenerated grpc modules are added to `.gitignore`, to generate them again, you can use the following command:\n```\nbash ./exec.codegen.sh\n```\nit runs `generate_grpc_codes.sh` inside `video-streaming` container that also will change import statement to fix `ModuleNotFoundError`. \n\n* after any changes on the gRPC proto file, you need run the script again.\n\n### APPs\n\napps located at `video-streaming/video_streaming/`\n\n\n|    | APP_NAME    | DESCRIPTION                         |\n|----|-------------|-------------------------------------|\n| 1  | core        | base classes and common modules     |\n| 2  | grpc        | the inclusion root of gRPC          |\n| 3  | ffmpeg      | video processing tasks using ffmpeg |\n\nafter create a new app, to discover celery tasks, add the app to `AUTO_DISCOVER_TASKS` in `settings.py`.\n\n### celery job workflow\n\n![celery tasks](https://user-images.githubusercontent.com/6056661/111865295-7f05c780-897b-11eb-9b34-5bc3bbf0662a.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmojtaba-arvin%2Fvideo-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmojtaba-arvin%2Fvideo-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmojtaba-arvin%2Fvideo-service/lists"}