{"id":20156099,"url":"https://github.com/openfun/django-peertube-runner-connector","last_synced_at":"2025-11-06T22:02:42.093Z","repository":{"id":193901839,"uuid":"688378176","full_name":"openfun/django-peertube-runner-connector","owner":"openfun","description":"django-peertube-runner-connector is a project allowing a django application to connect to a peertube runner","archived":false,"fork":false,"pushed_at":"2025-04-09T13:01:44.000Z","size":153,"stargazers_count":5,"open_issues_count":3,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-09T14:24:26.171Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openfun.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-07T08:18:03.000Z","updated_at":"2025-03-13T10:08:40.000Z","dependencies_parsed_at":"2023-11-15T11:30:47.439Z","dependency_job_id":"3440fd83-fd35-4636-9643-88a80c8a648a","html_url":"https://github.com/openfun/django-peertube-runner-connector","commit_stats":null,"previous_names":["openfun/django-peertube-runner-connector"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfun%2Fdjango-peertube-runner-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfun%2Fdjango-peertube-runner-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfun%2Fdjango-peertube-runner-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openfun%2Fdjango-peertube-runner-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openfun","download_url":"https://codeload.github.com/openfun/django-peertube-runner-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054676,"owners_count":21040043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T23:37:34.490Z","updated_at":"2025-11-06T22:02:37.060Z","avatar_url":"https://github.com/openfun.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# django-peertube-runner-connector: A django application to connect to a peertube runner and transcode videos \n\n[![Python version](https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue.svg)](https://www.python.org/)\n[![Django version](https://img.shields.io/badge/Django-4.2%20|%205.0%20-green.svg)](https://www.djangoproject.com/)\n[![CircleCI](https://circleci.com/gh/openfun/django-peertube-runner-connector/tree/main.svg?style=svg)](https://circleci.com/gh/openfun/django-peertube-runner-connector/tree/main)\n\n## Overview \ndjango-peertube-runner-connector is designed to use [Peertube](https://github.com/Chocobozzz/PeerTube/) transcoding runners outside of Peertube App. It implements a set of endpoints with [Django Rest Framework](https://www.django-rest-framework.org/) and a [SocketIO](https://python-socketio.readthedocs.io/en/latest/) server that allow runners to request jobs, updated job status, download media files and upload the transcoded media files. It provides a function that can be used by your app that will launch the transcoding process.\n\nTo make use of the SocketIO server, this app only work in ASGI.\n\n\n## Architecture\n\n### Runner API\n\nThis part will interact with Peertube runners. It is not designed to be used by a user as it reproduces what the Peertube App is doing in order to manage runners / jobs.\n\n\n#### Runner Behavior\n\nJobs are stored in a Database, and runners hit the `/request` endpoint to get the available jobs to transcode.\n\n\n### The transcode video function\n\nThe function receives a video file and a name, and then creates transcoding jobs for it.\n\nWe use function `probe` of [python-ffmpeg](https://github.com/kkroening/ffmpeg-python) library, to get a thumbnail and all the necessary metadata to create transcoding jobs. Once the jobs are created, the WebSocket server emits an event to inform runners of a new pending jobs.\n\n\n#### Job implementation\n\nCurrently we didn't implement all the transcoding jobs the runner can do. We are planning to implement more jobs in the future. For now, the API only implements the following jobs:\n\n- [x] HLS transcoding\n- [ ] VOD web video transcoding\n- [ ] Live transcoding\n- [ ] VOD audio merge transcoding\n- [ ] Video Studio transcoding\n\nTheses jobs are created and handled through their respective classes in `api.transcoding.utils.job_handlers` directory. Some of them are already almost implemented but are not used, so they are commented.\n\n### SocketIO server\n\nThe SocketIO server is used to communicate with runners. It is only used to inform runners of new jobs, thus, make this part very simple. It implements only one function that emits the event `available-jobs` to runners when a new job is created. Once a runner receives this event, it will hit the `/request` endpoint in the Runner API to get the new job.\n\n## Installation\n\nOnce you have installed the library, you will need to setup your project to use it (see [configuration](#configuration) part). You can find a demo application in the `tests` directory.\n\n### PyPi\n\nTo install the library with pip, enter the following command:\n\n```shell\npip install django-peertube-runner-connector\n```\n\n### Local\nTo install the library locally, enter the following commands at the root of the project:\n\nbuild the library:\n```shell\npython setup.py sdist bdist_wheel\n```\n\nThis should create a ``dist`` directory with the library files.\n\nThen you can go in your application, source your virtual environment and install the library with pip:\n```shell\npip install /path/to/django-peertube-runner-connector/dist/django_peertube_runner_connector-1.tar\n```\n\n## Setup\n\n### Configuration\n\n\n```python\n# settings.py\nINSTALLED_APPS = [\n    # ...\n    \"rest_framework\",\n    \"django_peertube_runner_connector.apps.DjangoPeertubeRunnerConnectorConfig\",\n    \"storages\", # optional django-storages library\n]\n\n# Transcoding resolution settings\nTRANSCODING_ALWAYS_TRANSCODE_ORIGINAL_RESOLUTION = False\nTRANSCODING_RESOLUTIONS_144P = False\nTRANSCODING_RESOLUTIONS_240P = False\nTRANSCODING_RESOLUTIONS_360P = True\nTRANSCODING_RESOLUTIONS_480P = True\nTRANSCODING_RESOLUTIONS_720P = True\nTRANSCODING_RESOLUTIONS_1080P = False\nTRANSCODING_RESOLUTIONS_1440P = False\nTRANSCODING_RESOLUTIONS_2160P = False\n\n# Transcoding fps settings\nTRANSCODING_FPS_MIN = 1\nTRANSCODING_FPS_STANDARD = [24, 25, 30]\nTRANSCODING_FPS_HD_STANDARD = [50, 60]\nTRANSCODING_FPS_AUDIO_MERGE = 25\nTRANSCODING_FPS_AVERAGE = 30\nTRANSCODING_FPS_MAX = 60\nTRANSCODING_FPS_KEEP_ORIGIN_FPS_RESOLUTION_MIN = 720\n\n# Max number of times a job can fail before being marked as failed\nTRANSCODING_RUNNER_MAX_FAILURE = 5\n\n# The callback path to a function that will be called when a video transcoding ended\nTRANSCODING_ENDED_CALLBACK_PATH = \"\"\n\n# The django-peertube-runner-connector app uses the django storage system to store the transcoded videos.\n# It uses the \"videos\" storage where you can configure the storage backend you want to use.\nSTORAGES = {\n    \"default\": {\n        \"BACKEND\": \"django.core.files.storage.FileSystemStorage\",\n    },\n    \"videos\": {  # This is the storage used to store the transcoded videos\n        \"BACKEND\": \"app.storage.MyCustomFileSystemVideoStorage\", # You can use the storage backend you want\n    },\n    \"staticfiles\": {\n        \"BACKEND\": \"django.contrib.staticfiles.storage.StaticFilesStorage\",\n    },\n}\n```\n\n\n#### Storage\n\nDjango-peertube-runner-connector uses the django storage system to store the transcoded videos. It uses the \"videos\" storage where you can configure the storage backend you want to use. To use an S3 like storage, you can use the [django-storages](https://django-storages.readthedocs.io/en/latest/) library. Here is an example of a custom storage backend that uses the S3 storage:\n\n```python\n# app/storage.py\nfrom storages.backends.s3boto3 import S3Boto3Storage\n\n\nclass MyS3VideoStorage(S3Boto3Storage):\n  \"\"\"Custom S3 storage class.\"\"\"\n\n  bucket_name = \"my-bucket\"\n```\n\nthen you can use it in your settings:\n\n```python\n# settings.py\n\n# ... S3 settings\n\nSTORAGES = {\n    \"default\": {\n        \"BACKEND\": \"django.core.files.storage.FileSystemStorage\",\n    },\n    \"videos\": {  \n        \"BACKEND\": \"app.storage.MyS3VideoStorage\", # Your custom storage backend\n    },\n    \"staticfiles\": {\n        \"BACKEND\": \"django.contrib.staticfiles.storage.StaticFilesStorage\",\n    },\n}\n```\n\n#### Server\n\nTo make use of the SocketIO server, you need to have ASGi server like [uvicorn](https://www.uvicorn.org/).\n\nHere is an example on how to configure your asgi server to use the SocketIO server:\n\n\n```python\nfrom configurations.asgi import get_asgi_application\n\ndjango_asgi_app = get_asgi_application()\n\n\n# its important to make all other imports below this comment\nimport socketio \n\nfrom django_peertube_runner_connector.socket import sio \n\n\napplication = socketio.ASGIApp(sio, django_asgi_app)\n\n```\n\nAdd the runners api views to your urls:\n\n```python\n# urls.py\nfrom django_peertube_runner_connector.urls import (\n    urlpatterns as django_peertube_runner_connector_urls,\n)\n\nurlpatterns += django_peertube_runner_connector_urls\n```\n\nIf your application is distributed on multiple servers, you will probably need\nto use a message queue. We manage redis and redis sentinel manager. For this,\nyou have to define this settings\n\n#### Redis sentinel\n\n- `DJANGO_PEERTUBE_RUNNER_CONNECTOR_SENTINELS`: A list of sentinel nodes. \nEach node is represented by a pair (hostname, port). Example: [('localhost', 26379)]\n- `DJANGO_PEERTUBE_RUNNER_CONNECTOR_SENTINELS_MASTER`: The master sentinel name. Example: mymaster\n\n#### Redis\n\n- `DJANGO_PEERTUBE_RUNNER_CONNECTOR_REDIS`: The redis url. Example: `redis://localhost:6379`\n\nVoilà! Your server should be ready!\n\n\n### Demo application\n\nFor testing purpose, you can find a basic django app using the django-peertube-runner-connector library in the `tests` directory. You use run it with the following commands:\n\nCreate your virtual environment:\n```shell\npython -m venv env\nsource env/bin/activate\n```\n\nInstall the dependencies:\n```shell\npip install -e .\"[dev]\"\n```\n\nGo to the tests directory:\n```shell\ncd tests\n```\n\nCreate the database and run the migrations:\n```shell\npython manage.py migrate\n```\n\nCollect the static files:\n```shell\npython manage.py collectstatic\n```\n\nCreate a super user:\n```shell\npython manage.py createsuperuser\n```\n\nLaunch the server with an asgi server like uvicorn:\n```shell\npython -m uvicorn app.asgi:application --reload\n```\n\nOnce the server is running, you can register your server to a peertube runner. \n\n#### Registering a Peertube runner\n\nFirst you will to generate a registration token. To do so, use the following command and keep the registrationToken for late:\n```shell\npython tests/manage.py create_runner_registration_token\n```\n\nFirst you will need a Peertube runner. To launch one, follow the instructions (theses instructions are made by me and for development purpose only)\n\nClone and go to the [peertube repository](https://github.com/Chocobozzz/PeerTube)\n```shell\ngit clone https://github.com/Chocobozzz/PeerTube\n```\n```shell\ncd PeerTube\n```\n\nInstall the dependencies\n```shell\ncd apps/peertube-runner\nnpm install\ncd ../../\n```\nBuild the runner\n```shell\nnpm run build:peertube-runner\n```\n\nLaunch the runner\n```shell\n./apps/peertube-runner/dist/peertube-runner.js server\n```\n\nOpen a new terminal in the same directory and register your runner to your django app\n```shell\n./apps/peertube-runner/dist/peertube-runner.js register --url http://localhost:8000 --registration-token $MY_TOKEN --runner-name transcode-api\n```\n\n#### Created a transcoding jobs and receive the transcoded video\n\nYou can now launch a transcoding job with using the ``http://127.0.0.1:8000/videos/upload`` end point of the django app by sending a multipart/form-data request with your file as the value of the ``videoFile`` key. This video view is given by the test app not by the django-peertube-runner-connector app. This should population a directory named ``video-[uuid]`` in the root of the project with the result of the transcoding job.\n\n### Launch test\n\nTo launch the tests, enter the following commands at the root of the project:\n\nCreate your virtual environment:\n```shell\npython -m venv env\nsource env/bin/activate\n```\n\nInstall the dependencies:\n```shell\npip install -e .\"[dev]\"\n```\n\nLaunch the tests:\n```shell\nmake test\n```\n\n\n## License\n\nThis work is released under the MIT License (see [LICENSE](./LICENSE)).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenfun%2Fdjango-peertube-runner-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenfun%2Fdjango-peertube-runner-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenfun%2Fdjango-peertube-runner-connector/lists"}