{"id":13674184,"url":"https://github.com/Totonyus/ydl_api_ng","last_synced_at":"2025-04-28T14:30:51.868Z","repository":{"id":40412683,"uuid":"482616314","full_name":"Totonyus/ydl_api_ng","owner":"Totonyus","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-01T20:13:54.000Z","size":303,"stargazers_count":108,"open_issues_count":1,"forks_count":13,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-04-01T21:28:00.038Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Totonyus.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-04-17T19:35:39.000Z","updated_at":"2024-04-15T12:10:01.580Z","dependencies_parsed_at":"2023-12-24T10:23:16.499Z","dependency_job_id":"a9ad4197-a14d-4d50-aeea-2520fc4c4b39","html_url":"https://github.com/Totonyus/ydl_api_ng","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Totonyus%2Fydl_api_ng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Totonyus%2Fydl_api_ng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Totonyus%2Fydl_api_ng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Totonyus%2Fydl_api_ng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Totonyus","download_url":"https://codeload.github.com/Totonyus/ydl_api_ng/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251330150,"owners_count":21572236,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T11:00:42.720Z","updated_at":"2025-04-28T14:30:51.523Z","avatar_url":"https://github.com/Totonyus.png","language":"Python","funding_links":[],"categories":["Software","Python","HarmonyOS"],"sub_categories":["Automation","Media Management","Windows Manager"],"readme":"# ydl_api_ng\n\nydl_api_ng is the new version of [ydl_api](https://github.com/Totonyus/ydl_api)\n\n## But why ?\n\nydl_api was built to fulfill my needs, and it works really well for me. However, it has a few drawbacks :\n\n- limited to a few parameters\n- complexity to add new parameters in the api\n- Not suitable for advanced youtube-dl users\n\n## Differences with ydl_api ?\n\n- A huge portion of the codebase has been rewritten\n- A new system of parameter file\n- More powerful : you can now add any youtube-dlp option to enhance your downloads\n- Better maintainability : no need new devs on this application if youtube-dlp adds new options\n- Suitable for advanced youtube-dlp users\n- Not complicated for basic users\n- New features are planned\n- (Optional) Use redis for a better queue management\n\n# Installation\n\n## With docker\n\nContainer on Docker Hub [here](https://hub.docker.com/r/totonyus/ydl_api_ng)\n\nThe parameters files are generated in the params volume on the first launch of the container.\n\n### Volumes mapping\n\nVolumes you could want to map :\n\n- `/app/downloads/` : where files will be downloaded\n- `/app/params/` : where the parameters files will be stored\n- `/app/params/hooks_utils/` : if you need to import your own python scripts for the hooks handlers\n- `/app/logs`\n\n### Ports\n\nThe internal port of the api is `80`\n\n### Docker-compose\n\nJust copy the `docker-compose.yml` file where you want and launch this command :\n\n```shell\ndocker-compose pull # pull the latest image\ndocker-compose up # to start the container\ndocker-compose down # to stop the container\n```\n\n## Force yt-dlp version:\nAdd the version of `yt-dlp` you want in docker environment :\n\n```\nFORCE_YTDLP_VERSION=2022.11.11\n```\n\n### By command line\n\n```shell\ndocker run -p 5011:80 totonyus/ydl_api_ng # don't forget to map the folders\n```\n\nNote : the standard image comes with redis enabled. Please view on the queue management documentation to know how to disable it.\n\nYou can also find the `docker-compose.yml` file.\n\n### UID and GID\n\nThe default user used in the container is `1000:1000`. You can edit those by changing the environment vars `UID`\nand `GID` in the `docker-compose.yml` or in the `docker run` command.\n\nThe user must be root (`UID=0` ans `GID=0`) or match the owner of the download repository.\n\nTo know your `UID` and `GID`, simple run this command with the right user or check your `/etc/passwd` file.\n\n```shell\necho $UID $GID\n```\n\n## Without Docker\n\n### Requirements\n\n* Installation with distribution package manager (`apt`, `yum`, ...) : `python3`, `python3-pip`, `ffmpeg`\n\n```shell\npip3 install -r pip_requirements\n``` \n\n### Download this repo\n\nDownload the latest release :\n\n```shell\nwget https://github.com/Totonyus/ydl_api_ng/archive/master.zip\n```\n\nThen unzip the file and rename the unzipped folder : (note you can use `unzip -d %your_path%` to specify where you want\nto unzip the file )\n\n```shell\nunzip master.zip; mv ydl_api_ng-master ydl_api_ng\n```\n\n### Modify listening port\n\nThe api listen by default on the port `80`. It's perfect for docker use but not really a good idea for a bare metal\nuser. Set the port you want in the `params.ini` file.\n\n### Install as daemon\n\nJust fill the `ydl_api_ng.service` file and move it in `/usr/lib/systemd/system/` for `systemd` linux.\n\n```shell\nmv ydl_api.service /usr/lib/systemd/system/\n```\n\nYou can change the name of the service by changing the name of the file.\n\nthen (you must run this command every time you change the service file)\n\n```shell\nsystemctl daemon-reload\n```\n\nCommands :\n\n```shell\nsystemctl start ydl_api_ng\nsystemctl stop ydl_api_ng\nsystemctl enable ydl_api_ng # start at boot\nsystemctl disable ydl_api_ng # don't start at boot\n```\n\n### Install the userscript\n\nInstall [Greasemonkey (firefox)](https://addons.mozilla.org/fr/firefox/addon/greasemonkey/)\nor [Tampermonkey (chrome)](https://chrome.google.com/webstore/detail/tampermonkey/dhdgffkkebhmkfjojejmpbldmpobfkfo?hl=fr)\nand add a new userscript. Then copy the content of `userscript.js` in the interface and save.\n\nThen change the default host set in the script. You can also customize all the presets to match the configuration file\nof your server.\n\nYou now should have access to download options on every site. You can also modify the match options to limit the usages.\n\n![Userscript in action](userscript.jpg)\n\n## Files you'll probably want to consult or customize\n\nThese files are generated in `params/`\n\n* `params.ini` all the default parameters of the application, everything is set up to offer you a working application\n  out of the box\n* `params_metadata.ini` describe how each parameter in `params.ini` should be interpreted\n* `userscript.js` a javascript file you can import\n  in [Greasemonkey (firefox)](https://addons.mozilla.org/fr/firefox/addon/greasemonkey/)\n  or [Tampermonkey (chrome)](https://chrome.google.com/webstore/detail/tampermonkey/dhdgffkkebhmkfjojejmpbldmpobfkfo?hl=fr)\n  to access the api from yout browser\n* `progress_hooks.py` youtube-dl progress hooks handler method\n* `postprocessor_hooks.py` youtube-dl postprocessor hooks handler method\n* `ydl_api_hooks.py` api hooks handler methods, only called at few moments\n* `hooks_requirements` the extra requirements you need to fully use the hooks\n\nThere files are in the base folder :\n\n* `docker-compose.yml` to configure easily the container\n* `ydl_api_ng.service` a systemd service file to launch this application as a daemon\n\n## Updating youtube-dlp\n\n### Docker\n\nSimply reload the container, an update is performed at launch\n\n### Without docker\n\n```shell\npip3 install yt_dlp --upgrade\n```\n\n## Configuration details\n\n### parameters_metadata.ini\n\nEach parameter in the `params.ini` file is a string. The `parameters_metadata.ini` file is used to describe how each\nparameter must be cast.\n\nAs I don't know any option of `youtuble-dlp`, I tried my best to arrange the fields correctly but some are probably\nwrong.\n\nPlease don't hesitate to make a merge request or to open a ticket on this repo to ask the correction.\n\nYou can also add your own parameters fields if you need it.\n\n### params.sample.ini\n\nThis file is a complete working configuration file who is fully documented to help you to understand how it works.\n\n### The `_cli` parameter\nThe `_cli` parameter allows the usage of a command line configuration to make `ydl_api_ng` even more simple than before.\n\n```\n[preset:AUDIO_CLI]\n_template = AUDIO\n_cli = -f bestaudio --embed-metadata --embed-thumbnail --extract-audio --audio-format mp3 --split-chapters\n```\n\nIs the same thing than:\n```\n[preset:AUDIO]\n_template = AUDIO\nformat = bestaudio\nwritethumbnail = true\nfinal_ext = mp3\npostprocessors: [{\"key\": \"FFmpegExtractAudio\", \"preferredcodec\": \"mp3\", \"preferredquality\": \"5\", \"nopostoverwrites\": false}, {\"key\": \"FFmpegMetadata\", \"add_chapters\": true, \"add_metadata\": true, \"add_infojson\": \"if_exists\"}, {\"key\": \"EmbedThumbnail\", \"already_have_thumbnail\": false}, {\"key\": \"FFmpegSplitChapters\", \"force_keyframes\": false}]}\n```\n\nTo use a yt-dlp configuration file:\n```\n[preset:AUDIO]\n_cli = --config-location params/conf_audio.conf\n```\n\n## User management\n\nIf the user management is activated (disabled by default), each request must have the `\u0026token` query parameter. You can\noverride some parameters for a specific user.\n\n## Site management\n\nYou can define parameters for given websites. It's useful to avoid long-running playlist download simulation.\n\n```ini\n# Every host matching this site\n_hosts = www.youtube.com,youtu.be\n# How to define the url is a video \n_video_indicators = /watch?\n# How to define the url is a playlist\n_playlist_indicators = ?list=,\u0026list=,/user/,/playlists\n```\n\nYou can also add parameters tied to the site like login information\n\n## Queue management\n\nA new system of queue management has been build to have a better view of current, past and future downloads.\n\nYou can choose the number of workers in the docker-compose file : `NB_WORKERS=5` (environment parameter). The number of\nworkers determines the number of parallel downloads.\n\n### Without docker\n\nYou can perfectly run ydl_api_ng with redis without docker. However, to keep it simple for basic users, the option is\ndisabled by default in the `params.ini` file :\n\n    _enable_redis = true # (false is not in parameter file)\n    _redis_host = localhost\n    _redis_port = 6379\n\nThe application will run downloads without redis and the old queue management system will be used for the\napi `/active_downloads` and `/terminate`\n\n### With docker\n\nIf you don't want to use redis with docker, you can remove the corresponding block in the docker-compose\nfile : `ydl_api_ng_redis`.\n\nYou must disable redis in the `params.ini` file : `_enable_redis = false`\n\nYou can also pass the `DISABLE_REDIS` environment parameter in the docker-compose file to avoid workers launching.\n\n## Programmation\nA new feature allow you to schedule ydl_dlp executions. This feature needs `redis`.\n\nThree usecases:\n- Continuously download a livestream when it's available\n- Archive a channel at given hours\n- Schedule a download at a precise date for a precise duration\n\n### Representation\nHere what a programmation object looks like in database :\n```json\n{\n\"id\": \"string (generated)\",\n\"url\": \"string\",\n\"user_token\": \"string (null)\",\n\"enabled\": \"bool (true)\",\n\"planning\": {\n    \"recording_start_date\": \"string date : YYYY-MM-DD hh:mm (null)\",\n    \"recording_duration\": \"int \u003e 0 (null)\",\n    \"recording_stops_at_end\": \"bool (false)\",\n    \"recording_restarts_during_duration\" : \"bool (true)\",\n\n    \"recurrence_cron\": \"string (null)\",\n    \"recurrence_start_date\": \"string date : YYYY-MM-DD hh:mm (null)\",\n    \"recurrence_end_date\": \"string date : YYYY-MM-DD hh:mm (null)\"\n    },\n\"presets\": [\"string\"], \n\"extra_parameters\" : {}\n}\n```\n\nFields:\n- `id` : generated by the api\n- `url` : url to download, it will not be checked\n- `user_token` : unused if the `_allow_programmation` is not explicitly set at true in the user config\n- `enabled` : if false, the programmation will be ignored\n- `planning.recording_start_date`\n- `planning.recording_duration` : how many minutes recording is supposed to long\n- `planning.recording_stops_at_end` : if true, the download will be force stopped when `recording_duration` is reached\n- `planning.recording_restarts_during_duration` : if False, the download will not be restarted if stopped before `recording_duration`\n- `planning.recurrence_cron` : same cron as linux\n- `planning.recurrence_start_date` : useful only if `recurrence_cron` is used\n- `planning.recurrence_end_date` : useful only if `recurrence_cron` is used\n- `presets` : list of presets names\n- `extra_parameters` : an arbitrary object of parameters you can use to store informations or directives to use in hooks\n\nNotes:\n- `recording_start_date` and `recurrence_cron` cannot be used at the same type\n- if `recording_duration` is set, the download will be relaunched automatically if stopped during the interval \n\n### Reliability\nThe launch hours are not perfectly exact and have an error range of `-1` minute\n\n### Examples of minimal configurations\n#### Continuous download\n\n```json\n{\n\"url\": \"string\"\n}\n```\n\n#### Archiving execution\n```json\n{\n\"url\": \"string\",\n\"planning\": {\n    \"recurrence_cron\": \"00 01 * * *\"\n    }\n}\n```\n\n#### Special program\nEvery day in december from 13:00 to 14:00\n```json\n{\n\"url\": \"string\",\n\"planning\": {\n    \"recording_duration\": 60,\n    \"recording_stops_at_end\": true,\n\n    \"recurrence_cron\": \"00 13 * * *\",\n    \"recurrence_start_date\": \"2022-12-01 00:00\",\n    \"recurrence_end_date\": \"2022-12-31 23:00\"\n    }\n}\n```\n\n#### One time special program\n\nWill last at least 4 hours\n```json\n{\n  \"url\": \"string\",\n  \"planning\": {\n    \"recording_start_date\" : \"2022-12-31 22:00\",\n    \"recording_duration\": 240\n  }\n}\n```\n\n#### Programmation with extra parameters\n\n```json\n{\n  \"url\": \"string\",\n  \"planning\": {\n    \"recurrence_cron\": \"00 * * * *\"\n  },\n  \"extra_parameters\": {\n    \"notification_level\": \"critical\",\n    \"video_description\": \"Josephine Ange Gardien - 25th anniversary epic trailer\"\n  }\n}\n```\n\n# API\n\n## Application information\n\nReturn the application complete parameters\n\n```shell\nGET http://localhost:5011/info\n```\n\n### Responses status\n\n* `401` : User is not permitted\n\n## Downloads\n\nOnly the url is required to use the api.\n\n```shell\n# Simplest case, uses the DEFAULT preset\nGET http://localhost:5011/download?url=https://www.youtube.com/watch?v=9Lgc3TxqgHA\n\n# You can download multiple presets at once, if no preset is valid, will download with DEFAULT preset, if at least one preset is valid, will download only valid presets\nGET http://localhost:5011/download?url=https://www.youtube.com/watch?v=Kf1XttuuIiQ\u0026presets=audio,hd\n\n# If the user management is enabled\nGET http://localhost:5011/download?url=https://www.youtube.com/watch?v=wV4wepiucf4\u0026token=dad_super_password\n```\n\n## Post request\n\nYou can download the video you want by providing the parameters directly in a post request. The order of the expandable attributes is important : each attribute will be expanded in this order. \n\n```shell\nPOST http://localhost:5011/download?url=https://www.youtube.com/watch?v=wV4wepiucf4 \u0026\ntoken=dad_super_password\nContent-Type: application/json\n\n{\n  \"cookies\" : \"URL encoded (RFC3986 format) netscape cookies format\",\n  \"presets\": [\n  {\n    \"_ignore_site_config\": false,    # (optional, default : false) if true, will not load parameters from site detection\n    \"_ignore_default_preset\": false, # (optional, default : false) if true, will not expand default preset\n    # You can expand parameters\n    \"_preset\" : \"AUDIO\",\n    \"_location\" : \"AUDIO\",\n    # just put below your standard youtube-dlp options\n    \"format\" : \"best[height=360]/bestvideo[height=360]+bestaudio/best\"\n  }\n  ]\n}\n\n```\n\nIt is possible to add a timer to stop the download (`recording_stops_at_end` will be automatically set on `True`) :\n\n```shell\nPOST http://localhost:5011/download?url=https://www.youtube.com/watch?v=wV4wepiucf4\u0026token=dad_super_password\nContent-Type: application/json\n\n{\n  \"programmation\": {\n    \"planning\": {\n      \"recording_duration\": 10\n    }\n  },\n  \"presets\": [\n    {\n      \"_preset\": \"HD\"\n    }\n  ]\n}\n```\n\nReminder : if you want to expand a preset : all presets automatically expand the `DEFAULT` preset. Basically, expand a\npreset with `_preset` means `_ignore_default_preset`can't be true.\n\nYou can use the `_cli` attribute here :\n```shell\nPOST http://localhost:5011/download?url=https://www.youtube.com/watch?v=wV4wepiucf4\nContent-Type: application/json\n\n{\n  \"presets\": [\n    {\n      \"_template\": \"AUDIO\",\n      \"_cli\" : \"-f bestaudio --embed-metadata --embed-thumbnail --extract-audio --audio-format mp3 --split-chapters\",\n    }\n  ]\n}\n```\n\n### Important notice\n\nAs the post request can be dangerous by allowing to write anywhere on your system\n(if not running in docker) a parameter `_allow_dangerous_post_requests` (`false` by default) has been added.\n\nFor each preset if `_allow_dangerous_post_requests` is false :\n\n- `paths` will be deleted and replaced by the `default` location parameter\n- `outtmpl` will be deleted and replaced by the `default` template parameter\n- You still can select a `paths` or a `outtmpl` by using expansion system\n- You can only use `paths` and `outtmpl` present in `params.ini`\n\n## Video information\n\nReturns the standard youtube-dlp extract info object.\n\n```shell\nGET http://localhost:5011/extract_info?url=https://www.youtube.com/watch?v=9Lgc3TxqgHA\n```\n\n### Responses status\n\n* `401` : User is not permitted\n\n## Process management\n\nThe process management system only works on livestreams\n\n### Global\n\n```shell\n# Get all active downloads (with PID)\nGET http://localhost:5011/active_downloads\n\n# Stop all active downloads\nGET http://localhost:5011/active_downloads/terminate\n\n# Stop the active download with it PID. It uses se system PID, this feature is safe, a non-child process cannot be killed\n# If using redis, also permit to cancel pending job\n# If job is finished or canceled, will delete from queue\nGET http://localhost:5011/active_downloads/terminate/{pid}\n```\n\n### Redis only\n\n```shell\n# Get all registries content\nGET http://localhost:5011/queue\n\n# Get registry content (all, workers, pending_job, started_job, finished_job, failed_job, deferred_job, scheduled_job, canceled_job)\nGET http://localhost:5011/queue/finished_job\n\n# Delete all jobs but pending and started jobs\nDELETE http://localhost:5011/queue\n```\n\n## Programmation\n```shell\n# Get programmations in database\nGET {{host}}/programmation?token=user_token\n\n# Add a new programmation\nPOST {{host}}/programmation?url=an_added_url\u0026token=user_token\n\n{\n  \"planning\": {\n    \"recording_duration\": 60,\n    \"recurrence_cron\": \"00 12 * * *\"\n  }\n} \n\n# Update a programmation\nPUT {{host}}/programmation/\u003cid\u003e\n\n{\n  \"planning\": {\n    \"recurrence_end_date\" : \"2022-12-31 00:00\"\n  }\n}\n\n# Delete a programmation by ID\nDELETE {{host}}/programmation/\u003cID\u003e\n\n# Delete all programmations for the URL\nDELETE {{host}}/programmation?url=url\n```\n\n\n## Responses status\n\n* `401` : User is not permitted\n\n# Redis queues\nIf redis is enabled, there is two redis queues that : `ydl_api_ng` and `ydl_api_ng_slow`.\n\nThe `ydl_api_ng_slow` is processed by an unique worker. It's designed to queue downloads to :\n- avoid throttle from websites \n- don't overcharge your connection / disk\n\nYou can add more redis queues or customize existing ones by editing the `params/workers.ini` file. All redis queues workers names must starts with `worker_`.\n\n```\n[program:worker_ydl_api_ng] -\u003e Real redis queue name : ydl_api_ng\n```\n\nThe first queue in the file will be the default.\n\n## Usage in presets\nExample of preset : \n```\n[preset:ARCHIVE]\n_redis_queue = ydl_api_ng_slow\n; Set redis TTL specifically for this preset\n_redis_ttl = 31400\n```\n\n## Usage in api\nExample :\n```shell\n# Get active downloads in all redis queues\nGET http://localhost:5011/active_downloads\n\n# Get active downloads is a given queue\nGET http://localhost:5011/active_downloads?redis_queue=ydl_api_ng_slow\n\n#  This queue doesn't exists, send a 404 error\nGET http://localhost:5011/active_downloads?redis_queue=ydl_api_ng_live\n```\n\n# iOS shortcut\n\nThere is now an iOS shortcut you can find [here](https://www.icloud.com/shortcuts/deb4fea950ee436daf9a1f668a55add4).\n\nIf the shortcut is launched outside the share interface, it uses the content of the clipboard.\n\n## Shortcut configuration\n\n```\nhost : download url (with endpoint)\ntoken : the user token\npreset_selection : if true, asks the preset to use. If false, use the default preset\npresets : a map with all the presets \ndefault_preset : the default preset if preset_selection is false\n```\n\n# Contributing\n\n- Found a bug ? Need an improvement ? Need help ? Open a ticket !\n- Found a typo in documentation ? That's normal ! I'm French. Don't hesitate to contact me if you don't understand a\n  sentence or if there are mistakes.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTotonyus%2Fydl_api_ng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTotonyus%2Fydl_api_ng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTotonyus%2Fydl_api_ng/lists"}