{"id":22874446,"url":"https://github.com/brettdavies/yt_dlp_async","last_synced_at":"2026-04-30T01:37:41.389Z","repository":{"id":257186900,"uuid":"832812896","full_name":"brettdavies/yt_dlp_async","owner":"brettdavies","description":"Asynchronous tool for fetching and processing YouTube audio data using yt-dlp, designed to aid scalable and efficient data collection and analysis for academic research projects.","archived":false,"fork":false,"pushed_at":"2024-11-07T18:46:40.000Z","size":1278,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-06T18:52:30.080Z","etag":null,"topics":["academic-project","asynchronous-io","audio-downloader","audio-metadata","data-collection","dataset-generation","parallel-processing-workerpool","poetry","poetryapp","python-poetry","research-project","youtube","yt-dlp","yt-dlp-wrapper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brettdavies.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T19:17:58.000Z","updated_at":"2024-11-07T18:46:44.000Z","dependencies_parsed_at":"2024-09-15T07:48:43.372Z","dependency_job_id":"2353e279-8e48-4f16-a72b-b2cdb6e05cc1","html_url":"https://github.com/brettdavies/yt_dlp_async","commit_stats":null,"previous_names":["brettdavies/yt_dlp_async"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettdavies%2Fyt_dlp_async","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettdavies%2Fyt_dlp_async/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettdavies%2Fyt_dlp_async/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brettdavies%2Fyt_dlp_async/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brettdavies","download_url":"https://codeload.github.com/brettdavies/yt_dlp_async/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246472582,"owners_count":20783225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["academic-project","asynchronous-io","audio-downloader","audio-metadata","data-collection","dataset-generation","parallel-processing-workerpool","poetry","poetryapp","python-poetry","research-project","youtube","yt-dlp","yt-dlp-wrapper"],"created_at":"2024-12-13T14:36:52.048Z","updated_at":"2026-04-30T01:37:41.384Z","avatar_url":"https://github.com/brettdavies.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yt_dlp_async\n\n\u003e **Project Overview:** See [PROJECT.md](PROJECT.md) for a high-level overview, achievements, and technical highlights.\n\n`yt_dlp_async` is an asynchronous Python application designed to fetch and process YouTube video and playlist IDs using `yt-dlp` and store them in a PostgreSQL database. The application supports parallel processing of user IDs, playlist IDs, and video IDs through a set of asynchronous worker tasks.\n\n## Features\n\n- **Asynchronous Processing**: Utilizes the power of `asyncio` to handle multiple tasks concurrently, improving efficiency and performance.\n- **Parallel Processing with Workers**: Introduces the concept of workers to enable parallel processing of user IDs, playlist IDs, and video IDs. This allows for even faster data retrieval and processing.\n- **Environment Configuration**: Loads configuration settings from a `.env` file, making it easy to customize the application for different environments.\n- **Database Operations**: Manages database connections and performs batch inserts using `psycopg2`, ensuring efficient and reliable data storage.\n- **Command-Line Interface (CLI)**: Provides a user-friendly CLI interface with commands to execute various functionalities.\n- **Detailed Module Documentation**: Comprehensive documentation is available for each module, facilitating easier understanding and maintenance.\n\nWith the addition of workers, `yt_dlp_async` can now handle multiple tasks simultaneously, significantly improving the overall performance and reducing the time required for fetching and processing YouTube IDs.\n\nEach worker operates independently, allowing for efficient utilization of system resources and maximizing throughput. This feature is particularly useful when dealing with large datasets or when time-sensitive operations are required.\n\nTo configure the number of workers, simply adjust the corresponding setting in the `.env` file or use command-line arguments. By fine-tuning the number of workers based on your system's capabilities, you can achieve optimal performance for your specific use case.\n\n## Prerequisites\n\nBefore using `yt_dlp_async`, ensure that you have the following prerequisites installed:\n\n- **Python 3.12 or higher**\n- **Poetry**\n- **yt-dlp**\n- **Access to a PostgreSQL database**\n\n## Installation\n\nTo install `yt_dlp_async`, follow these steps:\n\n1. **Clone the Repository**:\n\n   ```bash\n   git clone https://github.com/yourusername/yt_dlp_async.git\n   cd yt_dlp_async\n   ```\n\n2. **Install Dependencies**:\n\n   Use Poetry to handle dependencies:\n\n   ```bash\n   poetry install\n   ```\n\n3. **Create and Configure the `.env` File**:\n\n   Create a `.env` file in the root directory with the following content:\n\n   ```ini\n   DATABASE_URL=postgresql://admin:admin123@localhost:5432/video_data\n   ```\n\n   Adjust the `DATABASE_URL` to match your PostgreSQL setup.\n\n4. **Prepare the Database**\n\n   Execute `db_schema.sql` to create the necessary tables and functions.\n\n## Documentation\n\nDetailed documentation for each module is available in the `docs` directory:\n\n- [video_id.py](docs/video_id.md): Fetching video IDs from YouTube.\n\n- [video_metadata.py](docs/video_metadata.md): Retrieves and processes metadata for YouTube videos.\n    - **Requires a Google API key with access to the YouTube Data API v3.** You can obtain an API key and ensure appropriate access by following [Google's official instructions](https://developers.google.com/youtube/v3/getting-started#before-you-start).\n\n- [video_file.py](docs/video_file.md): Handles downloading of video files from YouTube.\n\n- [video_download.py](docs/video_download.md): Manages the processing and organization of downloaded videos.\n\n- [utils.py](docs/utils.md): Provides utility functions for URL preparation, file reading, and data extraction.\n\n- [e_events.py](docs/e_events.md): Fetches and processes event data from the ESPN API.\n\n- [metadata.py](docs/metadata.md): Contains mappings for standardizing team names and abbreviations.\n\n- [db_schema.sql](docs/db_schema.md): Defines the database schema for storing YouTube and ESPN data.\n\n- [database.py](docs/database.md): Manages database operations, including establishing SSH tunnels, handling database connections, and executing queries using a connection pool pattern for efficient connection management.\n\n## Usage\n\n### CLI Commands\n\n`yt_dlp_async` provides several CLI commands for fetching and processing YouTube IDs. The commands are accessible through the `fire` library.\n\n- **Fetch Video IDs**:\n\n  ```bash\n  poetry run get-video-id fetch [OPTIONS]\n  ```\n\n- **Fetch Video Metadata**:\n\n  ```bash\n  poetry run get-video-metadata fetch [OPTIONS]\n  ```\n\n- **Download Video Files**:\n\n  ```bash\n  poetry run get-video-file fetch [OPTIONS]\n  ```\n\n### Configuration\n\nThe application uses the `.env` file for configuration. Make sure to specify your `DATABASE_URL` to connect to your PostgreSQL database.\n\n### Example Usage\n\n#### Fetching Video IDs\n\nTo start fetching video IDs, you can use the following command:\n\n```bash\npoetry run get-video-id fetch \\\n  --video_ids \"dQw4w9WgXcQ,9bZkp7q19f0\" \\\n  --video_id_files \"video_ids.txt\" \\\n  --playlist_ids \"PLMC9KNkIncKtPzgY-5rmhvj7fax8fdxoj\" \\\n  --playlist_id_files \"playlist_ids.csv\" \\\n  --user_ids \"UC_x5XG1OV2P6uZZ5FSM9Ttw\" \\\n  --user_id_files \"user_ids.txt\" \\\n  --num_workers 4\n```\n\n- **Options**:\n  - `--video_ids`: Comma-separated list of video IDs to fetch.\n  - `--video_id_files`: One or more text files containing video IDs (one per line).\n  - `--playlist_ids`: Comma-separated list of playlist IDs to fetch.\n  - `--playlist_id_files`: One or more CSV files containing playlist IDs.\n  - `--user_ids`: Comma-separated list of user or channel IDs to fetch.\n  - `--user_id_files`: One or more text files containing user IDs (one per line).\n  - `--num_workers`: Number of workers to use for parallel processing (default is 1).\n\nYou can mix and match these options to fetch video IDs from different sources simultaneously. For example, you can provide a combination of video IDs, playlist IDs, and user IDs in a single command.\n\nFor detailed usage and examples, refer to the [video_id.py documentation](docs/video_id.md).\n\n#### Fetching Video Metadata\n\nTo retrieve metadata for YouTube videos, you can use the following command:\n\n```bash\npoetry run get-video-metadata fetch \\\n  --num_workers 4\n```\n\nThis command will process video IDs from the database that do not yet have metadata.\n\nFor more information, see the [video_metadata.py documentation](docs/video_metadata.md).\n\n#### Downloading Video Files\n\nTo download YouTube videos, you can use the following command:\n\n```bash\npoetry run get-video-file fetch \\\n  --existing_videos_dir \"/path/to/existing_videos\" \\\n  --num_workers 4\n```\n\n- **Options**:\n  - `--existing_videos_dir`: Path to the directory containing existing video files.\n  - `--num_workers`: Number of workers to use for parallel processing (default is 10).\n\nThis command will download video files for video IDs that are in the database but do not have associated files yet.\n\nFor additional details, refer to the [video_file.py documentation](docs/video_file.md).\n\n## License\n\nThis project is licensed under the BSD 3-Clause License. See the [LICENSE](LICENSE) file for details.\n\n## Acknowledgements\n\n- [yt-dlp](https://github.com/yt-dlp/yt-dlp) for the video extraction tool.\n- [Poetry](https://python-poetry.org/) for dependency management and packaging.\n- [Loguru](https://github.com/Delgan/loguru) for advanced logging.\n\n---\n\nBy utilizing the comprehensive documentation and modular design, `yt_dlp_async` provides a robust and efficient solution for fetching, processing, and storing YouTube data. For more detailed information on each module and its functionalities, please refer to the documentation linked above.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrettdavies%2Fyt_dlp_async","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrettdavies%2Fyt_dlp_async","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrettdavies%2Fyt_dlp_async/lists"}