{"id":29360276,"url":"https://github.com/paulo-granthon/yt-comment-dumper","last_synced_at":"2026-05-08T15:18:54.647Z","repository":{"id":297118219,"uuid":"995719731","full_name":"paulo-granthon/yt-comment-dumper","owner":"paulo-granthon","description":"ETL + flask server for dumping comments from youtube videos ","archived":false,"fork":false,"pushed_at":"2025-07-07T06:02:26.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-07T06:40:01.106Z","etag":null,"topics":["etl","flask","mongo","youtool","youtube-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulo-granthon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-03T23:07:23.000Z","updated_at":"2025-07-07T06:02:30.000Z","dependencies_parsed_at":"2025-06-04T07:10:13.709Z","dependency_job_id":"8b4e323a-e7e4-4dc3-be50-11655e825ea9","html_url":"https://github.com/paulo-granthon/yt-comment-dumper","commit_stats":null,"previous_names":["paulo-granthon/yt-comment-dumper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paulo-granthon/yt-comment-dumper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulo-granthon%2Fyt-comment-dumper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulo-granthon%2Fyt-comment-dumper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulo-granthon%2Fyt-comment-dumper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulo-granthon%2Fyt-comment-dumper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulo-granthon","download_url":"https://codeload.github.com/paulo-granthon/yt-comment-dumper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulo-granthon%2Fyt-comment-dumper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264411189,"owners_count":23603807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","flask","mongo","youtool","youtube-api"],"created_at":"2025-07-09T07:12:51.593Z","updated_at":"2026-05-08T15:18:49.587Z","avatar_url":"https://github.com/paulo-granthon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yt-comment-dumper\n\n\u003ch3 align=\"center\"\u003e\u003ci\u003e ETL + flask server for dumping comments from youtube videos \u003c/i\u003e\u003c/h3\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \nhttps://github.com/user-attachments/assets/bc36bdcd-c918-4315-935c-e96dcb1c1dd9\n\n```rb\nmake VIDEO=https://www.youtube.com/watch?v=IH3W4WrM8qc\n```\n\n\u003c/div\u003e\n\u003cbr/\u003e\n\n\u003ch2 align=\"center\"\u003e Projects: \u003c/h2\u003e\n\n## ETL\n\n- ***Extract***s the comments from the youtube video, along with data such as the number of likes, the `parent_id` (in case of a response comment) and the user's profile picture url;\n- ***Transform***s the data from the provider structure into the document structure used in the database;\n- ***Load***s the formatted data into a `MongoDB` collection.\n\n## Server\n\n- Provides a tree view page with the comments present in the collection.\n\n\u003cbr/\u003e\n\n\u003ch2 align=\"center\"\u003e Environment: \u003c/h2\u003e\n\n#### `yt-comment-dumper` will only work if there's a properly configured `.env` file in the root of the repository directory.\n\n\u003cbr/\u003e\n\n- Copy the `.env.example` file into a `.env` file and set your youtube api key:\n\n\u003e .env:\n```rb\n# shellcheck disable=2034,2148\n# YouTube API keys\nYOUTUBE_API_KEY=your_api_key_here\n[...]\n```\n\n- If you are not using the pre-configured MongoDB container of this repository, you'll also need to change the environment variables prefixed with `DB_` in your `.env` file:\n\n\u003e .env:\n```rb\n[...]\n# MongoDB configuration\nDB_HOST=localhost\nDB_PORT=27017\nDB_NAME=yt_comment_dumper\nDB_COLLECTION=comments\nDB_USERNAME=mongo\nDB_PASSWORD=secret\n```\n\n\u003cbr/\u003e\n\n\u003ch2 align=\"center\"\u003e Usage: \u003c/h2\u003e\n\n## Makefile:\n\nIt's possible to run everything using the Makefile commands.\n\n```yaml\nmake VIDEO=\u003cvideo_url_or_id\u003e        # Runs everything (db, etl, server)\n```\n\nDefault commands:\n\n```yaml\nmake db-up   # starts the mongodb container\n\nmake prep    # installs poetry dependencies\n\nmake etl     # runs the ETL\n\nmake serve   # runs the server\n```\n\nEnd of session:\n\n```yaml\nmake db-down # stops the mongodb container\n```\n\n## poetry:\n\nInstall project dependencies:\n\n```yaml\npoetry lock\npoetry install\n```\n\nRun projects:\n\n```yaml\npoetry run python src/etl/main.py VIDEO=\u003cvideo_url_or_id\u003e   # runs the ETL\npoetry run python src/server/app.py                         # runs server\n```\n\n## docker-compose:\n\nMongoDB container management (or use yours)\n\n```yaml\ndocker-compose up   # start mongodb container\ndocker-compose down # stop mongodb container\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulo-granthon%2Fyt-comment-dumper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulo-granthon%2Fyt-comment-dumper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulo-granthon%2Fyt-comment-dumper/lists"}