{"id":19706928,"url":"https://github.com/moha-cm/youtube-dataharvesting-","last_synced_at":"2026-05-08T02:03:18.822Z","repository":{"id":204218049,"uuid":"711352611","full_name":"Moha-cm/Youtube-Dataharvesting-","owner":"Moha-cm","description":"YouTube Data Harvesting and Warehousing using SQL, MongoDB and Streamlit","archived":false,"fork":false,"pushed_at":"2023-12-02T13:29:24.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-27T18:08:20.883Z","etag":null,"topics":["api-integration","data-collection","data-management","maria","mongodb","mongodb-atlas","pymongo","python-script","sql","streamlit"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Moha-cm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-10-29T01:38:42.000Z","updated_at":"2023-12-02T15:10:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"d27f47a0-fd49-45aa-b7c3-9cb058a9c1b8","html_url":"https://github.com/Moha-cm/Youtube-Dataharvesting-","commit_stats":{"total_commits":25,"total_committers":1,"mean_commits":25.0,"dds":0.0,"last_synced_commit":"ac5cc10e4b44ba4fb1c8773cd464d4245aa271eb"},"previous_names":["moha-cm/youtube-dataharvesting-"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Moha-cm/Youtube-Dataharvesting-","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Moha-cm%2FYoutube-Dataharvesting-","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Moha-cm%2FYoutube-Dataharvesting-/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Moha-cm%2FYoutube-Dataharvesting-/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Moha-cm%2FYoutube-Dataharvesting-/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Moha-cm","download_url":"https://codeload.github.com/Moha-cm/Youtube-Dataharvesting-/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Moha-cm%2FYoutube-Dataharvesting-/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285722810,"owners_count":27220618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-22T02:00:05.934Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-integration","data-collection","data-management","maria","mongodb","mongodb-atlas","pymongo","python-script","sql","streamlit"],"created_at":"2024-11-11T21:37:28.536Z","updated_at":"2025-11-22T02:04:14.687Z","avatar_url":"https://github.com/Moha-cm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Copy code\n# YouTube Data Collection and Analysis\n\n## Overview\nThis project is written in Python and is designed to collect data from YouTube channels using their unique channel IDs. The gathered data is then stored in both SQL and MongoDB databases, enabling further analysis. The results of this analysis can be visualized through a Streamlit application.\n\n## Process\n\n### 1. Google Cloud Setup\nTo begin with, you will need to set up a **Google Cloud account**. This includes enabling the necessary extensions and generating API credentials. You can access the Google Cloud Console through the following link: [Google Cloud Console](https://console.cloud.google.com/apis/dashboard?project=skilled-text-400719).\n\n### 2. YouTube Data Extraction\nUtilize the generated **API key** to extract data from YouTube channels. For detailed information on how to use the functions to retrieve YouTube data, refer to the official documentation: [YouTube Data API Documentation](https://developers.google.com/youtube/v3/docs).\n\n### 3. Data Storage\n\n#### Structured Data (SQL)\nThe structured data is stored in a **SQL database**, providing a tabular and relational format for the collected information. This format is particularly suitable for data that adheres to a well-defined schema.\n\n#### Unstructured Data (MongoDB)\nUnstructured data, which may not conform to a fixed schema, is stored in **MongoDB Atlas**. MongoDB is a NoSQL database that accommodates flexible and dynamic data structures, making it a suitable choice for diverse or evolving data.\n\n### 4. Data Analysis and Visualization\nAccess the stored data to uncover valuable insights from the YouTube channel. Visualize the data for user-friendly presentation, often using plotting techniques.\n\n### 5. Streamlit Application\nTo interact with and visualize the collected data, run the **Streamlit application**.Please make sure you have the necessary requirements installed, including Python, Streamlit, SQLAlchemy, MongoDB, SQL databases, MySQL, googleapiclient, and Plotly. These dependencies are crucial for the proper functioning of the project.\n\nFollow these steps:\n1. Download the source files provided for download.\n2. After downloading, navigate to the project directory in your terminal.\n3. Run the following command to start the application:\n\n\n\n```bash\nstreamlit run  Home.py\n```\n\n\n### Required Python Packages\n```\npip install pandas\n\npip install streamlit\n\npython -m pip install pymongo\n\npip install mysql-connector-python\n\nPip install sqlalchemy\n\npip install PyMySQL\n\npip install isodate\n\npip install google-api-python-client\n\npip install plotly\n\n```\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoha-cm%2Fyoutube-dataharvesting-","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoha-cm%2Fyoutube-dataharvesting-","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoha-cm%2Fyoutube-dataharvesting-/lists"}