{"id":15792917,"url":"https://github.com/shivendrra/web-graze","last_synced_at":"2025-05-13T01:04:12.487Z","repository":{"id":220944563,"uuid":"752999352","full_name":"shivendrra/web-graze","owner":"shivendrra","description":"scrape raw data from various sources of the internet, like wikipedia, internet archieve, britannica, youtube, unsplash, etc","archived":false,"fork":false,"pushed_at":"2024-09-03T18:28:17.000Z","size":6268,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-13T01:02:08.309Z","etag":null,"topics":["data-collection","data-collection-system","data-for-llm","data-for-ml","webscra","webscraper","webscrapper-python"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/webgraze/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shivendrra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-05T09:18:02.000Z","updated_at":"2024-09-03T20:24:18.000Z","dependencies_parsed_at":"2024-02-22T08:29:05.458Z","dependency_job_id":"f5499d7b-cc65-48d9-a5ce-bdf7bc2b55f3","html_url":"https://github.com/shivendrra/web-graze","commit_stats":{"total_commits":52,"total_committers":2,"mean_commits":26.0,"dds":"0.11538461538461542","last_synced_commit":"c05bd2089d23d88fa3da8e9ed742f1a5940bd200"},"previous_names":["shivendrra/datacollection","shivendrra/tan-tivan","shivendrra/data-collection","shivendrra/web-graze"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2Fweb-graze","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2Fweb-graze/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2Fweb-graze/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shivendrra%2Fweb-graze/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shivendrra","download_url":"https://codeload.github.com/shivendrra/web-graze/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253850874,"owners_count":21973672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-collection","data-collection-system","data-for-llm","data-for-ml","webscra","webscraper","webscrapper-python"],"created_at":"2024-10-04T23:06:59.093Z","updated_at":"2025-05-13T01:04:12.429Z","avatar_url":"https://github.com/shivendrra.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# web-graze\n\n## Introduction\nThis repository contains a collection of scripts to scrape content from various sources like YouTube, Wikipedia, and Britannica. It includes functionality to download video captions from YouTube, scrape Wikipedia articles, and fetch content from Britannica.\n\n## Table of Contents\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Queries](#1-queries)\n  - [YouTube Scraper](#2-youtube-scraper)\n  - [Wikipedia Scraper](#3-wikipedia-scraper)\n  - [Unsplash Scraper](#4-unsplash-scraper)\n  - [Britannica Scraper](#5-britannica-scraper)\n  - [Freesound Scraper](#6-freesound-scraper)\n  - [Pexels Scraper](#7-pexels-scraper)\n- [Configuration](#configuration)\n- [Logging](#logging)\n\n## Installation\n\n1. **Clone the repository:**\n   ```sh\n   git clone https://github.com/shivendrra/web-graze.git\n   cd web-scraper-suite\n   ```\n\n2. **Create and activate a virtual environment:**\n   ```sh\n   python -m venv venv\n   source venv/bin/activate   # On Windows: venv\\Scripts\\activate\n   ```\n\n3. **Install the required packages:**\n   ```sh\n   pip install -r requirements.txt\n   ```\n\n## Usage\n\nFor sample examples, use the [run.py](run.py) that contains example for each type of scraper.\n\n### 1. Queries\n\nThis library contains some topics, keywords, search queries \u0026 channel ids which you can just load \u0026 use it with the respective scrapers.\n\n#### Channel Ids\n\n```python\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"channel\")\n```\n\n#### Search Queries\n\n```python\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"search\")\n```\n\n#### Image Topics\n\n```python\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"channel\")\n```\n\n### 2. YouTube Scraper\n\nThe YouTube scraper fetches video captions from a list of channels.\n\n#### Configuration\n- Add your YouTube API key to a `.env` file:\n  ```env\n  yt_key=YOUR_API_KEY\n  ```\n\n- Create a `channelIds.json` file with the list of channel IDs:\n  ```json\n  [\n    \"UC_x5XG1OV2P6uZZ5FSM9Ttw\",\n    \"UCJ0-OtVpF0wOKEqT2Z1HEtA\"\n  ]\n  ```\n\n#### Running the Scraper\n\n```python\nimport os\nfrom dotenv import load_dotenv\nload_dotenv()\ncurrent_directory = os.path.dirname(os.path.abspath(__file__))\nos.chdir(current_directory)\n\napi_key = os.getenv('yt_key')\n\nfrom webgraze import Youtube\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"channel\")\n\nyoutube = Youtube(api_key=api_key, filepath='../transcripts', max_results=50)\nyoutube(channel_ids=queries(), videoUrls=True)\n```\n\n### 3. Wikipedia Scraper\n\nThe Wikipedia scraper generates target URLs from provided queries, fetches the complete web page, and writes it to a file.\n\n#### Running the Scraper\n\n```python\nfrom webgraze import Wikipedia\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"search\")\nwiki = Wikipedia(filepath='../data.txt', metrics=True)\n\nwiki(queries=queries(), extra_urls=True)\n```\n\n### 4. Unsplash Scraper\n\nThe Unsplash Image scraper fetches images based on given topics \u0026 saves them in their respective folders\n\n#### Configuration\n- Define your search queries like this:\n  ```python\n  search_queries = [\"topic1\", \"topic2\", \"topic3\"]\n  ```\n\n#### Running the Scraper\n\n```python\nfrom webgraze import Unsplash\nfrom webgraze.queries import Queries\n\ntopics = Queries(\"images\")\n\nimage = Unsplash(directory='../images', metrics=True)\nimage(topics=topics())\n```\n\n#### Output:\n```shell\nDownloading 'american football' images:\nDownloading : 100%|██████████████████████████| 176/176 [00:30\u003c00:00,  5.72it/s]\n\nDownloading 'indian festivals' images:\nDownloading : 100%|██████████████████████████| 121/121 [00:30\u003c00:00,  7.29it/s]\n```\n\n### 5. Britannica Scraper\n\nThe Britannica scraper generates target URLs from provided queries, fetches the complete web page, and writes it to a file.\n\n#### Running the scraper\n\n```python\nfrom webgraze import Britannica\nfrom webgraze.queries import Queries\n\nqueries = Queries(category=\"search\")\nscraper = Britannica(filepath='../data.txt', metrics=True)\n\nscraper(queries=queries())\n```\n\n### 6. Freesound Scraper\n\nScraper to download \u0026 save audios from [freesound.org](https://freesound.org/) using its official API. Saves audios in different directories according to the topics.\n\n#### Running the scraper\n\n```python\nimport os\ncurrent_directory = os.path.dirname(os.path.abspath(__file__))\nos.chdir(current_directory)\nfrom dotenv import load_dotenv\nload_dotenv()\n\nAPI_KEY = os.getenv(\"freesound_key\")\n\nfrom webgraze import Freesound\n\nsound = Freesound(api_key=API_KEY, download_dir=\"audios\", metrics=True)\nsound(topics=[\"clicks\", \"background\", \"nature\"])\n```\n\n#### Output\n\n```shell\nDownloading 'clicks' audio files:\nResponse status code: 200\nDownloading 'clicks' audio files: 100%|██████████████████████████████| 10/10 [00:20\u003c00:00,  2.01s/it] \n\nDownloading 'background' audio files:\nResponse status code: 200\nDownloading 'background' audio files: 100%|██████████████████████████████| 10/10 [00:53\u003c00:00,  5.37s/it] \n\nDownloading 'nature' audio files:\nResponse status code: 200\nDownloading 'nature' audio files: 100%|██████████████████████████████| 10/10 [01:57\u003c00:00, 11.78s/it] \n\nFreesound Scraper Metrics:\n\n-------------------------------------------\nTotal topics fetched: 3\nTotal audio files downloaded: 30\nTotal time taken: 3.26 minutes\n-------------------------------------------\n```\n\n### 7. Pexels Scraper\n\nScrapes \u0026 downloads pictures from [pexels.com](https://www.pexels.com/) \u0026 saves them in individual directory topic-wise.\n\n#### Running the scraper\n\n```python\nfrom webgraze import Pexels\nfrom webgraze.queries import Queries\n\nqueries = Queries(\"images\")\nscraper = Pexels(directory=\"./images\", metrics=True)\nscraper(topics=queries())\n```\n\n#### Output\n```shell\nDownloading 'american football' images:\nDownloading: 100%|████████████████████████████████| 24/24 [00:03\u003c00:00,  7.73it/s]\n\nDownloading 'india' images:\nDownloading: 100%|████████████████████████████████| 27/27 [00:04\u003c00:00,  5.99it/s]\n\nDownloading 'europe' images:\nDownloading: 100%|████████████████████████████████| 24/24 [00:06\u003c00:00,  3.55it/s]\n```\n\n## Configuration\n\n- **API Keys and other secrets:** Ensure that your API keys and other sensitive data are stored securely and not hard-coded into your scripts.\n\n- **Search Queries:** The search queries for Wikipedia and Britannica scrapers are defined in `queries.py`.\n\n## Logging\n\nEach scraper logs errors to respective `.log` file. Make sure to check this file for detailed error messages \u0026 troubleshooting information.\n\n## Contribution\nPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.\n\nCheck out [CONTRIBUTING.md](https://github.com/shivendrra/web-graze/blob/main/CONTRIBUTING.md) for more details\n\n## License\n\nThis project is licensed under the MIT License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivendrra%2Fweb-graze","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshivendrra%2Fweb-graze","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivendrra%2Fweb-graze/lists"}