{"id":20228989,"url":"https://github.com/anongecko/scrape","last_synced_at":"2025-11-28T18:05:01.610Z","repository":{"id":253194730,"uuid":"839071013","full_name":"anongecko/scrape","owner":"anongecko","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-07T17:58:23.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-13T23:44:55.287Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anongecko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-06T23:04:03.000Z","updated_at":"2024-08-15T02:41:28.000Z","dependencies_parsed_at":"2024-08-15T04:44:41.866Z","dependency_job_id":"e0ad20fe-6ca9-41fb-bce4-cdaed46f6e60","html_url":"https://github.com/anongecko/scrape","commit_stats":null,"previous_names":["anongecko/scrape"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fscrape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fscrape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fscrape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anongecko%2Fscrape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anongecko","download_url":"https://codeload.github.com/anongecko/scrape/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241670105,"owners_count":20000330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T07:33:50.980Z","updated_at":"2025-11-28T18:05:01.531Z","avatar_url":"https://github.com/anongecko.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Scraper Extraordinaire\n\nA powerful, flexible, and easy-to-use web scraping tool for extracting data from websites.\n\n**Table of Contents**\n\n1. [Features](#features)\n2. [Getting Started](#getting-started)\n3. [Installation](#installation)\n4. [Usage](#usage)\n5. [Examples](#examples)\n6. [Contributing](#contributing)\n7. [License](#license)\n\n## Features\n\n*   **Modular Design**: Easily customize and extend the scraper with your own plugins and scripts.\n*   **Robust Handling**: Automatically handles anti-scraping measures, rate limiting, and connection issues.\n*   **Multi-Threaded**: Quickly scrape large amounts of data with our built-in multi-threading capabilities.\n*   **Data Processing**: Clean, transform, and format your scraped data with our integrated data processing pipeline.\n*   **Support for Multiple Data Formats**: Save your scraped data in CSV, JSON, XML, or any other format you need.\n\n## Getting Started\n\nTo get started with the web scraper, follow these simple steps:\n\n1.  Install the scraper using pip: `pip install -r requirements.txt`\n2.  Create a new instance of the scraper: `scraper = WebScraper()`\n3.  Define your scraping task: `scraper.add_task(url, selector, handler)`\n4.  Run the scraper: `scraper.run()`\n\n## Installation\n\nTo install the web scraper, run the following command:\n\n```bash\npip install -r requirements.txt\n```\n\nThis will install all necessary dependencies and libraries.\n\n## Usage\n\nHere's an example of how to use the web scraper:\n\n```python\nfrom web_scraper import WebScraper\n\n# Create a new instance of the scraper\nscraper = WebScraper()\n\n# Define a scraping task\nscraper.add_task(\n    url=\"https://example.com\",\n    selector=\".title\",\n    handler=lambda x: x.text.strip()\n)\n\n# Run the scraper\nscraper.run()\n```\n\nThis will scrape the title from the webpage and print the result.\n\n## Examples\n\n*   Scrape all links on a webpage:\n\n    ```python\nscraper.add_task(\n    url=\"https://example.com\",\n    selector=\"a\",\n    handler=lambda x: x.get(\"href\")\n)\n```\n\n*   Scrape all images on a webpage:\n\n    ```python\nscraper.add_task(\n    url=\"https://example.com\",\n    selector=\"img\",\n    handler=lambda x: x.get(\"src\")\n)\n```\n\n*   Scrape data from a table:\n\n    ```python\nscraper.add_task(\n    url=\"https://example.com\",\n    selector=\"table tr\",\n    handler=lambda x: [td.text.strip() for td in x.find_all(\"td\")]\n)\n```\n\n## Contributing\n\nWe welcome all contributions! If you'd like to contribute to the project, please fork the repository and submit a pull request.\n\n## License\n\nThis project is licensed under the MIT License. See LICENSE for details.\n\nBy using this scraper, you acknowledge that you have read and agree to the terms of the license.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanongecko%2Fscrape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanongecko%2Fscrape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanongecko%2Fscrape/lists"}