{"id":26238035,"url":"https://github.com/hupe1980/scrapyrunner","last_synced_at":"2026-02-18T09:01:46.925Z","repository":{"id":272181422,"uuid":"915590669","full_name":"hupe1980/scrapyrunner","owner":"hupe1980","description":"A Python library to run Scrapy spiders directly from your code.","archived":false,"fork":false,"pushed_at":"2025-01-16T18:27:45.000Z","size":71,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-31T01:20:53.578Z","etag":null,"topics":["scraping","scrapy","web","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hupe1980.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-12T09:18:28.000Z","updated_at":"2025-01-16T18:27:47.000Z","dependencies_parsed_at":"2025-03-13T05:32:01.707Z","dependency_job_id":"aad64d6a-7c13-481b-81c3-299821ceba0b","html_url":"https://github.com/hupe1980/scrapyrunner","commit_stats":null,"previous_names":["hupe1980/scrapyrunner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hupe1980/scrapyrunner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hupe1980%2Fscrapyrunner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hupe1980%2Fscrapyrunner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hupe1980%2Fscrapyrunner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hupe1980%2Fscrapyrunner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hupe1980","download_url":"https://codeload.github.com/hupe1980/scrapyrunner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hupe1980%2Fscrapyrunner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29574065,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T08:38:15.585Z","status":"ssl_error","status_checked_at":"2026-02-18T08:38:14.917Z","response_time":162,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scraping","scrapy","web","webscraping"],"created_at":"2025-03-13T05:31:56.053Z","updated_at":"2026-02-18T09:01:46.888Z","avatar_url":"https://github.com/hupe1980.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# ScrapyRunner\n\nA Python library to run Scrapy spiders directly from your code.\n\n## Overview\n\nScrapyRunner is a lightweight library that enables you to run Scrapy spiders in your Python code, process scraped items using custom processors, and manage Scrapy signals seamlessly. It simplifies the process of starting and managing Scrapy spiders and integrates well with your existing Python workflows.\n\n## Features\n\n- Run Scrapy spiders directly from Python code.\n- Process scraped items in batches with a custom processor.\n- Manage Scrapy signals (e.g., on item scraped, on engine stopped).\n- Easy integration with the Scrapy framework.\n- Asynchronous processing of items using Twisted.\n\n## Installation\n\nTo install ScrapyRunner, you can use `pip`:\n\n```bash\npip install scrapyrunner\n```\n\n## Usage\n\n### Example\n\n```python\n# Importing necessary libraries\nfrom dataclasses import dataclass  # Used to create data classes for the processor\nfrom time import sleep  # Used to simulate a delay during item processing\n\nimport scrapy  # Scrapy library for creating spiders\n\nfrom scrapyrunner import ItemProcessor, ScrapyRunner  # Importing the custom Scrapy runner and processor classes\n\n\n# Define the spider to crawl a webpage and extract data\nclass MySpider(scrapy.Spider):\n    name = 'example'  # Name of the spider, used to identify it when running\n\n    def parse(self, response):\n        # This method is called to parse the response from the URL.\n        # We extract the title of the page using XPath and return it as a dictionary.\n        data = response.xpath(\"//title/text()\").extract_first()\n        return {\"title\": data}  # Returning the extracted title in a dictionary format\n\n# Define the item processor to process the items after they are scraped\n@dataclass(kw_only=True)\nclass MyProcessor(ItemProcessor):\n    prefix: str\n    suffix: str\n\n    def process_item(self, item: scrapy.Item) -\u003e None:\n        # A simulated delay is added here to mimic real processing time.\n        # In a real-world scenario, this could be a time-consuming task like data validation or saving to a database.\n        print(self.prefix, item, self.suffix)  # Print the processed item to the console\n        sleep(2)  # Sleep for 2 seconds to simulate processing time\n\n# Main block to execute the spider and processor\nif __name__ == '__main__':\n    # Create an instance of ScrapyRunner with the specified spider and processor.\n    # ScrapyRunner will handle crawling and managing the queue for items.\n    scrapy_runner = ScrapyRunner(spider=MySpider, processor=MyProcessor, processor_kwargs={\"prefix\": \"\u003e\u003e\u003e\", \"suffix\": \"\u003c\u003c\u003c\"})\n\n    # Run the Scrapy crawler, passing the starting URL to the spider\n    # The spider will start scraping the provided URL and the processor will handle the items.\n    scrapy_runner.run(start_urls=[\"https://example.org\", \"https://scrapy.org\"])  # Run the spider with the start URL\n```\n\n### How it works:\n\n1. **Define a Spider**: In this example, `MySpider` extracts the title of a webpage.\n2. **Define a Processor**: `MyProcessor` processes scraped items (here it simply sleeps for 2 seconds to simulate real processing).\n3. **Run the ScrapyRunner**: The `ScrapyRunner` class is used to run the spider and process the items. The `run()` method triggers the scraping, and each item scraped is passed to the custom processor.\n\n## Customization\n\n### Custom Processor\n\nTo create your own custom processor:\n\n1. Subclass `ItemProcessor`.\n2. Override the `process_item()` method to handle scraped items.\n3. Process each item as needed (e.g., save to a database, perform additional transformations, etc.).\n\n```python\n@dataclass(kw_only=True)\nclass MyCustomProcessor(ItemProcessor):\n    def process_item(self, item: scrapy.Item) -\u003e None:\n        # Custom processing logic goes here\n        print(\"Processing item:\", item)\n```\n\n### Custom Settings\n\nYou can pass custom Scrapy settings to `ScrapyRunner`:\n\n```python\nscrapy_settings = {\n    \"LOG_LEVEL\": \"DEBUG\",\n    \"USER_AGENT\": \"MyCustomAgent\",\n    # Add more custom settings as needed\n}\n\nrunner = ScrapyRunner(spider=MySpider, processor=MyProcessor, scrapy_settings=scrapy_settings)\nrunner.run(start_urls=[\"https://example.org\", \"https://scrapy.org\"])\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhupe1980%2Fscrapyrunner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhupe1980%2Fscrapyrunner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhupe1980%2Fscrapyrunner/lists"}