{"id":13796819,"url":"https://github.com/accordbox/awesome-scrapy","last_synced_at":"2025-08-06T15:31:51.190Z","repository":{"id":38205902,"uuid":"118918479","full_name":"AccordBox/awesome-scrapy","owner":"AccordBox","description":"A curated list of awesome packages, articles, and other cool resources from the Scrapy community.","archived":false,"fork":false,"pushed_at":"2022-12-28T08:59:28.000Z","size":53,"stargazers_count":524,"open_issues_count":2,"forks_count":64,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-05-28T21:45:28.245Z","etag":null,"topics":["awesome","awesome-scrapy","python","scrapy"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AccordBox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-25T13:49:58.000Z","updated_at":"2024-05-28T15:24:11.000Z","dependencies_parsed_at":"2023-01-31T06:15:29.657Z","dependency_job_id":null,"html_url":"https://github.com/AccordBox/awesome-scrapy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccordBox%2Fawesome-scrapy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccordBox%2Fawesome-scrapy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccordBox%2Fawesome-scrapy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccordBox%2Fawesome-scrapy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AccordBox","download_url":"https://codeload.github.com/AccordBox/awesome-scrapy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228920496,"owners_count":17992005,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome","awesome-scrapy","python","scrapy"],"created_at":"2024-08-03T23:01:15.761Z","updated_at":"2024-12-09T16:16:53.837Z","avatar_url":"https://github.com/AccordBox.png","language":null,"funding_links":[],"categories":["\u003ca id=\"8c5a692b5d26527ef346687e047c5c21\"\u003e\u003c/a\u003e收集"],"sub_categories":[],"readme":"# Awesome Scrapy [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) \n\n\u003e A curated list of awesome packages, articles, and other cool resources from the Scrapy community.\n\u003e [Scrapy](https://github.com/scrapy/scrapy) is a fast high-level web crawling \u0026 scraping framework for Python.\n\n## Table of Contents\n\n- [Apps](#apps)\n    - [Visual Web Scraping](#visual-web-scraping)\n    - [Distributed Spider](#distributed-spider)\n    - [Scrapy Service](#scrapy-service)\n    - [Front-End Scrapy Managers](#front-end-scrapy-managers)\n    - [Monitor](#monitor)\n    - [Avoid Ban](#avoid-ban)\n    - [Data Processing](#data-processing)\n    - [Process Javascript](#process-javascript)\n    - [Other Useful Extensions](#other-useful-extensions)\n- [Resources](#resources)\n    - [Articles](#articles)\n    - [Exercises](#exercises)\n    - [Video](#video)\n    - [Book](#book)\n\n## Apps\n\n### Visual Web Scraping\n\n* [Portia](https://github.com/scrapinghub/portia) Visual scraping for Scrapy\n\n### Distributed Spider\n\n* [scrapy-cluster](https://github.com/istresearch/scrapy-cluster) Distributed on demand scraping cluster using Redis and Kafka.\n* [scrapy-redis](https://github.com/rmax/scrapy-redis) Redis-based components for Scrapy.\n\n### Scrapy Service\n\n* [scrapyscript](https://github.com/jschnurr/scrapyscript) Run a Scrapy spider programmatically from a script or a Celery task - **no project required.**\n\n* [scrapyd](https://github.com/scrapy/scrapyd) A service daemon to run Scrapy spiders\n\n* [scrapyd-client](https://github.com/scrapy/scrapyd-client) Command line client for Scrapyd server\n\n* [python-scrapyd-api](https://github.com/djm/python-scrapyd-api) A Python wrapper for working with Scrapyd's API.\n\n* [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) A scalable admin ui for spider service\n\n* [scrapyrt](https://github.com/scrapinghub/scrapyrt) HTTP server which provides API for scheduling Scrapy spiders and making requests with spiders.\n\n### Front-End Scrapy Managers\n\n* [Gerapy](https://github.com/Gerapy/Gerapy) Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js\n\n* [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) admin ui for scrapy/open source scrapinghub.\n\n* [ScrapydWeb](https://github.com/my8100/scrapydweb) Scrapyd cluster management, Scrapy log analysis \u0026 visualization, Basic auth, Auto packaging, Timer Tasks, Email notice, and Mobile UI.\n\n### Monitor\n\n* [scrapy-sentry](https://github.com/llonchj/scrapy-sentry) Logs Scrapy exceptions into Sentry\n\n* [scrapy-statsd-middleware](https://github.com/zachgoldstein/scrapy-statsd) Statsd integration middleware for scrapy\n\n* [scrapy-jsonrpc](https://github.com/scrapy-plugins/scrapy-jsonrpc) An extension to control a running Scrapy web crawler via JSON-RPC\n\n* [scrapy-fieldstats](https://github.com/stummjr/scrapy-fieldstats) A Scrapy extension to log items coverage when the spider shuts down\n\n* [spidermon](https://github.com/scrapinghub/spidermon) Extension which provides useful tools for data validation, stats monitoring, and notification messages.\n\n### Avoid Ban\n\n* [HttpProxyMiddleware](https://github.com/kohn/HttpProxyMiddleware) A middleware for scrapy. Used to change HTTP proxy from time to time.\n\n* [scrapy-proxies](https://github.com/aivarsk/scrapy-proxies) Processes Scrapy requests using a random proxy from list to avoid IP ban and improve crawling speed.\n\n* [scrapy-rotating-proxies](https://github.com/TeamHG-Memex/scrapy-rotating-proxies) Use multiple proxies with Scrapy\n\n* [scrapy-random-useragent](https://github.com/cnu/scrapy-random-useragent) Scrapy Middleware to set a random User-Agent for every Request.\n\n* [scrapy-fake-useragent](https://github.com/alecxe/scrapy-fake-useragent) Random User-Agent middleware based on fake-useragent\n\n* [scrapy-crawlera](https://github.com/scrapy-plugins/scrapy-crawlera) Crawlera routes requests through a pool of IPs, throttling access by introducing delays and discarding IPs from the pool when they get banned from certain domains, or have other problems.\n\n### Data Processing\n\n* [scrapy-elasticsearch](https://github.com/knockrentals/scrapy-elasticsearch) A scrapy pipeline which send items to Elastic Search server\n\n* [scrapy-mongodb](https://github.com/sebdah/scrapy-mongodb) MongoDB pipeline for Scrapy.\n\n* [scrapy-mysql-pipeline](https://github.com/IaroslavR/scrapy-mysql-pipeline) MySQL pipeline to persist items in MySQL databases.\n\n* [scrapy-s3pipeline](https://github.com/orangain/scrapy-s3pipeline) Scrapy pipeline to store chunked items into AWS S3 bucket\n\n* [scrapy-sqs-exporter](https://github.com/multiplechoice/scrapy-sqs-exporter) Scrapy extension for outputting scraped items to an Amazon SQS instance\n\n* [scrapy-kafka-export](https://github.com/TeamHG-Memex/scrapy-kafka-export) Scrapy extension which writes crawled items to Kafka\n\n* [scrapy-rss-exporter](https://github.com/ljanyst/scrapy-rss-exporter) An RSS exporter for Scrapy\n\n### Process Javascript\n\n* [scrapy-playwright](https://github.com/scrapy-plugins/scrapy-playwright) Enable scraping dynamic pages using PlayWright.\n* [scrapy-puppeteer](https://github.com/clemfromspace/scrapy-puppeteer) Make Scrapy and Puppeteer work together to handle Javascript-rendered pages.\n* [scrapy-splash](https://github.com/scrapy-plugins/scrapy-splash) Make Scrapy can understand Javascript\n\n### Other Useful Extensions\n\n* [scrapy-djangoitem](https://github.com/scrapy-plugins/scrapy-djangoitem) Scrapy extension to write scraped items using Django models\n\n* [scrapy-deltafetch](https://github.com/scrapy-plugins/scrapy-deltafetch) Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls\n\n* [scrapy-crawl-once](https://github.com/TeamHG-Memex/scrapy-crawl-once) This package provides a Scrapy middleware which allows to avoid re-crawling pages which were already downloaded in previous crawls.\n\n* [scrapy-magicfields](https://github.com/scrapy-plugins/scrapy-magicfields) Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.\n\n* [scrapy-pagestorage](https://github.com/scrapy-plugins/scrapy-pagestorage) A scrapy extension to store requests and responses information in storage service.\n\n* [itemloaders](https://github.com/scrapy/itemloaders) Library to populate items using XPath and CSS with a convenient API.\n\n* [itemadapter](https://github.com/scrapy/itemadapter) Adapter which provides a common interface to handle objects of different types in an uniform manner.\n\n* [scrapy-poet](https://github.com/scrapinghub/scrapy-poet) Page Object pattern implementation which enables writing reusable and portable extraction and crawling code.\n\n## Resources\n\n### Articles\n\n* [Scrapy VS Beautiful Soup](https://blog.michaelyin.info/2017/08/10/scrapy-tutorial-1-scrapy-vs-beautiful-soup/)\n\n* [Scrapy VS Selenium](https://blog.michaelyin.info/2017/11/06/web-scraping-framework-review-scrapy-vs-selenium/)\n\n* [Web Scraping in Python using Scrapy (with multiple examples)](https://www.analyticsvidhya.com/blog/2017/07/web-scraping-in-python-using-scrapy/)\n\n* [Scrapy Tutorial Series: Web Scraping Using Python](https://www.accordbox.com/blog/scrapy-tutorial-series-web-scraping-using-python/)\n\n* [Advanced Web Scraping: Bypassing \"403 Forbidden,\" captchas, and more](http://sangaline.com/post/advanced-web-scraping-tutorial/)\n\n### Exercises\n\n* [Learn and improve web scraping skills with Web Scraping Exercises](https://scrapingclub.com/)\n\n### Video\n\n* [Scrapy: Powerful Web Scraping \u0026 Crawling with Python](https://www.udemy.com/scrapy-tutorial-web-scraping-with-python/) Online courses on Udemy\n\n### Book\n\n* [Learning Scrapy](https://www.amazon.com/Learning-Scrapy-Dimitrios-Kouzis-Loukas/dp/1784399787/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccordbox%2Fawesome-scrapy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faccordbox%2Fawesome-scrapy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccordbox%2Fawesome-scrapy/lists"}