awesome-scrapy
A curated list of awesome packages, articles, and other cool resources from the Scrapy community.
https://github.com/AccordBox/awesome-scrapy
Last synced: 18 days ago
JSON representation
Apps
Avoid Ban
- HttpProxyMiddleware
- scrapy-proxies
- scrapy-rotating-proxies
- scrapy-random-useragent - Agent for every Request.
- scrapy-fake-useragent - Agent middleware based on fake-useragent
Data Processing
- scrapy-elasticsearch
- scrapy-mongodb
- scrapy-mysql-pipeline
- scrapy-s3pipeline
- scrapy-sqs-exporter
- scrapy-kafka-export
- scrapy-rss-exporter
- scrapy-elasticsearch
Distributed Spider
- scrapy-cluster
- scrapy-redis - based components for Scrapy.
Front-End Scrapy Managers
- SpiderKeeper
- Gerapy - Client, Scrapyd-API, Django and Vue.js
- ScrapydWeb
Monitor
Other Useful Extensions
- scrapy-djangoitem
- scrapy-deltafetch
- scrapy-crawl-once - crawling pages which were already downloaded in previous crawls.
- scrapy-magicfields
- scrapy-pagestorage
- itemloaders
- itemadapter
- scrapy-poet
Process Javascript
- scrapy-playwright
- scrapy-puppeteer - rendered pages.
- scrapy-splash
Scrapy Service
- scrapyscript - no project required.
- scrapyd
- scrapyd-client
- python-scrapyd-api
- scrapyrt
Visual Web Scraping
Resources
Articles
- Scrapy VS Beautiful Soup
- Scrapy VS Selenium
- Web Scraping in Python using Scrapy (with multiple examples)
- Scrapy Tutorial Series: Web Scraping Using Python
- Advanced Web Scraping: Bypassing "403 Forbidden," captchas, and more
- Scrapy VS Selenium