awesome-scrapy
A curated list of awesome packages, articles, and other cool resources from the Scrapy community.
https://github.com/AccordBox/awesome-scrapy
Last synced: about 3 hours ago
JSON representation
-
Resources
-
Articles
-
Exercises
-
Video
-
Book
-
-
Apps
-
Visual Web Scraping
-
Distributed Spider
- scrapy-cluster
- scrapy-redis - based components for Scrapy.
-
Scrapy Service
- scrapyscript - **no project required.**
- scrapyd
- scrapyd-client
- python-scrapyd-api
- scrapyrt
-
Front-End Scrapy Managers
- Gerapy - Client, Scrapyd-API, Django and Vue.js
- ScrapydWeb
- SpiderKeeper
-
Monitor
-
Avoid Ban
- HttpProxyMiddleware
- scrapy-proxies
- scrapy-rotating-proxies
- scrapy-random-useragent - Agent for every Request.
- scrapy-fake-useragent - Agent middleware based on fake-useragent
- scrapy-crawlera
-
Data Processing
-
Process Javascript
- scrapy-playwright
- scrapy-puppeteer - rendered pages.
- scrapy-splash
-
Other Useful Extensions
- scrapy-djangoitem
- scrapy-deltafetch
- scrapy-crawl-once - crawling pages which were already downloaded in previous crawls.
- scrapy-magicfields
- scrapy-pagestorage
- itemloaders
- itemadapter
- scrapy-poet
-
Programming Languages
Categories
Sub Categories
Keywords
scrapy
18
python
8
scraping
5
crawler
4
spider
3
scrapyd
3
python3
3
dashboard
3
crawling
3
distributed
3
proxy
2
redis
2
scrapyd-ui
2
hacktoberfest2021
2
scrapyd-control
1
scrapyd-cluster-management
1
scrapyd-api
1
scrapyd-admin
1
scrapyd-keeper
1
scrapy-visualization
1
scrapy-log-analysis
1
log-parsing
1
log-analysis
1
webspider
1
scrapyd-log-analysis
1
vuejs
1
vue
1
kafka
1
gerapy
1
docker
1
django
1
api-wrapper
1
webcrawling
1
webcrawler
1
twisted
1
scraper
1
scrapy-spider
1
sqs
1
chrome-headless
1
firefox-headless
1
headless-browser
1
javascript-renderer
1
playwright
1
playwright-python
1
python-asyncio
1
webkit-headless
1
puppeteer
1
headless-browsers
1
metadata
1
python-attrs
1