Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
awesome
a list of some interesting repositories, tools
https://github.com/lin-zone/awesome
Last synced: 4 days ago
JSON representation
-
Scrapy Distributed
- crawlab - 基于Golang的分布式爬虫管理平台
- Gerapy - Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapydweb - ScrapydWeb: Web app for Scrapyd cluster management
-
Scrapy Middleware
- scrapy-mongodb - MongoDB pipeline for Scrapy
- scrapy-splitvariants - Scrapy spider middleware to split an item into multiple items using a multi-valued key
- scrapy-proxies - Random proxy middleware for Scrapy
- scrapy-fake-useragent - Random User-Agent middleware based on fake-useragent
- scrapy-selenium - Scrapy middleware to handle javascript pages using selenium
- scrapy-crawlera - Crawlera middleware for Scrapy
- scrapy-deltafetch - Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
- scrapy-random-useragent - Scrapy Middleware to set a random User-Agent for every Request.
- scrapy-crawl-once - Scrapy middleware which allows to crawl only new content
- scrapy-magicfields - Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
- crawlera - The World's Smartest Proxy Network
- scrapy-crawlera - Crawlera middleware for Scrapy
-
command line
-
HTML parser
- scrapely - A pure-python HTML screen-scraping library
-
Crawler
- Douyin-Bot - Python 抖音机器人，论如何在抖音上找到漂亮小姐姐
- SinaSpider - 新浪微博爬虫（Scrapy、Redis）
- ECommerceCrawlers - 实战多种网站、电商数据爬虫
- examples-of-web-crawlers - 一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站
- PythonCrawler - 用python编写的爬虫项目集合
- amemv-crawler - 下载指定的抖音号的视频,抖音爬虫
- course-crawler - 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载
- zhihu_crawler - Crawler of zhihu.com
- awesome-spider - 爬虫集合
- python-spider - Python3网络爬虫实战
- Anti-Anti-Spider - 处理反爬
- FunpySpiderSearchEngine - Scrapy 1.6.0爬取数据 + ElasticSearch6.8.0+Django2.2搜索引擎
- spider163 - 抓取网易云音乐热门评论
- Python-Spider - Python 爬虫
- ScrapyProject - Scrapy实战项目合集
-
Tools
- qrcode - Python 艺术二维码生成器
-
utils
- queuelib - Collection of persistent (disk-based) queues
Categories
Sub Categories
Keywords
python
11
scrapy
11
spider
9
crawler
6
python3
5
django
3
webspider
3
selenium
3
scrapyd
2
plugin
2
redis
2
mysql
2
proxy
2
command-line
2
cli
2
scraping
2
crawler-detection
2
wechat
2
dashboard
2
scrapyd-ui
2
lagou
2
docker
2
crawling
1
web-scraping
1
scrapyd-visualization
1
hacktoberfest2021
1
alitask
1
baidu
1
baidu-tieba
1
baotu
1
boss
1
ctrip
1
dazhong-spider
1
crawlab
1
crawling-tasks
1
go
1
platform
1
spiders-management
1
web-crawler
1
webcrawler
1
distributed
1
gerapy
1
vue
1
vuejs
1
log-analysis
1
log-parsing
1
scrapy-log-analysis
1
scrapy-visualization
1
scrapyd-admin
1
scrapyd-api
1