awesome
a list of some interesting repositories, tools
https://github.com/lin-zone/awesome
Last synced: 23 days ago
JSON representation
Crawler
- Douyin-Bot - Python 抖音机器人，论如何在抖音上找到漂亮小姐姐
- SinaSpider - 新浪微博爬虫（Scrapy、Redis）
- ECommerceCrawlers - 实战多种网站、电商数据爬虫
- examples-of-web-crawlers - 一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站
- PythonCrawler - 用python编写的爬虫项目集合
- amemv-crawler - 下载指定的抖音号的视频,抖音爬虫
- course-crawler - 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载
- zhihu_crawler - Crawler of zhihu.com
- python-spider - Python3网络爬虫实战
- Anti-Anti-Spider - 处理反爬
- FunpySpiderSearchEngine - Scrapy 1.6.0爬取数据 + ElasticSearch6.8.0+Django2.2搜索引擎
- spider163 - 抓取网易云音乐热门评论
- Python-Spider - Python 爬虫
- ScrapyProject - Scrapy实战项目合集
- awesome-spider - 爬虫集合
HTML parser
- scrapely - A pure-python HTML screen-scraping library
Scrapy Distributed
- crawlab - 基于Golang的分布式爬虫管理平台
- Gerapy - Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapydweb - ScrapydWeb: Web app for Scrapyd cluster management
Scrapy Middleware
- scrapy-mongodb - MongoDB pipeline for Scrapy
- scrapy-splitvariants - Scrapy spider middleware to split an item into multiple items using a multi-valued key
- scrapy-proxies - Random proxy middleware for Scrapy
- scrapy-fake-useragent - Random User-Agent middleware based on fake-useragent
- scrapy-selenium - Scrapy middleware to handle javascript pages using selenium
- scrapy-deltafetch - Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
- scrapy-random-useragent - Scrapy Middleware to set a random User-Agent for every Request.
- scrapy-crawl-once - Scrapy middleware which allows to crawl only new content
- scrapy-magicfields - Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
- scrapy-crawlera - Crawlera middleware for Scrapy
Tools
- qrcode - Python 艺术二维码生成器
utils
- queuelib - Collection of persistent (disk-based) queues