https://github.com/xiantang/mini_scrapy

模仿scrapy的轻量级爬虫框架
https://github.com/xiantang/mini_scrapy

crawler python3 requets scrapy

Last synced: 3 days ago
JSON representation

模仿scrapy的轻量级爬虫框架

• Host: GitHub
• URL: https://github.com/xiantang/mini_scrapy
• Owner: xiantang
• Created: 2018-07-16T05:55:03.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2019-05-10T14:25:15.000Z (over 5 years ago)
• Last Synced: 2025-01-27T17:58:50.798Z (8 days ago)
• Topics: crawler, python3, requets, scrapy
• Language: Python
• Size: 6.92 MB
• Stars: 2
• Watchers: 0
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# mini_scrapy

模仿scrapy的轻量级爬虫框架

- [x] 支持中间件，可以从`@from_crawler`中读取配置。

- [ ] 实现断点续爬/爬虫中断之后恢复爬取队列和记录已爬取的队列

- [ ] 合并ORM到爬虫

- [ ] 实现批量入库

- [ ] 支持断线重连