Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/xunzhuo/airspider
A Fast and Light Python Spider Framework 🕷️
https://github.com/xunzhuo/airspider
asynchronous crawler crawler-python distributed python3 redis spider spider-framework web
Last synced: 2 months ago
JSON representation
A Fast and Light Python Spider Framework 🕷️
- Host: GitHub
- URL: https://github.com/xunzhuo/airspider
- Owner: Xunzhuo
- License: apache-2.0
- Created: 2020-03-20T09:01:12.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-11-14T09:39:06.000Z (about 1 year ago)
- Last Synced: 2024-10-14T12:17:13.105Z (3 months ago)
- Topics: asynchronous, crawler, crawler-python, distributed, python3, redis, spider, spider-framework, web
- Language: Python
- Homepage: https://pypi.org/project/airspider/
- Size: 13.4 MB
- Stars: 8
- Watchers: 2
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
Distributed - Asynchronous - Easy-to-use
An Easy-to-use and Fast Python Spider Framework
---
## Overview
> + **AirSpider** is a high-performance asynchronous crawler framework for developers
> + Based on Redis: task distribution, task deduplication, and distributed ☁️
## Requirements
- Python 3.6
- Works on Linux, Windows, macOS
## Features
+ Quick to Start
+ Low Coupling
+ High Cohesion
+ Easy Expansion
+ Orderly Workflow
## Installation
``` shell
# For Linux && MacOS
pip3 install airspider
# For Windows
pip3 install airspider
```
## Documents
+ [Introduction](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/introduction.md)：Introduction to **AirSpider**
+ [Tutorials](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/tutorials.md)： Quick to program a Spider By **AirSpider**
+ [Plugins](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/plugins.md)：Program extensions for **AirSpider**
## Topics
- [Item](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/item.md)：Define Target in HTML
- [Selector](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/selector.md)：Select Target from HTML
- [Request](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/request.md)：Request Part
- [Response](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/response.md)：Response Part
- [Middleware](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/middleware.md)：Make Airspider Support to Extensions
- [Spider](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/spider.md)：Entrance for AirSpider
## TODO
+ Complete Plugins of Redis
+ Complete Distributed Architecture
## Contributing
**AirSpider**🕷️ is still under **Developing**
> Feel free to open issues💬 and pull requests
- Report or Fix bugs
- Build Powerful plugins
- Make documentation Better
- Add Examples of Spiders
