{"id":16837422,"url":"https://github.com/xunzhuo/airspider","last_synced_at":"2025-03-23T15:31:34.715Z","repository":{"id":62559785,"uuid":"248710928","full_name":"Xunzhuo/AirSpider","owner":"Xunzhuo","description":"A Fast and Light Python Spider Framework 🕷️","archived":false,"fork":false,"pushed_at":"2023-11-14T09:39:06.000Z","size":14018,"stargazers_count":8,"open_issues_count":1,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T21:54:18.432Z","etag":null,"topics":["asynchronous","crawler","crawler-python","distributed","python3","redis","spider","spider-framework","web"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/airspider/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Xunzhuo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-20T09:01:12.000Z","updated_at":"2021-09-22T15:39:27.000Z","dependencies_parsed_at":"2024-10-28T16:40:04.160Z","dependency_job_id":"64b66303-f390-4a3f-bcdb-a94928d0843d","html_url":"https://github.com/Xunzhuo/AirSpider","commit_stats":{"total_commits":89,"total_committers":7,"mean_commits":"12.714285714285714","dds":0.3370786516853933,"last_synced_commit":"a56e4b1c640e19113b2b078c9a8e7f3a02b2f721"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xunzhuo%2FAirSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xunzhuo%2FAirSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xunzhuo%2FAirSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xunzhuo%2FAirSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Xunzhuo","download_url":"https://codeload.github.com/Xunzhuo/AirSpider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245122836,"owners_count":20564387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","crawler","crawler-python","distributed","python3","redis","spider","spider-framework","web"],"created_at":"2024-10-13T12:17:16.102Z","updated_at":"2025-03-23T15:31:33.656Z","avatar_url":"https://github.com/Xunzhuo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://picreso.oss-cn-beijing.aliyuncs.com/airs.png\" width=\"150px\"\u003e\n  \u003ch1 align=\"center\"\u003e\u003cimg src=\"http://picreso.oss-cn-beijing.aliyuncs.com/font.png\" /\u003e\u003c/h1\u003e\n  \u003cp align=\"center\" style=\"font-weight:bold\"\u003e\n    Distributed  -  Asynchronous  - Easy-to-use\n  \u003cbr\u003e\n   \u003cbr\u003e\n \t\tAn Easy-to-use and Fast Python Spider Framework\n  \u003cimg src=\"http://picreso.oss-cn-beijing.aliyuncs.com/airdemo.png\"\u003e\n\u003c/p\u003e\n\n\n\n\n---\n\n## Overview\n\n\u003e + **AirSpider** is a high-performance asynchronous crawler framework for developers \n\u003e + Based on Redis: task distribution, task deduplication, and distributed ☁️\n\n## Requirements\n\n- Python 3.6\n- Works on Linux, Windows, macOS\n\n## Features\n\n+ Quick to Start \n+ Low Coupling \n+ High Cohesion\n+ Easy Expansion \n+ Orderly Workflow \n\n## Installation\n\n``` shell\n# For Linux \u0026\u0026 MacOS\npip3 install airspider\n\n\n# For Windows\npip3 install airspider\n\n\n```\n\n  ## Documents\n\n  + [Introduction](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/introduction.md)：Introduction to **AirSpider**\n  + [Tutorials](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/tutorials.md)： Quick to program a Spider By **AirSpider**\n  + [Plugins](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/plugins.md)：Program extensions for **AirSpider**\n\n  ## Topics\n\n  - [Item](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/item.md)：Define Target in HTML\n  - [Selector](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/selector.md)：Select Target from HTML\n  - [Request](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/request.md)：Request Part\n  - [Response](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/response.md)：Response Part\n  - [Middleware](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/middleware.md)：Make Airspider Support to Extensions\n  - [Spider](https://github.com/Xunzhuo/AirSpider/tree/master/Docs/cn/topics/spider.md)：Entrance for AirSpider\n\n## TODO\n\n+ Complete Plugins of Redis\n+ Complete Distributed Architecture\n\n## Contributing\n\n**AirSpider**🕷️ is still under **Developing**\n\n\u003e Feel free to open issues💬 and pull requests\n\n- Report or Fix bugs\n- Build Powerful plugins\n- Make documentation Better\n- Add Examples of Spiders \n\n![](http://picreso.oss-cn-beijing.aliyuncs.com/monkey.jpg)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxunzhuo%2Fairspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxunzhuo%2Fairspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxunzhuo%2Fairspider/lists"}