{"id":15038071,"url":"https://github.com/boris-code/feapder","last_synced_at":"2025-05-14T04:10:23.603Z","repository":{"id":37750339,"uuid":"336937837","full_name":"Boris-code/feapder","owner":"Boris-code","description":"🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度","archived":false,"fork":false,"pushed_at":"2025-03-17T11:27:30.000Z","size":1551,"stargazers_count":3250,"open_issues_count":97,"forks_count":509,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-05-13T09:57:58.850Z","etag":null,"topics":["crawler","feapder","feaplat","python","scrapy","spider"],"latest_commit_sha":null,"homepage":"http://feapder.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Boris-code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-02-08T02:21:53.000Z","updated_at":"2025-05-13T09:33:34.000Z","dependencies_parsed_at":"2024-01-13T17:31:48.509Z","dependency_job_id":"96496542-a424-499d-a5e2-95cc6a131277","html_url":"https://github.com/Boris-code/feapder","commit_stats":{"total_commits":672,"total_committers":21,"mean_commits":32.0,"dds":"0.30952380952380953","last_synced_commit":"e08f843fef6f72cd3b450e5c3b80f2d94001a930"},"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Boris-code%2Ffeapder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Boris-code%2Ffeapder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Boris-code%2Ffeapder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Boris-code%2Ffeapder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Boris-code","download_url":"https://codeload.github.com/Boris-code/feapder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254070121,"owners_count":22009559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","feapder","feaplat","python","scrapy","spider"],"created_at":"2024-09-24T20:36:59.045Z","updated_at":"2025-05-14T04:10:18.592Z","avatar_url":"https://github.com/Boris-code.png","language":"Python","readme":"# FEAPDER\n\n![](https://img.shields.io/badge/python-3.6-brightgreen)\n![](https://img.shields.io/github/watchers/Boris-code/feapder?style=social)\n![](https://img.shields.io/github/stars/Boris-code/feapder?style=social)\n![](https://img.shields.io/github/forks/Boris-code/feapder?style=social)\n[![Downloads](https://pepy.tech/badge/feapder)](https://pepy.tech/project/feapder)\n[![Downloads](https://pepy.tech/badge/feapder/month)](https://pepy.tech/project/feapder)\n[![Downloads](https://pepy.tech/badge/feapder/week)](https://pepy.tech/project/feapder)\n\n## 简介\n\n1. feapder是一款上手简单，功能强大的Python爬虫框架，内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。\n2. 支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。\n3. 更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度\n\n读音: `[ˈfiːpdə]`\n\n![feapder](http://markdown-media.oss-cn-beijing.aliyuncs.com/2023/09/04/feapder.jpg)\n\n\n## 文档地址\n\n- 官方文档：https://feapder.com\n- github：https://github.com/Boris-code/feapder\n- 更新日志：https://github.com/Boris-code/feapder/releases\n- 爬虫管理系统：http://feapder.com/#/feapder_platform/feaplat\n\n\n## 环境要求：\n\n- Python 3.6.0+\n- Works on Linux, Windows, macOS\n\n## 安装\n\nFrom PyPi:\n\n精简版\n\n```shell\npip install feapder\n```\n\n浏览器渲染版：\n```shell\npip install \"feapder[render]\"\n```\n\n完整版：\n\n```shell\npip install \"feapder[all]\"\n```\n\n三个版本区别：\n\n1. 精简版：不支持浏览器渲染、不支持基于内存去重、不支持入库mongo\n2. 浏览器渲染版：不支持基于内存去重、不支持入库mongo\n3. 完整版：支持所有功能\n\n完整版可能会安装出错，若安装出错，请参考[安装问题](docs/question/安装问题.md)\n\n## 小试一下\n\n创建爬虫\n\n```shell\nfeapder create -s first_spider\n```\n\n创建后的爬虫代码如下：\n\n```python\nimport feapder\n\n\nclass FirstSpider(feapder.AirSpider):\n    def start_requests(self):\n        yield feapder.Request(\"https://www.baidu.com\")\n\n    def parse(self, request, response):\n        print(response)\n\n\nif __name__ == \"__main__\":\n    FirstSpider().start()\n        \n```\n\n直接运行，打印如下：\n\n```shell\nThread-2|2021-02-09 14:55:11,373|request.py|get_response|line:283|DEBUG|\n                -------------- FirstSpider.parse request for ----------------\n                url  = https://www.baidu.com\n                method = GET\n                body = {'timeout': 22, 'stream': True, 'verify': False, 'headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36'}}\n\n\u003cResponse [200]\u003e\nThread-2|2021-02-09 14:55:11,610|parser_control.py|run|line:415|DEBUG| parser 等待任务...\nFirstSpider|2021-02-09 14:55:14,620|air_spider.py|run|line:80|INFO| 无任务，爬虫结束\n```\n\n代码解释如下：\n\n1. start_requests： 生产任务\n2. parse： 解析数据\n\n## 参与贡献\n\n贡献之前请先阅读 [贡献指南](./CONTRIBUTING.md)\n\n感谢所有做过贡献的人!\n\n\u003ca href=\"https://github.com/Boris-code/feapder/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=Boris-code/feapder\" /\u003e\n\u003c/a\u003e\n\n## 爬虫工具推荐\n\n1. 爬虫在线工具库：http://www.spidertools.cn\n2. 爬虫管理系统：http://feapder.com/#/feapder_platform/feaplat\n3. 验证码识别库：https://github.com/sml2h3/ddddocr\n\n## 微信赞赏\n\n如果您觉得这个项目帮助到了您，您可以帮作者买一杯咖啡表示鼓励 🍹\n\n也可和作者交个朋友，解决您在使用过程中遇到的问题\n\n\n![赞赏码](http://markdown-media.oss-cn-beijing.aliyuncs.com/2021/03/16/zan-shang-ma.png)\n\n## 学习交流\n\n\u003ctable border=\"0\"\u003e \n    \u003ctr\u003e \n     \u003ctd\u003e 知识星球：17321694 \u003c/td\u003e \n     \u003ctd\u003e 作者微信： boris_tm \u003c/td\u003e \n     \u003ctd\u003e QQ群号：521494615\u003c/td\u003e\n    \u003c/tr\u003e \n    \u003ctr\u003e \n    \u003ctd\u003e \u003cimg src=\"http://markdown-media.oss-cn-beijing.aliyuncs.com/2020/02/16/zhi-shi-xing-qiu.jpeg\" width=250px\u003e\n \u003c/td\u003e \n     \u003ctd\u003e \u003cimg src=\"http://markdown-media.oss-cn-beijing.aliyuncs.com/2021/07/12/er-wei-ma.jpeg?x-oss-process=style/markdown-media\" width=\"250px\" /\u003e \u003c/td\u003e \n     \u003ctd\u003e \u003cimg src=\"http://markdown-media.oss-cn-beijing.aliyuncs.com/2024/04/28/17142933285892.jpg\" width=\"250px\" /\u003e \u003c/td\u003e \n    \u003c/tr\u003e \n  \u003c/table\u003e \n\n\n\n  加好友备注：feapder\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboris-code%2Ffeapder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboris-code%2Ffeapder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboris-code%2Ffeapder/lists"}