{"id":20848662,"url":"https://github.com/derekwin/spider_api","last_synced_at":"2026-06-08T16:05:03.771Z","repository":{"id":112312519,"uuid":"581204259","full_name":"derekwin/spider_api","owner":"derekwin","description":"这是一个兼备爬虫管理功能和内容管理的restful api后台, django based","archived":false,"fork":false,"pushed_at":"2024-01-04T12:09:58.000Z","size":703,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-12T12:14:14.621Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derekwin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-12-22T14:49:34.000Z","updated_at":"2024-01-04T12:10:12.000Z","dependencies_parsed_at":"2023-04-25T20:16:04.316Z","dependency_job_id":null,"html_url":"https://github.com/derekwin/spider_api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/derekwin/spider_api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2Fspider_api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2Fspider_api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2Fspider_api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2Fspider_api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derekwin","download_url":"https://codeload.github.com/derekwin/spider_api/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2Fspider_api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34069504,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T02:27:02.156Z","updated_at":"2026-06-08T16:05:03.741Z","avatar_url":"https://github.com/derekwin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003e\n    Spider-api\n  \u003c/h1\u003e\n  \u003ch3 align=\"center\"\u003e\n     A web spider manager and base api app backend, django Ninja based\n  \u003c/h3\u003e\n\n\u003c/div\u003e\n\n\u003cbr\u003e\n\u003cbr\u003e\n\n\u003e 注意：这只是一个简单的起步项目，目的为了开发一个任务管理+内容管理后台和api的本地工具站，如果你的需求和这个项目不谋而合，可以把它当成是一个起步模板，在这个基础上进行开发。\n\n## 介绍\n\n基于django，Ninja开发的爬虫管理应用，附带提供restful api的基础后端服务。\n\n- /findphd  :  restful api 服务应用，(Ninja开发)\n    - / 提供了用户注册，邮箱验证，分页等api，开箱即用。\n- /spider   :  任务管理模块\n    - /spiders ： 存放要运行的所有任务\n        - 项目随带了 示例文件，三个职位网站爬虫，一个调用google翻译api的翻译任务\n        - 三个爬虫，一个是bs4的基础爬虫，两个是selenium的爬虫\n- /ml       :  emmm，我搞忘了这个是啥app了，反正还没写，正好扩展你的需求。\n\n\n## 演示\n\nadmin/\napi-v1/\nspider/\n\n任务管理界面\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./pics/spider.png\"\u003e\n\u003c/div\u003e\n\nNinja提供的swager api页面\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./pics/api.png\"\u003e\n\u003c/div\u003e\n\ndjango自带的后台\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./pics/admin.png\"\u003e\n\u003c/div\u003e\n\n\n\n## 使用\n\n1. 克隆到本地：\n\n` git clone https://github.com/derekwin/spider_api.git `\n\n2. 建议你创建虚拟环境\n- venv\n- miniconda\n\n此处不赘述，自行选择\n\n安装依赖包：\n` pip install -r requirements.txt `\n\n3. 自带了sqlite3的一个数据库文件，有些基础数据\n\n你也可以修改数据到你本地的数据库，参考django的官方文档\n\n\n创建用户\n` python manager.py createsuperuser 【username】 `\n\n4. 启动项目\n\n` python manager.py runserver `\n\n配置注意：\n\n- 如果需要，自行修改数据库配置\n\n- 如果要使用邮箱验证，需要你在backend/setting.py最后配置你的服务邮箱\n\n---\n\n\n## 任务模块开发\n\n按照job example的示例 在main函数实现模块的功能\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./pics/jobexam.png\"\u003e\n\u003c/div\u003e\n\n具体怎么写参考其他四个任务\n\n\n## 附：\n- google翻译的那个库对应的api可能在中国大陆地区失效了\n- 虽然一个很小的项目，但是可以很方便的扩展成一个内容管理后台，并且作为你前端的restful api服务。\n- 本项目随附的两种爬虫是比较常见的python爬虫写法，无头浏览器爬虫基本所有网站都可以爬。\n- 本项目没有提供任何代理或者反爬相关内容。\n- 适合自己折腾定时任务，树莓派上来做一个内容维护的工具，不限于爬虫，hhh。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderekwin%2Fspider_api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderekwin%2Fspider_api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderekwin%2Fspider_api/lists"}