{"id":21193544,"url":"https://github.com/moqsien/scrapx","last_synced_at":"2025-07-10T03:32:40.378Z","repository":{"id":57464669,"uuid":"338843317","full_name":"moqsien/scrapx","owner":"moqsien","description":"scrapy定制版; A customized and enhanced version of scrapy for managing hundreds or even thousands of spiders.","archived":false,"fork":false,"pushed_at":"2021-02-22T15:07:47.000Z","size":161,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-13T06:52:32.362Z","etag":null,"topics":["crawler","framework","pymongo","scrapy","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moqsien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-14T16:04:09.000Z","updated_at":"2022-11-17T06:42:30.000Z","dependencies_parsed_at":"2022-08-31T02:00:12.319Z","dependency_job_id":null,"html_url":"https://github.com/moqsien/scrapx","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fscrapx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fscrapx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fscrapx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moqsien%2Fscrapx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moqsien","download_url":"https://codeload.github.com/moqsien/scrapx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225615164,"owners_count":17496946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","framework","pymongo","scrapy","spider"],"created_at":"2024-11-20T19:14:42.585Z","updated_at":"2024-11-20T19:14:43.239Z","avatar_url":"https://github.com/moqsien.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[README_EN.md](https://github.com/moqsien/scrapx/tree/main/docs/README_EN.md)\n### 什么是 scrapx？\n\nscrapx 是一个基于 scrapy 的定制包。其主要特点有：\n\n- 优化了 scrapy 的 project 目录结构\n  - 采用三层目录，分别是 workspace、project、spiders。其中 workspace 作为整个爬虫项目的目录，其下自动生成一个 scrapx_globals 目录用于存放整个爬虫项目都能自动加载的配置、中间件、pipeline 等。project 必须在 workspace 模块中。具体的爬虫则放在 project 下的 spiders 模块中。\n- 去除了一些相对冗余的文件\n  - 例如，在 project 中不再有 settings.py、pipeline.py、middleware.py 等文件。而是将 pipeline 和 middleware 统一放在 scrapx_globals，便于大项目的统一管理。原来的 settings.py 变为 run_xxx.py，同时自动生成的 run_xxx.py 更便于在 IDE 中以脚本形式运行，方便调试。item.py与spider放在一起，方便开发时随时修改。\n  - 如果需要集成错误日志收集服务，则可把相应中间件放在scpax_globals中\n  - 代理中间件也可以放在scrpax_globals中，在相应的run_xxx.py中可以根据需要随时调用\n  - 只要中间件设计得足够通用，可以减少很多冗余代码\n- 大量爬虫管理方案\n  - 使用 MongoDB 作为存储 DB。在使用模板生成爬虫时，自动生成爬虫相关信息，修改之后，在爬虫第一次运行时，自动将爬虫信息存入 MongoDB，并自动生成一个唯一 id 用于标识该爬虫。每一次爬虫运行完毕，自动记录该次运行的统计数据到 MongoDB。一个 project，设计为同一类 data_type，用于归类不同类型的数据。\n- 定制脚手架命令\n  - 目前主要支持 initiate、genproject、genspider、crawl 四个命令，分别对应于创建 workspace、创建 project、创建 spider、命令行运行 spider。这些脚手架命令实现时使用 argparse 替换了已经废弃的 optparse。\n- 完全兼容 scrapy 原有的配置参数\n- 完全兼容 scrapy 原有的中间件、pipeline 接口\n- 不影响原生 scrapy 的使用\n\n### 如何使用 scrapx？\n\n- clone 代码\n\n```bash\n# install form source\ngit clone git@github.com:moqsien/scrapx.git\ncd scrapx\npython setup.py build\npython setup.py install\n```\n\n- 安装\n\n```bash\n# from pypi\npip install scrapx\n```\n\n- 创建 workspace\n\n```bash\nscrapx -h # 查看命令\nscrapx initiate example\n```\n\n- 创建 project\n\n```bash\ncd example\nscrapx genproject test1\n```\n\n- 创建 spider\n\n```bash\ncd test1\nscrapx genspider $spider_name $domain\n```\n\n- 安装本地测试mongodb(docker)\n```bash\ndocker pull mongo:latest\n\ndocker run -itd --name mongo -p 27017:27017 -v ~/mongodb:/data/db mongo --auth\n\ndocker exec -it mongo mongo admin\n\n# authorization configure\ndb.createUser({ user:'admin',pwd:'654321',roles:[ { role:'userAdminAnyDatabase', db: 'admin'}]});\ndb.auth(\"admin\", \"654321\")\ndb.grantRolesToUser(\"admin\", [ { role: \"readWrite\", db: \"crawler\" } ])\ndb.grantRolesToUser(\"admin\", [ { role: \"readWrite\", db: \"crawler_statistic\" } ])\n```\n\n- 接着就是根据用户自己的情况修改 scrapx_globals 的一些公共配置（如 MongoDB 参数等）、修改 run_xxx.py 中的 CRAWLER_INFO 等配置、编写爬虫，最后调试和运行。\n\n- 命令行运行 spider\n\n```bash\nscrapx crawl tzrb # in a project\n# or\nscrapx crawl test1.tzrb # in a workspace\n# or\n# you can just run the run_xxx.py in an IDE\n```\n\n- mongodb效果\n![avatar](https://github.com/moqsien/scrapx/blob/main/docs/mongodb.png)\n\n### scrapx 的目标\n\n减少大型爬虫项目的冗余代码，方便管理和维护。\n\n### TODO\n\n- [ ] 类似 scrapyd 的部署工具开发\n- [ ] 代理池集成\n- [ ] 错误日志监控方案\n- [ ] 部署和监控的可视化方案\n\n### 声明\n\n本项目仅做学习研究用，用户有因使用本项目造成的任何法律后果，与本项目无关。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoqsien%2Fscrapx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoqsien%2Fscrapx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoqsien%2Fscrapx/lists"}