{"id":16201647,"url":"https://github.com/bin-huang/nodespider","last_synced_at":"2025-09-19T05:32:39.976Z","repository":{"id":37686681,"uuid":"74873888","full_name":"Bin-Huang/NodeSpider","owner":"Bin-Huang","description":"[DEPRECATED] Simple, flexible, delightful web crawler/spider package","archived":false,"fork":false,"pushed_at":"2023-02-27T22:43:08.000Z","size":5786,"stargazers_count":38,"open_issues_count":9,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-06T22:01:45.420Z","etag":null,"topics":["async","crawl","crawler","node","pipeline","promise","spider","web"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bin-Huang.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-11-27T07:34:04.000Z","updated_at":"2024-10-10T03:47:48.000Z","dependencies_parsed_at":"2023-02-01T07:01:01.465Z","dependency_job_id":null,"html_url":"https://github.com/Bin-Huang/NodeSpider","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bin-Huang%2FNodeSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bin-Huang%2FNodeSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bin-Huang%2FNodeSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bin-Huang%2FNodeSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bin-Huang","download_url":"https://codeload.github.com/Bin-Huang/NodeSpider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233494969,"owners_count":18684729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","crawl","crawler","node","pipeline","promise","spider","web"],"created_at":"2024-10-10T09:39:22.983Z","updated_at":"2025-09-19T05:32:39.591Z","avatar_url":"https://github.com/Bin-Huang.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEPRECATED / 不再维护\n\n其实这个框架用起来还不错，只是我暂时没有更多兴趣来维护它\n\n----------------------------\n\n\nNodeSpider 是基于 Nodejs 的新一代爬虫框架。\n\n# Feature\n\n- **开箱即用**，用最少的代码开发五脏俱全的爬虫程序\n- **计划规则与任务相分离**，再复杂的爬取需求也可以轻松实现\n- **简单好用的数据管道**，保存抓取的数据是一件轻松的事情\n- **自动转码 utf8、jQ选择器**……可爱的小功能该有的都有\n- **性能优异**，你对异步并发有绝对的控制自由\n- **丰富、简约的拓展接口**，玩在手里就像灵活的积木\n- 支持现代 promise 和 async function\n\n# Install\n\nnpm:\n\n```\nnpm install nodespider --save\n```\n\nyarn:\n\n```\nyarn add nodespider\n```\n\n# Example\n\n```javascript\nconst { Spider, jqPlan, csvPipe } = require(\"nodespider\")\nconst s = new Spider()\n\n// 声明一个数据管道\ns.pipe(csvPipe({\n  name: \"data\",\n  path: \"./data.csv\",\n  items: [\"url\", \"count\"],\n}))\n\n// 声明一个爬取计划\ns.plan(jqPlan({\n  name: \"extract\",\n  toUtf8: true, // 自动转码为 utf8\n  retries: 3, // 失败自动重试\n  handle: ($, current) =\u003e {\n    const title = $(\"title\").text() // 你想要的 jq 选择器\n    console.log(title)\n    s.save(\"data\", {\n      count: $(\"body\").text().length,\n      url: current.url,\n    }) // 使用管道保存数据\n    s.addU(\"extract\", $(\"a\").urls())  // 添加新任务\n  },\n}))\n\ns.add(\"extract\", \"https://github.com/Bin-Huang/NodeSpider\") // 添加新任务\n```\n\n# Document\n\n[设置 Options](./doc/cn/option.md)\n\n[爬取计划 Plan](./doc/cn/plans.md)\n\n[数据管道 Pipe](./doc/cn/pipes.md)\n\n[方法 API](./doc/cn/method.md)\n\n[事件 Events](./doc/cn/event.md)\n\n# 与 0.9.x 版本的不同\n\n当前版本几乎是 0.9 的重构，不管是 api 还设计理念，大部分均已改变。如果你的项目依赖 0.9.x 版本，这里保留了 [0.9.3 版本文档](./doc/0.9.3/doc.md)\n\n# Contribute\n\n- 任何疑问、建议、Bug，欢迎提交 `Issuse`\n- 分享这个年轻的项目给其他开发者、社区、邮件组\n- 欢迎 `Pull Request`，尤其是：\n  - 翻译文档到其他语言\n  - 文档的修改和补充\n  - ……\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbin-huang%2Fnodespider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbin-huang%2Fnodespider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbin-huang%2Fnodespider/lists"}