{"id":51551450,"url":"https://github.com/cliff-byte/alipay-doc-fetcher","last_synced_at":"2026-07-10T00:01:54.075Z","repository":{"id":367018681,"uuid":"1278881019","full_name":"cliff-byte/alipay-doc-fetcher","owner":"cliff-byte","description":"把 JS 渲染的支付宝开放平台文档抓取为 Agent 友好的 Markdown 的跨 Agent Skill（Claude Code / Codex / Cursor 通用）","archived":false,"fork":false,"pushed_at":"2026-06-24T10:00:54.000Z","size":46,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-24T10:11:57.915Z","etag":null,"topics":["agent-skill","alipay","claude-code","codex","cursor","documentation","llm-tools","markdown","playwright","scraper"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cliff-byte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-24T07:15:34.000Z","updated_at":"2026-06-24T10:01:14.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cliff-byte/alipay-doc-fetcher","commit_stats":null,"previous_names":["cliff-byte/alipay-doc-fetcher"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/cliff-byte/alipay-doc-fetcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliff-byte%2Falipay-doc-fetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliff-byte%2Falipay-doc-fetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliff-byte%2Falipay-doc-fetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliff-byte%2Falipay-doc-fetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cliff-byte","download_url":"https://codeload.github.com/cliff-byte/alipay-doc-fetcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cliff-byte%2Falipay-doc-fetcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35315894,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-skill","alipay","claude-code","codex","cursor","documentation","llm-tools","markdown","playwright","scraper"],"created_at":"2026-07-10T00:01:53.026Z","updated_at":"2026-07-10T00:01:54.069Z","avatar_url":"https://github.com/cliff-byte.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eALIPAY-DOC-FETCHER\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/cliff-byte/alipay-doc-fetcher/actions/workflows/ci.yml\"\u003e\u003cimg alt=\"CI\" src=\"https://github.com/cliff-byte/alipay-doc-fetcher/actions/workflows/ci.yml/badge.svg\"\u003e\u003c/a\u003e\u0026nbsp;\u003cimg alt=\"Node\" src=\"https://img.shields.io/badge/node-%3E%3D24-339933?logo=node.js\u0026logoColor=white\"\u003e\u0026nbsp;\u003cimg alt=\"Playwright\" src=\"https://img.shields.io/badge/Playwright-%5E1.40-2EAD33?logo=playwright\u0026logoColor=white\"\u003e\u0026nbsp;\u003cimg alt=\"curl\" src=\"https://img.shields.io/badge/runtime-curl-073551?logo=curl\u0026logoColor=white\"\u003e\u0026nbsp;\u003cimg alt=\"Agent Skill\" src=\"https://img.shields.io/badge/Agent%20Skill-npx%20skills-5A4FCF\"\u003e\u0026nbsp;\u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-MIT-green\"\u003e\n\u003c/p\u003e\n\n把 JS 渲染的**支付宝开放平台文档**（`opendocs.alipay.com` / `opendoc.alipay.com`）抓取为对**编程 Agent 友好**的 Markdown —— 含标准代码块、层级化接口参数、表格、本地图片。\n\n\u003e 依赖：**Node ≥ 24**、**Playwright**（`^1.40`，需 `npx playwright install chromium`）、系统 **curl**（图片下载）。\n\n## 为什么需要它\n\n支付宝文档站是 SPA：`WebFetch`/`curl` 只能拿到标题；代码块（CodeMirror）和接口参数依赖懒加载与折叠/Tab 交互，直接抓 DOM 会大面积缺失。本工具用无头浏览器渲染 + 模拟交互 + 结构化提取，把这些坑一次性封装好。\n\n\u003e 完整的踩坑经验与设计依据见 [PLAYBOOK.md](./PLAYBOOK.md)。**强烈建议改代码前先读它。**\n\n## 安装为 Skill（推荐）\n\n本仓库是符合 [skills.sh](https://skills.sh) / `npx skills` 规范的开源 Skill 仓库（`skills/fetch-alipay-doc/SKILL.md`，跨 Claude Code / Codex / Cursor 等 70+ Agent 通用）。一条命令安装：\n\n```bash\n# 安装到所有已检测到的 Agent（交互式选择）\nnpx skills add cliff-byte/alipay-doc-fetcher\n\n# 或指定 Agent\nnpx skills add cliff-byte/alipay-doc-fetcher -a claude-code -a codex -a cursor\n```\n\n安装后进入该 Skill 目录补齐运行依赖（Playwright + chromium，图片下载另需系统 `curl`）：\n\n```bash\ncd ~/.claude/skills/fetch-alipay-doc        # Codex 为 ~/.codex/skills/...，Cursor 为 ~/.cursor/skills/...\nnpm install \u0026\u0026 npx playwright install chromium\n```\n\n之后在任一支持的 Agent 里，直接说「把这篇支付宝文档抓到本地：\u003c链接\u003e」即可触发。\n\n\u003e **产物落点**：抓取结果默认落到你**当前项目**下的 `./alipay-docs/`（建议按产品分子目录），**不会**写进 skill 安装目录。全局安装时，工具会拒绝把产物写入 `~/.claude|.codex|.cursor/skills/...` 自身目录。\n\n\u003e 各 Agent 的 skills 目录：Claude Code `~/.claude/skills/`、Codex `~/.codex/skills/`、Cursor `~/.cursor/skills/`（Cursor 也兼容读取前两者）。\n\n## 提高触发率（推荐）\n\nSkill 是否被调用由 Agent 自行判断。实测发现：对「抓个 URL」这类请求，Agent 常**误以为自己能直接 WebFetch** 而不去调本 Skill（直到拿到空正文才回头）。最可靠的办法是在你项目根的 `CLAUDE.md`（Codex 用 `AGENTS.md`，Cursor 用 `.cursor/rules/`）里加一条路由规则，把下面这段复制进去：\n\n```markdown\n## Skill 路由\n\n当用户想要任何**支付宝开放平台文档**（opendocs.alipay.com / opendoc.alipay.com）的正文内容——抓取 / 下载 / 本地化 API 接口文档、接入指南、错误码、产品文档，或粘贴 opendocs/opendoc 链接、给出文档 ID（如 07kszv）或接口名（如 alipay.trade.fastpay.refund.query）——一律用 `fetch-alipay-doc` 技能，**不要直接 WebFetch/curl**：该站是 JS 渲染的 SPA，直接抓只会得到空正文。\n```\n\n## 直接当 CLI 用\n\n不走 Skill 也行，clone 后直接命令行调用：\n\n```bash\ngit clone https://github.com/cliff-byte/alipay-doc-fetcher.git\ncd alipay-doc-fetcher/skills/fetch-alipay-doc\nnpm install \u0026\u0026 npx playwright install chromium\n```\n\n\u003e 若未 `npm install`，脚本会回退到全局 `playwright`（homebrew 路径）。\n\n## 用法\n\n\u003e 以下命令均在 Skill 目录（`skills/fetch-alipay-doc/`）下执行。\n\n抓单篇：\n```bash\nnode scripts/fetch-alipay-docs.cjs --url \"https://opendocs.alipay.com/open/07kszv\" --name 01-权限集介绍 --out ./alipay-docs/消费者投诉\n```\n\n批量（推荐）：自己建一个 config 文件（如 `urls.json`），内容为 JSON 数组：\n```json\n[\n  { \"name\": \"01-权限集介绍\", \"url\": \"https://opendocs.alipay.com/open/07kszv?pathHash=af92d9c3\" },\n  { \"name\": \"退款查询\", \"url\": \"https://opendocs.alipay.com/open/c1cb8815_alipay.trade.fastpay.refund.query\" }\n]\n```\n`name` 可省略（缺省用页面 H1 生成文件名）。然后：\n```bash\nnode scripts/fetch-alipay-docs.cjs --config urls.json --out ./alipay-docs/消费者投诉\n```\n\n\u003e `--out` 缺省为当前目录下的 `./alipay-docs`。作为全局安装的 Skill 使用时，请在你的项目目录下用脚本绝对路径调用，详见上方 SKILL.md 指引。\n\n## 产出\n\n```\n\u003cout\u003e/                 # 默认 ./alipay-docs（建议按产品分子目录）\n├── \u003cname\u003e.md          # 每篇文档\n└── images/            # 文档页正文图片（\u003cname\u003e-\u003cidx\u003e.png）\n```\n\n- **文档页**（指南/说明类）：正文 + 标题分层（`##`/`###`）+ 代码块 + 表格 + 内联图片。\n- **API 接口页**：通用按 H2 分段；业务参数渲染为**层级列表**（含嵌套子属性、枚举值），公共参数/业务错误码渲染为**表格**，请求示例取 **Java** 代码块、响应取 JSON（含**正常+异常**两种示例）；**公共错误码保留为超链接**（各接口通用，不内联以免重复、省 token，需要时可顺链接自取）。\n- **超链接**：正文 / 表格 / 参数描述里的链接全部保留为 Markdown 链接，相对链接补全为绝对 URL。\n\n## 测试\n\n`render.cjs`（结构化数据 → Markdown）与 `fetch.cjs` 的产出自检是纯函数，有零依赖单测（`node:test`）：\n\n```bash\ncd skills/fetch-alipay-doc \u0026\u0026 npm test\n```\n\n## 可靠性\n\n- 抓取失败的篇目会在结尾**汇总**，图片下载失败会**显式告警**，任一失败进程**非零退出**（便于自动化/CI 感知，不再静默部分失败）。\n- 抓完做 **DOM 漂移合理性自检**：产出异常稀少（疑似支付宝改版导致选择器失效）时打印告警，而非静默吐残缺文档。\n\n## 设计原则：忠实\n\n正文文字 100% 来自页面渲染结果，不虚构、不总结、不改写、不解读。工具只做机械格式化。\n\n## 结构\n\n```\nskills/fetch-alipay-doc/          # 自包含 Skill（npx skills 自动探测并安装）\n├── SKILL.md                      # Skill 定义 + 抓完自检清单（跨 Agent 通用）\n├── package.json                  # 依赖（playwright）\n├── scripts/\n│   ├── fetch-alipay-docs.cjs     # CLI 入口（编排浏览器、下载图片、落盘）\n│   └── lib/\n│       ├── fetch.cjs             # 抓取 + 页面内结构化提取 + 产出自检\n│       ├── render.cjs            # 结构化数据 → Markdown\n│       ├── validate.cjs          # 输入校验（config / URL）\n│       ├── util.cjs              # 文件名 slug / 去重\n│       ├── paths.cjs             # 输出落点护栏\n│       └── *.test.cjs            # 各模块零依赖单测（node:test）\n\nPLAYBOOK.md                       # 维护者经验手册（仓库根，不随 Skill 安装分发）\n```\n\n## 状态\n\n早期版本（v0.3）。已在「消费者投诉」12 篇文档 + 退款查询 API 页上验证。v0.2 补完可靠性（健壮导航+重试、输入校验、文件名防撞、失败汇总+非零退出、DOM 漂移告警）；v0.3 增强能力（异常示例响应、公共错误码内联、跨平台 curl→fetch 兜底）。已知局限见 [PLAYBOOK.md](./PLAYBOOK.md) §4 与 [TODOS.md](./TODOS.md)。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcliff-byte%2Falipay-doc-fetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcliff-byte%2Falipay-doc-fetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcliff-byte%2Falipay-doc-fetcher/lists"}