{"id":15363129,"url":"https://github.com/techquery/web-fetch","last_synced_at":"2025-06-30T10:04:41.056Z","repository":{"id":34642995,"uuid":"181468731","full_name":"TechQuery/Web-fetch","owner":"TechQuery","description":"Hexo Migrator plugin for common Web pages","archived":false,"fork":false,"pushed_at":"2024-11-25T15:23:17.000Z","size":502,"stargazers_count":5,"open_issues_count":7,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-28T01:04:35.621Z","etag":null,"topics":["hexo","migrate","plugin","web"],"latest_commit_sha":null,"homepage":"http://tech-query.me/Web-fetch/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TechQuery.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":["https://paypal.me/TechQuery","https://tech-query.me/image/TechQuery-Alipay.jpg"]}},"created_at":"2019-04-15T10:59:00.000Z","updated_at":"2024-11-25T15:25:00.000Z","dependencies_parsed_at":"2024-12-14T13:12:30.319Z","dependency_job_id":"3cbd9c37-f188-4512-8206-34b086fa11b7","html_url":"https://github.com/TechQuery/Web-fetch","commit_stats":{"total_commits":23,"total_committers":2,"mean_commits":11.5,"dds":0.3913043478260869,"last_synced_commit":"d52ada42a9f55eef8eb152a029c94d48ffdf7032"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/TechQuery/Web-fetch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TechQuery%2FWeb-fetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TechQuery%2FWeb-fetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TechQuery%2FWeb-fetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TechQuery%2FWeb-fetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TechQuery","download_url":"https://codeload.github.com/TechQuery/Web-fetch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TechQuery%2FWeb-fetch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262753011,"owners_count":23358879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hexo","migrate","plugin","web"],"created_at":"2024-10-01T13:04:49.631Z","updated_at":"2025-06-30T10:04:40.718Z","avatar_url":"https://github.com/TechQuery.png","language":"TypeScript","funding_links":["https://paypal.me/TechQuery","https://tech-query.me/image/TechQuery-Alipay.jpg"],"categories":[],"sub_categories":[],"readme":"# Web fetch\n\nAsset Crawler for common Web pages\n\n[![NPM Dependency](https://img.shields.io/librariesio/github/TechQuery/Web-fetch.svg)][1]\n[![CI \u0026 CD](https://github.com/TechQuery/Web-fetch/actions/workflows/main.yml/badge.svg)][2]\n\n[![NPM](https://nodei.co/npm/web-fetch.png?downloads=true\u0026downloadRank=true\u0026stars=true)][3]\n\n## Usage\n\n### As a Node.js package\n\n```javascript\nimport { savePage } from 'web-fetch';\n\nsavePage({\n    source: 'http://URL.to/one/of/your/old/posts/'\n});\n```\n\n#### Docker example\n\nhttps://github.com/kaiyuanshe/KYS-service\n\n### As a Command Line tool\n\n```shell\nweb-fetch http://URL.to/one/of/your/old/posts/\n```\n\n### In GitHub actions\n\n```yml\nname: Fetch Web pages\non:\n    issues:\n        types:\n            - opened\njobs:\n    Fetch-and-Save:\n        runs-on: ubuntu-latest\n        permissions:\n            contents: write\n            issues: write\n        steps:\n            - uses: actions/checkout@v4\n\n            - uses: pnpm/action-setup@v4\n              with:\n                  version: 9\n            - uses: actions/setup-node@v4\n              with:\n                  node-version: 20\n                  cache: pnpm\n            - uses: browser-actions/setup-chrome@v1\n\n            - name: Setup Web-fetch\n              run: pnpm i web-fetch -g\n\n            - name: Fetch first URL in Issue Body\n              run: web-fetch $(echo \"${{ github.event.issue.Body }}\" | grep -Eo \"https?://\\S+\")\n\n            - uses: stefanzweifel/git-auto-commit-action@v5\n              with:\n                  commit_message: '${{ github.event.issue.title }}'\n```\n\n## Supported Structure\n\nhttps://github.com/TechQuery/Web-fetch/blob/d4c574adb6ed4b3030995f4ea26911e88d00c1fc/source/config.ts#L10-L29\n\n## Renderer\n\n1. [Puppeteer](https://pptr.dev/) (default)\n\n2. [JSDOM](https://github.com/jsdom/jsdom)\n\n## Wrapper\n\n1. [Hexo plugin](https://github.com/TechQuery/Web-fetch/tree/master/wrapper/Hexo/)\n\n[1]: https://libraries.io/npm/web-fetch\n[2]: https://github.com/TechQuery/Web-fetch/actions/workflows/main.yml\n[3]: https://nodei.co/npm/web-fetch/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechquery%2Fweb-fetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechquery%2Fweb-fetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechquery%2Fweb-fetch/lists"}