{"id":22181166,"url":"https://github.com/viclafouch/fetch-crawler","last_synced_at":"2026-03-17T18:02:50.955Z","repository":{"id":40747943,"uuid":"185611190","full_name":"viclafouch/Fetch-Crawler","owner":"viclafouch","description":"📌 A Node.JS Web crawler using the API Fetch to scrap static websites","archived":false,"fork":false,"pushed_at":"2023-01-06T02:05:15.000Z","size":707,"stargazers_count":11,"open_issues_count":14,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-26T21:42:43.512Z","etag":null,"topics":["cheerio","crawler","crawling-sites","fetch-api","nodejs","promises","scrapping"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@viclafouch/fetch-crawler","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viclafouch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-08T13:26:43.000Z","updated_at":"2024-06-21T18:15:47.000Z","dependencies_parsed_at":"2023-02-05T02:32:06.800Z","dependency_job_id":null,"html_url":"https://github.com/viclafouch/Fetch-Crawler","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/viclafouch/Fetch-Crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viclafouch%2FFetch-Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viclafouch%2FFetch-Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viclafouch%2FFetch-Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viclafouch%2FFetch-Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viclafouch","download_url":"https://codeload.github.com/viclafouch/Fetch-Crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viclafouch%2FFetch-Crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268291648,"owners_count":24226924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cheerio","crawler","crawling-sites","fetch-api","nodejs","promises","scrapping"],"created_at":"2024-12-02T09:26:49.612Z","updated_at":"2026-03-17T18:02:45.920Z","avatar_url":"https://github.com/viclafouch.png","language":"JavaScript","readme":"# Fetch Crawler [![Build Status](https://travis-ci.com/viclafouch/Fetch-Crawler.svg?branch=master)](https://travis-ci.com/viclafouch/Fetch-Crawler)\n\n###### [API](https://github.com/viclafouch/Fetch-Crawler/blob/master/docs/API.md) | [Examples](https://github.com/viclafouch/Fetch-Crawler/tree/master/examples) | [Changelog](https://github.com/viclafouch/Fetch-Crawler/blob/master/docs/CHANGELOG.md)\n\nDistributed crawler powered by Fetch Crawler in Node JS.\n\n## Features\n\nFetch Crawler is designed to provide a basic, flexible and robust API for crawling websites.\n\nThe crawler provides [simple APIs](#api-reference) to crawl these static websites with the following features:\n\n* Distributed crawling\n* Configure parallel, retry, max requests, time between requests (to avoid being blocked by the website) ...\n* Support both [depth-first search](https://en.wikipedia.org/wiki/Depth-first_search) and [breadth-first search](https://en.wikipedia.org/wiki/Breadth-first_search) algorithm\n* Stop after a maximum amount of of requests have been executed\n* Insert [Cheerio](https://cheerio.js.org/) automatically for scraping\n* [__Promise]__ support\n\nYou can find the documentation [here](https://github.com/viclafouch/Fetch-Crawler/blob/master/docs/API.md).\n\n## Getting Started\n\n### Installation\n\n```\n# npm i @viclafouch/fetch-crawler\n```\n\n\u003e **Note**: Fetch Crawler does not contain [Puppeteer](https://github.com/GoogleChrome/puppeteer). It using the [fetch-node](https://www.npmjs.com/package/node-fetch) module.\n\n### Usage\n\n```js\nconst FetchCrawler = require('@viclafouch/fetch-crawler')\n\n// I use Cheerio to get the content of the page\n// See https://cheerio.js.org\nconst collectContent = $ =\u003e\n  $('body')\n    .find('h1')\n    .text()\n    .trim()\n\n// After getting content of the page, do what you want :)\n// Accept async function\nconst doSomethingWith = (content, url) =\u003e console.log(`Here the title '${content}' from ${url}`)\n\n// Here I start my crawler\n// You can await for it if you want\nFetchCrawler.launch({\n  url: 'https://github.com',\n  evaluatePage: $ =\u003e collectContent($),\n  onSuccess: ({ result, url }) =\u003e doSomethingWith(result, url),\n  onError: ({ error, url }) =\u003e console.log('Whouaa something wrong happened :('),\n  maxRequest: 20\n})\n```\n\n\n## FAQ\n\n### How is this different from Puppeteer?\n\nThis crawler is built on top of [node-fetch](https://www.npmjs.com/package/node-fetch).\n\n[Puppeteer](https://github.com/GoogleChrome/puppeteer) is a project from the Google Chrome team which enables us to control a Chrome (or any other Chrome DevTools Protocol based browser) and execute common actions, much like in a real browser. Fetch Crawler is a static crawler based on simple requests to HTML files. So it will be difficult to scrap the contents when the HTML dynamically changes on browsers.\n\n## Node Support Policy\n\nFetch Crawler is currently supported by the Node Foundation.\n\nCurrently supported versions:\n\n- ~~8.x~~\n- 10.x\n- 12.x\n\n## Contributing\n\nAny contributions and/or pull requests would be welcome.\n\n## License\n\nCopyright (c) 2019, Victor de la Fouchardiere.\n\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification,\nare permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n  list of conditions and the following disclaimer.\n* Redistributions in binary form must reproduce the above copyright notice, this\n  list of conditions and the following disclaimer in the documentation and/or\n  other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviclafouch%2Ffetch-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviclafouch%2Ffetch-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviclafouch%2Ffetch-crawler/lists"}