{"id":13447311,"url":"https://github.com/gajus/headless-crawler","last_synced_at":"2025-04-15T21:22:29.958Z","repository":{"id":57262105,"uuid":"163210011","full_name":"gajus/headless-crawler","owner":"gajus","description":"A crawler implemented using a headless browser (Chrome).","archived":false,"fork":false,"pushed_at":"2018-12-30T11:49:56.000Z","size":105,"stargazers_count":14,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T01:23:51.049Z","etag":null,"topics":["chrome","crawler","headless","puppeteer","spider"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gajus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-26T19:13:11.000Z","updated_at":"2023-06-29T04:53:21.000Z","dependencies_parsed_at":"2022-08-25T06:11:07.051Z","dependency_job_id":null,"html_url":"https://github.com/gajus/headless-crawler","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fheadless-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fheadless-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fheadless-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fheadless-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gajus","download_url":"https://codeload.github.com/gajus/headless-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249154777,"owners_count":21221472,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome","crawler","headless","puppeteer","spider"],"created_at":"2024-07-31T05:01:13.559Z","updated_at":"2025-04-15T21:22:29.940Z","avatar_url":"https://github.com/gajus.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"\u003ca name=\"headless-crawler\"\u003e\u003c/a\u003e\n# headless-crawler 👻\n\n[![Travis build status](http://img.shields.io/travis/gajus/headless-crawler/master.svg?style=flat-square)](https://travis-ci.org/gajus/headless-crawler)\n[![Coveralls](https://img.shields.io/coveralls/gajus/headless-crawler.svg?style=flat-square)](https://coveralls.io/github/gajus/headless-crawler)\n[![NPM version](http://img.shields.io/npm/v/headless-crawler.svg?style=flat-square)](https://www.npmjs.org/package/headless-crawler)\n[![Canonical Code Style](https://img.shields.io/badge/code%20style-canonical-blue.svg?style=flat-square)](https://github.com/gajus/canonical)\n[![Twitter Follow](https://img.shields.io/twitter/follow/kuizinas.svg?style=social\u0026label=Follow)](https://twitter.com/kuizinas)\n\nA crawler implemented using a headless browser (Chrome).\n\n* [headless-crawler 👻](#headless-crawler)\n    * [Features](#headless-crawler-features)\n    * [Usage](#headless-crawler-usage)\n    * [Configuration](#headless-crawler-configuration)\n        * [Default `headlessCrawlerConfiguration.extractContent`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-extractcontent)\n        * [Default `headlessCrawlerConfiguration.filterLink`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-filterlink)\n        * [Default `headlessCrawlerConfiguration.onError`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-onerror)\n        * [Default `headlessCrawlerConfiguration.onResult`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-onresult)\n        * [Default `headlessCrawlerConfiguration.sortQueuedLinks`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-sortqueuedlinks)\n        * [Default `headlessCrawlerConfiguration.waitFor`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-waitfor)\n    * [Create default handlers](#headless-crawler-create-default-handlers)\n    * [Recipes](#headless-crawler-recipes)\n        * [Inject jQuery](#headless-crawler-recipes-inject-jquery)\n        * [Configure request parameters](#headless-crawler-recipes-configure-request-parameters)\n        * [Capture a screenshot](#headless-crawler-recipes-capture-a-screenshot)\n        * [Configure a proxy](#headless-crawler-recipes-configure-a-proxy)\n    * [Types](#headless-crawler-types)\n    * [Logging](#headless-crawler-logging)\n    * [FAQ](#headless-crawler-faq)\n        * [What makes `headless-crawler` different from `headless-chrome-crawler`?](#headless-crawler-faq-what-makes-headless-crawler-different-from-headless-chrome-crawler)\n\n\n\u003ca name=\"headless-crawler-features\"\u003e\u003c/a\u003e\n## Features\n\n* Scrapes websites using user-provided `extractContent` function and follows the observed URLs as instructed by `filterLink` and `onResult`.\n* Configurable concurrency.\n* Respects [robots.txt](https://en.wikipedia.org/wiki/Robots_exclusion_standard) (configurable) (see [Default `headlessCrawlerConfiguration.filterLink`](#headless-crawler-configuration-default-headlesscrawlerconfiguration-filterlink)).\n\n\u003ca name=\"headless-crawler-usage\"\u003e\u003c/a\u003e\n## Usage\n\n```js\nimport puppeteer from 'puppeteer';\nimport {\n  createHeadlessCrawler\n} from 'headless-crawler';\n\nconst main = async () =\u003e {\n  const browser = puppeteer.launch();\n\n  // See Configuration documentation.\n  const headlessCrawler = createHeadlessCrawler({\n    onResult: (resource) =\u003e {\n      console.log(resource.content.title);\n    },\n    browser\n  });\n\n  await headlessCrawler.queue('http://gajus.com/');\n};\n\nmain();\n\n```\n\n\u003ca name=\"headless-crawler-configuration\"\u003e\u003c/a\u003e\n## Configuration\n\n```js\n/**\n * @property browser Instance of [Puppeteer Browser](https://pptr.dev/#?product=Puppeteer\u0026version=v1.11.0\u0026show=api-class-browser).\n * @property concurrency Maximum concurrency. Defaults to 5.\n * @property extractContent A function [evaluted](https://pptr.dev/#?product=Puppeteer\u0026version=v1.11.0\u0026show=api-pageevaluatepagefunction-args) in the context of the browser. The result of the function is used to describe the contents of the website (see `ScrapeResultType#content` property).\n * @property filterLink Identifies which URLs to follow.\n * @property onPage Invoked when [Puppeteer Page](https://pptr.dev/#?product=Puppeteer\u0026version=v1.11.0\u0026show=api-class-page) instance is instantiated.\n * @property onResult Invoked after content is extracted from a new page. Must return a boolean value indicating whether the crawler should advance to the next URL.\n * @property sortQueuedLinks Sorts queued links.\n * @property waitFor Invoked before links are aggregated from the website and before `extractContent`.\n */\ntype HeadlessCrawlerUserConfigurationType = {|\n  +browser: PuppeteerBrowserType,\n  +concurrency: number,\n  +extractContent?: ExtractContentHandlerType,\n  +filterLink?: FilterLinkHandlerType,\n  +onPage?: PageHandlerType,\n  +onResult?: ResultHandlerType,\n  +sortQueuedLinks?: SortQueuedLinksHandlerType,\n  +waitFor?: WaitForHandlerType\n|};\n\n```\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-extractcontent\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.extractContent\u003c/code\u003e\n\nThe default `extractContent` function extracts page title.\n\n```js\n(): ExtractContentHandlerType =\u003e {\n  return `(() =\u003e {\n    return {\n      title: document.title\n    };\n  })();`;\n};\n\n```\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-filterlink\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.filterLink\u003c/code\u003e\n\nThe default `filterLink` function includes all URLs allowed by robots.txt and does not visit previously scraped URLs.\n\n```js\ntype DefaultFilterLinkHandlerConfigurationType = {|\n  +maxLinkDepth: number,\n  +respectRobots: boolean\n|};\n\ntype DefaultFilterLinkHandlerUserConfigurationType = {|\n  +maxLinkDepth?: number,\n  +respectRobots?: boolean\n|};\n\n(userConfiguration: DefaultFilterLinkHandlerUserConfigurationType): FilterLinkHandlerType =\u003e {\n  const configuration: DefaultFilterLinkHandlerConfigurationType = {\n    maxLinkDepth: 10,\n    respectRobots: true,\n    ...userConfiguration\n  };\n\n  let robotsAgent;\n\n  if (configuration.respectRobots) {\n    robotsAgent = createRobotsAgent();\n  }\n\n  return async (link, scrapedLinkHistory) =\u003e {\n    if (link.linkDepth \u003e configuration.maxLinkDepth) {\n      return false;\n    }\n\n    if (configuration.respectRobots \u0026\u0026 robotsAgent.isRobotsAvailable(link.linkUrl) \u0026\u0026 !robotsAgent.isAllowed(link.linkUrl)) {\n      return false;\n    }\n\n    for (const scrapedLink of scrapedLinkHistory) {\n      if (scrapedLink.linkUrl === link.linkUrl) {\n        return false;\n      }\n    }\n\n    return true;\n  };\n};\n\n```\n\nNote: robots.txt support is implemented using [`robots-agent`](https://github.com/gajus/robots-agent).\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-onerror\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.onError\u003c/code\u003e\n\n```js\n(): ErrorHandlerType =\u003e {\n  return (error) =\u003e {};\n};\n\n```\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-onresult\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.onResult\u003c/code\u003e\n\nThe default `onResult` logs the result and advances crawler to the next URL.\n\n```js\n(): ResultHandlerType =\u003e {\n  return (scrapeResult) =\u003e {\n    log.debug({\n      scrapeResult\n    }, 'new result');\n\n    return true;\n  };\n};\n\n```\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-sortqueuedlinks\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.sortQueuedLinks\u003c/code\u003e\n\n```js\n(): SortQueuedLinksHandlerType =\u003e {\n  return (links) =\u003e {\n    return links;\n  };\n};\n\n```\n\n\u003ca name=\"headless-crawler-configuration-default-headlesscrawlerconfiguration-waitfor\"\u003e\u003c/a\u003e\n### Default \u003ccode\u003eheadlessCrawlerConfiguration.waitFor\u003c/code\u003e\n\n```js\n(): WaitForHandlerType =\u003e {\n  return (page) =\u003e {\n    return page.waitForNavigation({\n      waitUntil: 'networkidle2'\n    });\n  };\n};\n\n```\n\n\u003ca name=\"headless-crawler-create-default-handlers\"\u003e\u003c/a\u003e\n## Create default handlers\n\nYou can import factory functions to create default handlers:\n\n```js\nimport {\n  createDefaultExtractContentHandler,\n  createDefaultFilterLinkHandler,\n  createDefaultResultHandler,\n  createDefaultSortQueuedLinksHandler,\n  createDefaultWaitForHandler\n} from 'headless-crawler';\n\n```\n\nThis is useful for extending the default handlers, e.g.\n\n```js\nconst defaultFilterHandler = createDefaultFilterLinkHandler();\n\nconst myCustomFilterLinkHandler = (link, scrapedLinkHistory) =\u003e {\n  if (link.linkUrl.startsWith('https://google.com/')) {\n    return false;\n  }\n\n  return defaultFilterHandler(link, scrapedLinkHistory);\n};\n\n```\n\n\u003ca name=\"headless-crawler-recipes\"\u003e\u003c/a\u003e\n## Recipes\n\n\u003ca name=\"headless-crawler-recipes-inject-jquery\"\u003e\u003c/a\u003e\n### Inject jQuery\n\nUse `extractContent` to manipulate the Puppeteer Page object after it has been determined to be ready and create the function used to extract content from the website.\n\n```js\nconst main = async () =\u003e {\n  const browser = await puppeteer.launch();\n\n  const headlessCrawler = createHeadlessCrawler({\n    browser,\n    extractContent: async (page) =\u003e {\n      await page.addScriptTag({\n        url: 'https://code.jquery.com/jquery-3.3.1.min.js'\n      });\n\n      return `(() =\u003e {\n        return $('title').text();\n      })()`;\n    }\n  });\n};\n\nmain();\n\n```\n\n\u003ca name=\"headless-crawler-recipes-configure-request-parameters\"\u003e\u003c/a\u003e\n### Configure request parameters\n\nRequest parameters (such as geolocation, user-agent and viewport) can be configured using `onPage` handler, e.g.\n\n```js\nconst main = async () =\u003e {\n  const browser = await puppeteer.launch();\n\n  const onPage = async (page, scrapeConfiguration) =\u003e {\n    await page.setGeolocation({\n      latitude: 59.95,\n      longitude: 30.31667\n    });\n    await page.setUserAgent('headless-crawler');\n  };\n\n  const headlessCrawler = createHeadlessCrawler({\n    browser,\n    onPage\n  });\n};\n\nmain();\n\n```\n\n\u003ca name=\"headless-crawler-recipes-capture-a-screenshot\"\u003e\u003c/a\u003e\n### Capture a screenshot\n\nThe `extractContent` method can capture the screenshot of the website as it was at the time just before the content-extraction function is executed, e.g.\n\n```js\nconst extractContent = async (page) =\u003e {\n  await page.screenshot({\n    fullPage: true,\n    path: 'screenshot.png'\n  });\n\n  return `(() =\u003e {\n    return {\n      title: document.title\n    };\n  })()`;\n};\n\n```\n\nRefer to Puppeteer [`page#screenshot`](https://pptr.dev/#?product=Puppeteer\u0026version=v1.11.0\u0026show=api-pagescreenshotoptions) documentation for other properties.\n\n\u003ca name=\"headless-crawler-recipes-configure-a-proxy\"\u003e\u003c/a\u003e\n### Configure a proxy\n\nNote: These instructions are not specific `headless-crawler`; these are generic instructions for instructing Puppeteer to use HTTP proxy.\n\nYou must:\n\n1. Configure `ignoreHTTPSErrors`\n2. Configure `--proxy-server`\n\nExample:\n\n```js\nimport puppeteer from 'puppeteer';\nimport {\n  createHeadlessCrawler\n} from 'headless-crawler';\n\nconst main = async () =\u003e {\n  const browser = puppeteer.launch({\n    args: [\n      '--proxy-server=http://127.0.0.1:8080'\n    ],\n    ignoreHTTPSErrors: true\n  });\n\n  const headlessCrawler = createHeadlessCrawler({\n    onResult: (resource) =\u003e {\n      console.log(resource.content.title);\n    },\n    browser\n  });\n\n  await headlessCrawler.queue('http://gajus.com/');\n};\n\nmain();\n\n```\n\n\u003ca name=\"headless-crawler-types\"\u003e\u003c/a\u003e\n## Types\n\nThis package is using [Flow](https://flow.org/) type annotations.\n\nRefer to [`./src/types.js`](./src/types.js) for method parameter and result types.\n\n\u003ca name=\"headless-crawler-logging\"\u003e\u003c/a\u003e\n## Logging\n\nThis package is using [`roarr`](https://www.npmjs.com/package/roarr) logger to log the program's state.\n\nExport `ROARR_LOG=true` environment variable to enable log printing to stdout.\n\nUse [`roarr-cli`](https://github.com/gajus/roarr-cli) program to pretty-print the logs.\n\n\u003ca name=\"headless-crawler-faq\"\u003e\u003c/a\u003e\n## FAQ\n\n\u003ca name=\"headless-crawler-faq-what-makes-headless-crawler-different-from-headless-chrome-crawler\"\u003e\u003c/a\u003e\n### What makes \u003ccode\u003eheadless-crawler\u003c/code\u003e different from \u003ccode\u003eheadless-chrome-crawler\u003c/code\u003e?\n\n[`headless-chrome-crawler`](https://github.com/yujiosaka/headless-chrome-crawler) is the only other headless crawler in the Node.js ecosystem.\n\nIt appears that `headless-chrome-crawler` is no longer maintained. At the time of this writing, the author of `headless-chrome-crawler` has not made public contributions in over 6 months and the package includes bugs as a result of [hardcoded dependency versions](https://github.com/yujiosaka/headless-chrome-crawler/blob/ad95c2c4b356c8fdc60d16f8b013cc9a043a9bc6/package.json#L28-L34).\n\nMaintenance issues aside, the `headless-chrome-crawler` is a feature-rich and configuration-driven framework. Meanwhile, `headless-crawler` provides a bare-bones framework for navigating the website and extracting the content. The consumer of the framework can extend the functionality using the provided handlers and directly consuming the Puppeteer API (e.g. see [Capture a screenshot](#headless-crawler-recipes-capture-a-screenshots)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgajus%2Fheadless-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgajus%2Fheadless-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgajus%2Fheadless-crawler/lists"}