{"id":16402634,"url":"https://github.com/saltyshiomix/nest-crawler","last_synced_at":"2025-03-16T16:31:56.338Z","repository":{"id":38427792,"uuid":"203879724","full_name":"saltyshiomix/nest-crawler","owner":"saltyshiomix","description":"An easiest crawling and scraping module for NestJS","archived":false,"fork":false,"pushed_at":"2023-01-04T07:56:32.000Z","size":510,"stargazers_count":66,"open_issues_count":12,"forks_count":9,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-15T20:47:49.002Z","etag":null,"topics":["crawler","nestjs","nodejs","scraper","typescript"],"latest_commit_sha":null,"homepage":"https://npm.im/nest-crawler","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saltyshiomix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-22T22:01:31.000Z","updated_at":"2025-03-12T18:30:51.000Z","dependencies_parsed_at":"2023-02-02T04:16:52.744Z","dependency_job_id":null,"html_url":"https://github.com/saltyshiomix/nest-crawler","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltyshiomix%2Fnest-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltyshiomix%2Fnest-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltyshiomix%2Fnest-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saltyshiomix%2Fnest-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saltyshiomix","download_url":"https://codeload.github.com/saltyshiomix/nest-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243790950,"owners_count":20348379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","nestjs","nodejs","scraper","typescript"],"created_at":"2024-10-11T05:46:48.074Z","updated_at":"2025-03-16T16:31:56.031Z","avatar_url":"https://github.com/saltyshiomix.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://nestjs.com/img/logo_text.svg\" width=\"320\" alt=\"Nest Logo\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e😎 nest-crawler 😎\u003c/p\u003e\n\u003cp align=\"center\"\u003eCrawler and Scraper Module for NestJS\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://npm.im/nest-crawler\" alt=\"A version of nest-crawler\"\u003e\n    \u003cimg src=\"https://img.shields.io/npm/v/nest-crawler.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://npm.im/nest-crawler\" alt=\"Downloads of nest-crawler\"\u003e\n    \u003cimg src=\"https://img.shields.io/npm/dt/nest-crawler.svg\"\u003e\n  \u003c/a\u003e\n  \u003cimg src=\"https://img.shields.io/npm/l/nest-crawler.svg\" alt=\"Package License (MIT)\"\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\n$ npm install --save nest-crawler\n```\n\n## Usage\n\nFirst, register it in the application module so that Nest can handle dependencies:\n\n```ts\nimport { Module } from '@nestjs/common';\nimport { NestCrawlerModule } from 'nest-crawler';\n\n@Module({\n  imports: [\n    NestCrawlerModule,\n  ],\n})\nexport class AppModule {}\n```\n\nThen, just import it and use it:\n\n**crawler.module.ts**\n\n```ts\nimport { Module } from '@nestjs/common';\nimport { NestCrawlerModule } from 'nest-crawler';\n@Module({\n  imports: [\n    NestCrawlerModule,\n  ],\n})\nexport class CrawlerModule {}\n```\n\n**crawler.service.ts**\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  // scraping the specific page\n  public async scrape(): Promise\u003cvoid\u003e {\n    interface ExampleCom {\n      title: string;\n      info: string;\n      content: string;\n    }\n\n    const data: ExampleCom = await this.crawler.fetch({\n      target: 'http://example.com',\n      fetch: {\n        title: 'h1',\n        info: {\n          selector: 'p \u003e a',\n          attr: 'href',\n        },\n        content: {\n          selector: '.content',\n          how: 'html',\n        },\n      },\n    });\n\n    console.log(data);\n    // {\n    //   title: 'Example Domain',\n    //   info: 'http://www.iana.org/domains/example',\n    //   content: '\u003cdiv\u003e\u003ch1\u003eExample Heading\u003c/h1\u003e\u003cp\u003eExample Paragraph\u003c/p\u003e\u003c/div\u003e'\n    // }\n  }\n\n  // crawling multi pages is also supported\n  public async crawl(): Promise\u003cvoid\u003e {\n    interface HackerNewsPage {\n      title: string;\n    }\n\n    const pages: HackerNewsPage[] = await this.crawler.fetch({\n      target: {\n        url: 'https://news.ycombinator.com',\n        iterator: {\n          selector: 'span.age \u003e a',\n          convert: (x: string) =\u003e `https://news.ycombinator.com/${x}`,\n        },\n      },\n      fetch: (data: any, index: number, url: string) =\u003e ({\n        title: '.title \u003e a',\n      }),\n    });\n\n    console.log(pages);\n    // [\n    //   { title: 'Post Title 1' },\n    //   { title: 'Post Title 2' },\n    //   ...\n    //   ...\n    //   { title: 'Post Title 30' }\n    // ]\n  }\n}\n```\n\n## Recipe\n\n### Single Page Scraping\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  public async scrape(): Promise\u003cvoid\u003e {\n    interface ExampleCom {\n      title: string;\n      info: string;\n      content: string;\n    }\n\n    const data: ExampleCom = await this.crawler.fetch({\n      target: 'http://example.com',\n      fetch: {\n        title: 'h1',\n        info: {\n          selector: 'p \u003e a',\n          attr: 'href',\n        },\n        content: {\n          selector: '.content',\n          how: 'html',\n        }\n      },\n    });\n\n    console.log(data);\n    // {\n    //   title: 'Example Domain',\n    //   info: 'http://www.iana.org/domains/example',\n    //   content: '\u003cdiv\u003e\u003ch1\u003eExample Heading\u003c/h1\u003e\u003cp\u003eExample Paragraph\u003c/p\u003e\u003c/div\u003e'\n    // }\n  }\n}\n```\n\n### Multi Pages Crawling\n\n#### You Know the target urls already\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  public async crawl(): Promise\u003cvoid\u003e {\n    interface Site {\n      title: string;\n    }\n\n    const sites: Site[] = await this.crawler.fetch({\n      target: [\n        'https://example1.com',\n        'https://example2.com',\n        'https://example3.com',\n      ],\n      fetch: (data: any, index: number, url: string) =\u003e ({\n        title: 'h1',\n      }),\n    });\n\n    console.log(sites);\n    // [\n    //   { title: 'An easiest crawling and scraping module for NestJS' },\n    //   { title: 'A minimalistic boilerplate on top of Webpack, Babel, TypeScript and React' },\n    //   { title: '[Experimental] React SSR as a view template engine' }\n    // ]\n  }\n}\n```\n\n#### You Don't Know the Target Urls so Want to Crawl Dynamically\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  public async crawl(): Promise\u003cvoid\u003e {\n    interface Page {\n      title: string;\n    }\n\n    const pages: Page[] = await this.crawler.fetch({\n      target: {\n        url: 'https://news.ycombinator.com',\n        iterator: {\n          selector: 'span.age \u003e a',\n          convert: (x: string) =\u003e `https://news.ycombinator.com/${x}`,\n        },\n      },\n      // fetch each `https://news.ycombinator.com/${x}` and scrape data\n      fetch: (data: any, index: number, url: string) =\u003e ({\n        title: '.title \u003e a',\n      }),\n    });\n\n    console.log(pages);\n    // [\n    //   { title: 'Post Title 1' },\n    //   { title: 'Post Title 2' },\n    //   ...\n    //   ...\n    //   { title: 'Post Title 30' }\n    // ]\n  }\n}\n```\n\n#### You Need to Pass Data Dynamically\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  public async crawl(): Promise\u003cvoid\u003e {\n    interface Img {\n      src: string;\n    }\n\n    const images: Img[] = await this.crawler.fetch({\n      target: {\n        url: 'https://some.image.com',\n        iterator: {\n          selector: 'span.age \u003e a',\n          convert: (x: string) =\u003e `https://some.image.com${x}`,\n        },\n        fetch: {\n          imageIds: {\n            listItem: 'div.image',\n            data: {\n              id: {\n                selector: 'div.image-wrapper',\n                attr: 'data-image-id',\n              },\n            },\n          },\n        },\n      },\n      // fetch each `https://some.image.com${x}`, pass data and scrape data\n      fetch: (data: any, index: number, url: string) =\u003e ({\n        src: {\n          convert: () =\u003e `https://some.image.com/images/${data.imageIds[index]}.png`,\n        },\n      }),\n    });\n\n    console.log(images);\n    // [\n    //   { src: 'https://some.image.com/images/1.png' },\n    //   { src: 'https://some.image.com/images/2.png' },\n    //   ...\n    //   ...\n    //   { src: 'https://some.image.com/images/100.png' }\n    // ]\n  }\n}\n```\n\n#### Waitable (by using `puppeteer`)\n\n```ts\nimport { Injectable } from '@nestjs/common';\nimport { NestCrawlerService } from 'nest-crawler';\n\n@Injectable()\nexport class CrawlerService {\n  constructor(\n    private readonly crawler: NestCrawlerService,\n  ) {}\n\n  public async crawl(): Promise\u003cvoid\u003e {\n    interface Page {\n      title: string;\n    }\n\n    const pages: Page[] = await this.crawler.fetch({\n      target: {\n        url: 'https://news.ycombinator.com',\n        iterator: {\n          selector: 'span.age \u003e a',\n          convert: (x: string) =\u003e `https://news.ycombinator.com/${x}`,\n        },\n      },\n      waitFor: 3 * 1000, // wait for the content loaded! (like single page apps)\n      fetch: (data: any, index: number, url: string) =\u003e ({\n        title: '.title \u003e a',\n      }),\n    });\n\n    console.log(pages);\n    // [\n    //   { title: 'Post Title 1' },\n    //   { title: 'Post Title 2' },\n    //   ...\n    //   ...\n    //   { title: 'Post Title 30' }\n    // ]\n  }\n}\n```\n\n## Related\n\n- [@web-master/node-web-fetch](https://github.com/saltyshiomix/web-master/blob/master/packages/node-web-fetch)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaltyshiomix%2Fnest-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaltyshiomix%2Fnest-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaltyshiomix%2Fnest-crawler/lists"}