https://github.com/brendonboshell/supercrawler

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
https://github.com/brendonboshell/supercrawler

crawler distributed-crawler robot sitemap web-crawler

Last synced: 3 months ago
JSON representation

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

• Host: GitHub
• URL: https://github.com/brendonboshell/supercrawler
• Owner: brendonboshell
• License: apache-2.0
• Created: 2016-07-17T21:08:09.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2022-12-30T18:25:30.000Z (about 2 years ago)
• Last Synced: 2024-09-30T23:18:37.146Z (4 months ago)
• Topics: crawler, distributed-crawler, robot, sitemap, web-crawler
• Language: JavaScript
• Homepage:
• Size: 664 KB
• Stars: 374
• Watchers: 11
• Forks: 61
• Open Issues: 23
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-nodejs - supercrawler - Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.  (Repository / Crawler)