https://github.com/turnersoftware/infinitycrawler

A simple but powerful web crawler library for .NET
https://github.com/turnersoftware/infinitycrawler

crawler robots-txt spider web-crawler web-crawling

Last synced: 9 days ago
JSON representation

A simple but powerful web crawler library for .NET

• Host: GitHub
• URL: https://github.com/turnersoftware/infinitycrawler
• Owner: TurnerSoftware
• License: mit
• Created: 2018-12-28T09:49:57.000Z (over 6 years ago)
• Default Branch: main
• Last Pushed: 2023-12-15T05:08:18.000Z (over 1 year ago)
• Last Synced: 2025-06-05T00:54:18.848Z (25 days ago)
• Topics: crawler, robots-txt, spider, web-crawler, web-crawling
• Language: C#
• Homepage:
• Size: 326 KB
• Stars: 252
• Watchers: 10
• Forks: 37
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: License.txt

Awesome Lists containing this project