https://github.com/turnersoftware/robotsexclusiontools

A "robots.txt" parsing and querying library for .NET
https://github.com/turnersoftware/robotsexclusiontools

norobots-rfc parse parser robots-txt user-agent

Last synced: 11 months ago
JSON representation

A "robots.txt" parsing and querying library for .NET

• Host: GitHub
• URL: https://github.com/turnersoftware/robotsexclusiontools
• Owner: TurnerSoftware
• License: mit
• Created: 2018-12-22T13:36:14.000Z (over 7 years ago)
• Default Branch: main
• Last Pushed: 2024-09-03T21:38:23.000Z (over 1 year ago)
• Last Synced: 2025-06-18T02:05:19.577Z (12 months ago)
• Topics: norobots-rfc, parse, parser, robots-txt, user-agent
• Language: C#
• Homepage:
• Size: 226 KB
• Stars: 29
• Watchers: 2
• Forks: 9
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: License.txt

Awesome Lists containing this project