https://github.com/rimiti/robotstxt

Robots.txt parser and generator - Work in progress
https://github.com/rimiti/robotstxt

golang-package robots-parser robots-txt

Last synced: about 5 hours ago
JSON representation

Robots.txt parser and generator - Work in progress

• Host: GitHub
• URL: https://github.com/rimiti/robotstxt
• Owner: rimiti
• Created: 2018-04-26T09:46:40.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-04-26T13:29:13.000Z (over 6 years ago)
• Last Synced: 2024-05-01T13:28:53.087Z (6 months ago)
• Topics: golang-package, robots-parser, robots-txt
• Language: Go
• Homepage:
• Size: 24.4 KB
• Stars: 3
• Watchers: 3
• Forks: 0
• Open Issues: 0

Awesome Lists containing this project