https://github.com/josecarneiro/mr-roboto
🤖 Handle and parse a site's robots.txt file and extract actionable information
https://github.com/josecarneiro/mr-roboto
extract parser parsing robots robots-txt
Last synced: about 1 year ago
JSON representation
🤖 Handle and parse a site's robots.txt file and extract actionable information
- Host: GitHub
- URL: https://github.com/josecarneiro/mr-roboto
- Owner: josecarneiro
- Created: 2017-12-04T23:00:14.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-12-04T23:00:36.000Z (over 8 years ago)
- Last Synced: 2025-01-22T05:29:50.202Z (over 1 year ago)
- Topics: extract, parser, parsing, robots, robots-txt
- Language: JavaScript
- Homepage:
- Size: 36.1 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Mr-Roboto
Mr-Roboto is aimed at handling and parsing a website's `robots.txt` file, providing a easy interface to validate if a specific path is allowed or not.
Mr-Roboto is NOT production ready, and as such should not be used in any mission critical codebase.