Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/josecarneiro/mr-roboto
🤖 Handle and parse a site's robots.txt file and extract actionable information
https://github.com/josecarneiro/mr-roboto
extract parser parsing robots robots-txt
Last synced: 5 days ago
JSON representation
🤖 Handle and parse a site's robots.txt file and extract actionable information
- Host: GitHub
- URL: https://github.com/josecarneiro/mr-roboto
- Owner: josecarneiro
- Created: 2017-12-04T23:00:14.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2017-12-04T23:00:36.000Z (about 7 years ago)
- Last Synced: 2024-11-21T12:20:04.157Z (2 months ago)
- Topics: extract, parser, parsing, robots, robots-txt
- Language: JavaScript
- Homepage:
- Size: 36.1 KB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0