https://github.com/3n3a-archive/robotstxt-parser

A Robots.Txt Parser available as Golang Module
https://github.com/3n3a-archive/robotstxt-parser

Last synced: 2 months ago
JSON representation

A Robots.Txt Parser available as Golang Module

• Host: GitHub
• URL: https://github.com/3n3a-archive/robotstxt-parser
• Owner: 3n3a-archive
• License: mit
• Archived: true
• Created: 2023-02-06T18:29:51.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2023-02-27T16:46:02.000Z (almost 2 years ago)
• Last Synced: 2024-11-20T05:17:42.877Z (2 months ago)
• Language: Go
• Size: 11.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# robotstxt-parser

A Robots.Txt Parser available as Golang Module

## Features

- [x] Different parsing of Robots.txt (from Ua to UA and Sitemap separate)