https://github.com/muratgozel/robotstxt-util

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

rfc-5234 robots-builder robots-exclusion-protocol robots-generator robots-parser robots-txt

Last synced: 29 Jul 2025