https://github.com/antoinegagne/robots

A parser for robots.txt with support for wildcards. See also RFC 9309.

crawling erlang erlang-library parser parsing parsing-library rfc-9309 robots-exclusion-standard robots-parser robots-txt

Last synced: 12 May 2025