Projects in Awesome Lists tagged with robots-exclusion-standard
A curated list of projects in awesome lists tagged with robots-exclusion-standard .
https://github.com/crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
hacktoberfest robots-exclusion-protocol robots-exclusion-standard robots-txt robots-txt-parser web-crawling web-scraping
Last synced: 13 May 2025
https://github.com/antoinegagne/robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
crawling erlang erlang-library parser parsing parsing-library rfc-9309 robots-exclusion-standard robots-parser robots-txt
Last synced: 12 May 2025
https://github.com/0xibra/robots-txt-component
Fully native robots.txt parsing component without any dependencies.
nodejs robots-exclusion-standard robots-node robots-parser robots-txt robots-txt-node
Last synced: 23 Feb 2025
