https://github.com/0xibra/robots-txt-component

Fully native robots.txt parsing component without any dependencies.

nodejs robots-exclusion-standard robots-node robots-parser robots-txt robots-txt-node

Last synced: 10 May 2026