https://github.com/p0dalirius/robotsvalidator

A python script to check if URLs are allowed or disallowed by a robots.txt file.

allow bugbounty bypass check disallow robots-txt web

Last synced: 30 Dec 2024