https://github.com/p0dalirius/robotstester

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
https://github.com/p0dalirius/robotstester

bugbounty crawler pentesting python robots tool

Last synced: about 2 months ago
JSON representation

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.

• Host: GitHub
• URL: https://github.com/p0dalirius/robotstester
• Owner: p0dalirius
• License: gpl-3.0
• Created: 2021-07-05T12:03:38.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2023-10-01T19:15:30.000Z (about 1 year ago)
• Last Synced: 2024-06-15T17:41:24.933Z (6 months ago)
• Topics: bugbounty, crawler, pentesting, python, robots, tool
• Language: Python
• Homepage: https://podalirius.net/
• Size: 236 KB
• Stars: 39
• Watchers: 2
• Forks: 4
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
![](./.github/banner.png)



  

  

  

  




With this script, you can enumerate all URLs present in robots.txt files, and test whether you can access them or not.

![](./.github/example.gif)

## Setup

Clone the repository and install the dependencies :

```sh

git clone https://github.com/p0dalirius/robotstester

cd robotstester

python3 setup.py install

```

## Usage

```sh

robotstester -u http://www.example.com/

```

You can find here a complete list of options :

```

[~] Robots.txt tester, v1.2.0

usage: robotstester.py [-h] (-u URL | -f URLSFILE) [-v] [-q] [-k] [-L] [-t THREADS] [-p] [-j JSONFILE] [-x PROXY] [-b COOKIES]

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.

optional arguments:

  -h, --help            show this help message and exit

  -u URL, --url URL     URL to the robots.txt to test e.g. https://example.com:port/path

  -f URLSFILE, --urlsfile URLSFILE

                        List of robots.txt urls to test

  -v, --verbose         verbosity level (-v for verbose, -vv for debug)

  -q, --quiet           Show no information at all

  -k, --insecure        Allow insecure server connections when using SSL (default: False)

  -L, --location        Follow redirects (default: False)

  -t THREADS, --threads THREADS

                        Number of threads (default: 5)

  -p, --parsable        Parsable output

  -j JSONFILE, --jsonfile JSONFILE

                        Save results to specified JSON file.

  -x PROXY, --proxy PROXY

                        Specify a proxy to use for requests (e.g., http://localhost:8080)

  -b COOKIES, --cookies COOKIES

                        Specify cookies to use in requests. (e.g., --cookies "cookie1=blah;cookie2=blah")

```

## Contributing

Pull requests are welcome. Feel free to open an issue if you want to add other features.