Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/p0dalirius/robotstester
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
https://github.com/p0dalirius/robotstester
bugbounty crawler pentesting python robots tool
Last synced: 22 days ago
JSON representation
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
- Host: GitHub
- URL: https://github.com/p0dalirius/robotstester
- Owner: p0dalirius
- License: gpl-3.0
- Created: 2021-07-05T12:03:38.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-01T19:15:30.000Z (9 months ago)
- Last Synced: 2024-05-01T17:26:43.327Z (2 months ago)
- Topics: bugbounty, crawler, pentesting, python, robots, tool
- Language: Python
- Homepage: https://podalirius.net/
- Size: 236 KB
- Stars: 39
- Watchers: 2
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Lists
- osint_stuff_tool_collection - RobotTester
README
With this script, you can enumerate all URLs present in robots.txt files, and test whether you can access them or not.
## Setup
Clone the repository and install the dependencies :
```sh
git clone https://github.com/p0dalirius/robotstester
cd robotstester
python3 setup.py install
```
## Usage
```sh
robotstester -u http://www.example.com/
```
You can find here a complete list of options :
```
[~] Robots.txt tester, v1.2.0
usage: robotstester.py [-h] (-u URL | -f URLSFILE) [-v] [-q] [-k] [-L] [-t THREADS] [-p] [-j JSONFILE] [-x PROXY] [-b COOKIES]
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
optional arguments:
-h, --help show this help message and exit
-u URL, --url URL URL to the robots.txt to test e.g. https://example.com:port/path
-f URLSFILE, --urlsfile URLSFILE
List of robots.txt urls to test
-v, --verbose verbosity level (-v for verbose, -vv for debug)
-q, --quiet Show no information at all
-k, --insecure Allow insecure server connections when using SSL (default: False)
-L, --location Follow redirects (default: False)
-t THREADS, --threads THREADS
Number of threads (default: 5)
-p, --parsable Parsable output
-j JSONFILE, --jsonfile JSONFILE
Save results to specified JSON file.
-x PROXY, --proxy PROXY
Specify a proxy to use for requests (e.g., http://localhost:8080)
-b COOKIES, --cookies COOKIES
Specify cookies to use in requests. (e.g., --cookies "cookie1=blah;cookie2=blah")
```
## Contributing
Pull requests are welcome. Feel free to open an issue if you want to add other features.