{"id":17994943,"url":"https://github.com/p0dalirius/robotstester","last_synced_at":"2025-08-21T17:06:48.450Z","repository":{"id":41381804,"uuid":"383126814","full_name":"p0dalirius/robotstester","owner":"p0dalirius","description":"This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.","archived":false,"fork":false,"pushed_at":"2023-10-01T19:15:30.000Z","size":242,"stargazers_count":41,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-18T18:45:22.452Z","etag":null,"topics":["bugbounty","crawler","pentesting","python","robots","tool"],"latest_commit_sha":null,"homepage":"https://podalirius.net/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/p0dalirius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"github":"p0dalirius","patreon":"Podalirius"}},"created_at":"2021-07-05T12:03:38.000Z","updated_at":"2024-10-07T06:42:39.000Z","dependencies_parsed_at":"2024-01-14T10:17:24.881Z","dependency_job_id":"2aa12367-a265-4f03-9763-624382e67b75","html_url":"https://github.com/p0dalirius/robotstester","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Frobotstester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Frobotstester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Frobotstester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Frobotstester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/p0dalirius","download_url":"https://codeload.github.com/p0dalirius/robotstester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231907802,"owners_count":18444185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","crawler","pentesting","python","robots","tool"],"created_at":"2024-10-29T20:17:15.747Z","updated_at":"2024-12-30T19:43:53.732Z","avatar_url":"https://github.com/p0dalirius.png","language":"Python","readme":"![](./.github/banner.png)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"GitHub release (latest by date)\" src=\"https://img.shields.io/github/v/release/p0dalirius/robotstester\"\u003e\n  \u003ca href=\"https://twitter.com/intent/follow?screen_name=podalirius_\" title=\"Follow\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/podalirius_?label=Podalirius\u0026style=social\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.youtube.com/c/Podalirius_?sub_confirmation=1\" title=\"Subscribe\"\u003e\u003cimg alt=\"YouTube Channel Subscribers\" src=\"https://img.shields.io/youtube/channel/subscribers/UCF_x5O7CSfr82AfNVTKOv_A?style=social\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n\u003c/p\u003e\n\nWith this script, you can enumerate all URLs present in robots.txt files, and test whether you can access them or not.\n\n![](./.github/example.gif)\n\n## Setup\n\nClone the repository and install the dependencies :\n\n```sh\ngit clone https://github.com/p0dalirius/robotstester\ncd robotstester\npython3 setup.py install\n```\n\n## Usage\n\n```sh\nrobotstester -u http://www.example.com/\n```\n\nYou can find here a complete list of options :\n\n```\n[~] Robots.txt tester, v1.2.0\n\nusage: robotstester.py [-h] (-u URL | -f URLSFILE) [-v] [-q] [-k] [-L] [-t THREADS] [-p] [-j JSONFILE] [-x PROXY] [-b COOKIES]\n\nThis Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -u URL, --url URL     URL to the robots.txt to test e.g. https://example.com:port/path\n  -f URLSFILE, --urlsfile URLSFILE\n                        List of robots.txt urls to test\n  -v, --verbose         verbosity level (-v for verbose, -vv for debug)\n  -q, --quiet           Show no information at all\n  -k, --insecure        Allow insecure server connections when using SSL (default: False)\n  -L, --location        Follow redirects (default: False)\n  -t THREADS, --threads THREADS\n                        Number of threads (default: 5)\n  -p, --parsable        Parsable output\n  -j JSONFILE, --jsonfile JSONFILE\n                        Save results to specified JSON file.\n  -x PROXY, --proxy PROXY\n                        Specify a proxy to use for requests (e.g., http://localhost:8080)\n  -b COOKIES, --cookies COOKIES\n                        Specify cookies to use in requests. (e.g., --cookies \"cookie1=blah;cookie2=blah\")\n```\n\n## Contributing\n\nPull requests are welcome. Feel free to open an issue if you want to add other features.\n","funding_links":["https://github.com/sponsors/p0dalirius","https://patreon.com/Podalirius"],"categories":["[](#table-of-contents) Table of contents"],"sub_categories":["[](#dorkspentestvulnerabilities)Dorks/Pentest/Vulnerabilities"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0dalirius%2Frobotstester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fp0dalirius%2Frobotstester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0dalirius%2Frobotstester/lists"}