{"id":25074752,"url":"https://github.com/rix4uni/robotxt","last_synced_at":"2026-02-08T03:31:07.133Z","repository":{"id":265181811,"uuid":"894809465","full_name":"rix4uni/robotxt","owner":"rix4uni","description":"Extract endpoints marked as Allow and Disallow in robots.txt","archived":false,"fork":false,"pushed_at":"2024-11-27T03:12:00.000Z","size":2,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T19:47:00.818Z","etag":null,"topics":["bug-bounty","bugbounty","bugbountytips","hacking","infosec","osint","osint-resources","osint-tool","penetration-testing","pentest-tool","pentesting","recon","reconnaissance","robots-txt","security","security-tools","threat-intelligence"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rix4uni.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-27T03:08:47.000Z","updated_at":"2025-01-20T05:57:50.000Z","dependencies_parsed_at":"2024-11-28T07:18:13.613Z","dependency_job_id":null,"html_url":"https://github.com/rix4uni/robotxt","commit_stats":null,"previous_names":["rix4uni/robotxt"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rix4uni%2Frobotxt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rix4uni%2Frobotxt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rix4uni%2Frobotxt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rix4uni%2Frobotxt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rix4uni","download_url":"https://codeload.github.com/rix4uni/robotxt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253167789,"owners_count":21864708,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bug-bounty","bugbounty","bugbountytips","hacking","infosec","osint","osint-resources","osint-tool","penetration-testing","pentest-tool","pentesting","recon","reconnaissance","robots-txt","security","security-tools","threat-intelligence"],"created_at":"2025-02-07T00:18:28.188Z","updated_at":"2026-02-08T03:31:02.112Z","avatar_url":"https://github.com/rix4uni.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## robotxt\n\nExtract endpoints marked as Allow and Disallow in robots.txt\n\n## Installation\n```\ngo install github.com/rix4uni/robotxt@latest\n```\n\n## Download prebuilt binaries\n```\nwget https://github.com/rix4uni/robotxt/releases/download/v0.0.1/robotxt-linux-amd64-0.0.1.tgz\ntar -xvzf robotxt-linux-amd64-0.0.1.tgz\nrm -rf robotxt-linux-amd64-0.0.1.tgz\nmv robotxt ~/go/bin/robotxt\n```\nOr download [binary release](https://github.com/rix4uni/robotxt/releases) for your platform.\n\n## Compile from source\n```\ngit clone --depth 1 github.com/rix4uni/robotxt.git\ncd robotxt; go install\n```\n\n## Usage\n```\nUsage of robotxt:\n  -H string\n        Custom User-Agent for requests (default \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36\")\n  -complete\n        Include the full URL in the output\n  -delay duration\n        Delay between requests (default 0s)\n  -o string\n        File to save the output (default: print to stdout)\n  -silent\n        silent mode.\n  -timeout duration\n        Timeout duration for HTTP requests (default 10s)\n  -type string\n        Specify which one to extract: 'Disallow' or 'Allow'\n  -types-count\n        Print the count of types at the end\n  -verbose\n        Enable verbose output\n  -version\n        Print the version of the tool and exit.\n```\n\n## Usage\nSingle Target:\n```\n▶ echo \"https://www.google.com\" | robotxt -silent\n```\n\nMultiple Targets:\n```\n▶ cat targets.txt\nhttps://www.google.com\nhttps://www.facebook.com\n\n▶ cat targets.txt | robotxt -silent\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frix4uni%2Frobotxt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frix4uni%2Frobotxt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frix4uni%2Frobotxt/lists"}