{"id":21917413,"url":"https://github.com/mhmdiaa/waybackrobots","last_synced_at":"2025-04-19T08:06:13.118Z","repository":{"id":153394547,"uuid":"626172021","full_name":"mhmdiaa/waybackrobots","owner":"mhmdiaa","description":"Enumerate old versions of robots.txt paths using Wayback Machine for content discovery","archived":false,"fork":false,"pushed_at":"2023-09-21T15:25:43.000Z","size":5,"stargazers_count":44,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T07:01:46.922Z","etag":null,"topics":["content-discovery","recon","robots-txt","wayback-machine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mhmdiaa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-11T00:15:14.000Z","updated_at":"2025-03-24T05:35:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"2f1e424a-87b5-498a-afb7-b5082678a5fc","html_url":"https://github.com/mhmdiaa/waybackrobots","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhmdiaa%2Fwaybackrobots","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhmdiaa%2Fwaybackrobots/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhmdiaa%2Fwaybackrobots/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mhmdiaa%2Fwaybackrobots/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mhmdiaa","download_url":"https://codeload.github.com/mhmdiaa/waybackrobots/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249644310,"owners_count":21305462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["content-discovery","recon","robots-txt","wayback-machine"],"created_at":"2024-11-28T19:31:49.620Z","updated_at":"2025-04-19T08:06:13.112Z","avatar_url":"https://github.com/mhmdiaa.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# waybackrobots\nEnumerate old versions of robots.txt paths using Wayback Machine for content discovery.\n\n## Usage\nPass a list of domains/URLs through stdin and waybackrobots will write the full URLs to stdout.\n```sh\n$ cat targets.txt | waybackrobots\n\nEnumerating https://google.com/robots.txt versions... 100% |███████████████████████████████████████████| (50/50, 10 it/s)\n\nhttps://google.com/analytics/reporting/\nhttps://google.com/ebooks?*q=related:*\nhttps://google.com/compare/*/apply*\n...\n```\n\n## Command-line options\n\n| Option   | Description                                                    | Default |\n|----------|----------------------------------------------------------------|---------|\n| -limit   | Limit the number of crawled snapshots. Use -1 for unlimited.   | 50       |\n| -recent  | Use the most recent snapshots without evenly distributing them | false   |\n\n## Snapshot Distribution\nBy default, `waybackrobots` evenly distributes the snapshots it analyzes across the file's history when a limit is set. This is done to diversify the results and get a broader view of the `robots.txt` file over time.\n\nFor example, if you set the limit to 5 and there are 10 snapshots, waybackrobots will analyze every other snapshot starting from the latest one. This means it will analyze the first, third, fifth, seventh, and ninth most recent snapshots.\n\nThis default behavior can be changed with the `-recent` option, which tells `waybackrobots` to use only the most recent snapshots.\n\n```sh\n$ echo google.com | waybackrobots | wc\n     422     422   13973\n$ echo google.com | waybackrobots -recent | wc\n     277     277    9100\n```\n\n## Installation\n### Binary\nCheck out the [latest release](https://github.com/mhmdiaa/waybackrobots/releases/latest).\n\n### Go install\n```\ngo install github.com/mhmdiaa/waybackrobots@latest\n```\n\n## References\n- This tool is an improved and updated version of [waybackrobots.py](https://gist.github.com/mhmdiaa/2742c5e147d49a804b408bfed3d32d07).\n- If you need a more customizable tool for working with Wayback Machine data, check out [chronos](https://github.com/mhmdiaa/chronos).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmhmdiaa%2Fwaybackrobots","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmhmdiaa%2Fwaybackrobots","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmhmdiaa%2Fwaybackrobots/lists"}