{"id":13633410,"url":"https://github.com/s0md3v/uro","last_synced_at":"2025-05-14T01:05:47.302Z","repository":{"id":40518522,"uuid":"401081140","full_name":"s0md3v/uro","owner":"s0md3v","description":"declutters url lists for crawling/pentesting","archived":false,"fork":false,"pushed_at":"2025-02-23T11:53:31.000Z","size":48,"stargazers_count":1323,"open_issues_count":0,"forks_count":157,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-04-03T09:39:48.151Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s0md3v.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"s0md3v","custom":"https://paypal.me/s0md3v"}},"created_at":"2021-08-29T15:50:52.000Z","updated_at":"2025-03-31T19:52:10.000Z","dependencies_parsed_at":"2023-01-21T06:20:38.615Z","dependency_job_id":"946a02c1-8b85-42a6-a027-56a23b637b29","html_url":"https://github.com/s0md3v/uro","commit_stats":{"total_commits":29,"total_committers":4,"mean_commits":7.25,"dds":"0.10344827586206895","last_synced_commit":"fa649f052e069afef86d9f66502390fc7dc38c85"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s0md3v%2Furo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s0md3v%2Furo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s0md3v%2Furo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s0md3v%2Furo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s0md3v","download_url":"https://codeload.github.com/s0md3v/uro/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248228635,"owners_count":21068731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T23:00:38.665Z","updated_at":"2025-04-10T13:52:05.249Z","avatar_url":"https://github.com/s0md3v.png","language":"Python","readme":"# uro\nUsing a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; **uro** aims to solve that.\n\nIt doesn't make any http requests to the URLs and removes:\n- incremental urls e.g. `/page/1/` and `/page/2/`\n- blog posts and similar human written content e.g. `/posts/a-brief-history-of-time`\n- urls with same path but parameter value difference e.g. `/page.php?id=1` and `/page.php?id=2`\n- images, js, css and other \"useless\" files\n\n![uro-demo](https://i.ibb.co/x2tWCC5/uro-demo.png)\n\n#### Installation\nThe recommended way to install uro is as follows:\n```\npipx install uro\n```\n\u003e Note: If you are using an older version of python, use `pip` instead of `pipx`\n\n### Basic Usage\nThe quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.\n```\ncat urls.txt | uro\n```\n\n### Advanced usage\n#### Reading urls from a file (-i/--input)\n\n`uro -i input.txt`\n\n#### Writing urls to a file (-o/--output)\nIf the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.\n\n`uro -i input.txt -o output.txt`\n\n#### Whitelist (`-w/--whitelist`)\nuro will ignore all other extensions except the ones provided.\n\n`uro -w php asp html`\n\n**Note:** Extensionless pages e.g. `/books/1` will still be included. To remove them too, use  `--filter hasext`.\n\n#### Blacklist (`-b/--blacklist`)\nuro will ignore the given extensions.\n\n`uro -b jpg png js pdf`\n\n**Note:** uro has a list of \"useless\" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use `--filter hasext`.\n\n#### Filters (-f/--filters)\nFor granular control, uro supports the following filters:\n\n1. **hasparams:** only output urls that have query parameters e.g. `http://example.com/page.php?id=`\n2. **noparams:** only output urls that have no query parameters e.g. `http://example.com/page.php`\n3. **hasext:** only output urls that have extensions e.g. `http://example.com/page.php`\n4. **noext:** only output urls that have no extensions e.g. `http://example.com/page`\n5. **allexts:** don't remove any page based on extension e.g. keep `.jpg` which would be removed otherwise\n6. **keepcontent:** keep human written content e.g. blogs.\n7. **keepslash:** don't remove trailing slash from urls e.g. `http://example.com/page/`\n8. **vuln:** only output urls with parameters that are know to be vulnerable. [More info.](https://github.com/s0md3v/parth)\n\nExample: `uro --filters hasexts hasparams`\n","funding_links":["https://github.com/sponsors/s0md3v","https://paypal.me/s0md3v"],"categories":["Python","Miscellaneous","Weapons","其他_安全与渗透","红队\u0026渗透测试"],"sub_categories":["Useful","Tools","网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs0md3v%2Furo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs0md3v%2Furo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs0md3v%2Furo/lists"}