Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/s0md3v/uro
declutters url lists for crawling/pentesting
https://github.com/s0md3v/uro
Last synced: 20 days ago
JSON representation
declutters url lists for crawling/pentesting
- Host: GitHub
- URL: https://github.com/s0md3v/uro
- Owner: s0md3v
- License: apache-2.0
- Created: 2021-08-29T15:50:52.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-27T19:04:09.000Z (5 months ago)
- Last Synced: 2024-10-01T15:21:56.775Z (about 1 month ago)
- Language: Python
- Size: 40 KB
- Stars: 1,124
- Watchers: 18
- Forks: 136
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-bugbounty-tools - uro - declutters url lists for crawling/pentesting (Miscellaneous / Useful)
- WebHackersWeapons - uro
- awesome-hacking-lists - s0md3v/uro - declutters url lists for crawling/pentesting (Python)
- StarryDivineSky - s0md3v/uro
README
# uro
Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; **uro** aims to solve that.
It doesn't make any http requests to the URLs and removes:
- incremental urls e.g. `/page/1/` and `/page/2/`
- blog posts and similar human written content e.g. `/posts/a-brief-history-of-time`
- urls with same path but parameter value difference e.g. `/page.php?id=1` and `/page.php?id=2`
- images, js, css and other "useless" files
#### Installation
The recommended way to install uro is through pip as follows:
```
pip3 install uro
```
### Basic Usage
The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.
```
cat urls.txt | uro
```
### Advanced usage
#### Reading urls from a file (-i/--input)
`uro -i input.txt`
#### Writing urls to a file (-o/--output)
If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.
`uro -i input.txt -o output.txt`
#### Whitelist (`-w/--whitelist`)
uro will ignore all other extensions except the ones provided.
`uro -w php asp html`
**Note:** Extensionless pages e.g. `/books/1` will still be included. To remove them too, use `--filter hasext`.
#### Blacklist (`-b/--blacklist`)
uro will ignore the given extensions.
`uro -b jpg png js pdf`
**Note:** uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use `--filter hasext`.
#### Filters (-f/--filters)
For granular control, uro supports the following filters:
1. **hasparams:** only output urls that have query parameters e.g. `http://example.com/page.php?id=`
2. **noparams:** only output urls that have no query parameters e.g. `http://example.com/page.php`
3. **hasext:** only output urls that have extensions e.g. `http://example.com/page.php`
4. **noext:** only output urls that have no extensions e.g. `http://example.com/page`
5. **keepcontent:** keep human written content e.g. blogs.
6. **keepslash:** don't remove trailing slash from urls e.g. `http://example.com/page/`
7. **vuln:** only output urls with parameters that are know to be vulnerable. [More info.](https://github.com/s0md3v/parth)
Example: `uro --filters hasexts hasparams`