https://github.com/wfinn/urlame
declutters URL lists for hacking (not just grep -v)
https://github.com/wfinn/urlame
bugbounty pentesting reconnaissance url-filter urls
Last synced: 5 months ago
JSON representation
declutters URL lists for hacking (not just grep -v)
- Host: GitHub
- URL: https://github.com/wfinn/urlame
- Owner: wfinn
- Created: 2023-01-06T08:24:17.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-04-29T11:35:44.000Z (about 3 years ago)
- Last Synced: 2024-06-20T01:51:31.346Z (about 2 years ago)
- Topics: bugbounty, pentesting, reconnaissance, url-filter, urls
- Language: Go
- Homepage:
- Size: 48.8 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# urlame
This tool can reduce a list of URLs in a way which should be useful for pentesting / bug bounty.
E.g., when searching interesting URLs in the output of tools like `waymore`, this can do some initial filtering.
`urlame` aims to print one URL per feature of the website in addition to blocking known lame URLs.
This is done by converting a URL into a pattern and matching that against the patterns seen before.
## Things urlame considers lame
As a first step, `urlame` will filter out:
- lame directories like `/docs`
- files with lame extentensions like `.png`
- URLs that look like blog posts
- user profile/referral pages like `/user/FooBar`
This tool also ignores query values, so that only if a new parameter appears on a specific endpoint, the URL will be listed.
This means once `/foo?id=bar` was seen, `/foo?id=baz` will not be printed.
Certain URL query parameters are ignored completely, so that `/foo` and `/foo?utm_source=twitter` are considered equal.
It further can detect some patterns in parts of URLs which are ignored when comparing URLs.
- language codes
- numeric IDs
- hashes
- UUIDs
This means that `/en-US/upload/item/1` and `/de-DE/upload/item/5` are considered equal, so only the first will be printed.
## Usage
If you don't have Go installed read [this](https://go.dev/doc/install).
```sh
# installation
go install github.com/wfinn/urlame@latest
# basic usage
urlame < many_urls.txt > less_urls.txt
# practical example
waymore example.org | tee all_urls.txt | urlame > filtered_urls.txt
```
---
If you have ideas for more stuff to filter out or find a bug, [let me know](https://github.com/wfinn/urlame/issues/new).
Inspired by [uro](https://github.com/s0md3v/uro)