{"id":27071076,"url":"https://github.com/hahwul/urx","last_synced_at":"2026-06-27T03:04:38.383Z","repository":{"id":284880144,"uuid":"956297926","full_name":"hahwul/urx","owner":"hahwul","description":"Extracts URLs from OSINT Archives for Security Insights","archived":false,"fork":false,"pushed_at":"2025-06-18T14:09:10.000Z","size":4893,"stargazers_count":141,"open_issues_count":3,"forks_count":13,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-26T14:09:20.300Z","etag":null,"topics":["osint","osint-tool","security","url","urx","wayback-machine"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/urx","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hahwul.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"hahwul"}},"created_at":"2025-03-28T02:48:27.000Z","updated_at":"2025-06-25T07:38:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"ae75bea0-692e-4a6f-9885-c16cfa6a544a","html_url":"https://github.com/hahwul/urx","commit_stats":null,"previous_names":["hahwul/urx"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/hahwul/urx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahwul%2Furx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahwul%2Furx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahwul%2Furx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahwul%2Furx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hahwul","download_url":"https://codeload.github.com/hahwul/urx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hahwul%2Furx/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263083730,"owners_count":23411166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["osint","osint-tool","security","url","urx","wayback-machine"],"created_at":"2025-04-05T23:00:50.068Z","updated_at":"2026-06-27T03:04:38.377Z","avatar_url":"https://github.com/hahwul.png","language":"Rust","funding_links":["https://github.com/sponsors/hahwul"],"categories":["Weapons","Rust"],"sub_categories":["Tools"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cpicture\u003e\n        \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/static/images/urx-dark.png\" width=\"500px;\"\u003e\n        \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"docs/static/images/urx-light.png\" width=\"500px;\"\u003e\n        \u003cimg alt=\"Urx Logo\" src=\"docs/static/images/urx-dark.png\" width=\"500px;\"\u003e\n  \u003c/picture\u003e\n  \u003cp\u003eExtracts URLs from OSINT Archives for Security Insights.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/hahwul/urx/releases/latest\"\u003e\u003cimg src=\"https://img.shields.io/github/v/release/hahwul/urx?style=for-the-badge\u0026logoColor=%23000000\u0026label=urx\u0026labelColor=%23000000\u0026color=%23000000\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://app.codecov.io/gh/hahwul/urx\"\u003e\u003cimg src=\"https://img.shields.io/codecov/c/gh/hahwul/urx?style=for-the-badge\u0026logoColor=%23000000\u0026labelColor=%23000000\u0026color=%23000000\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/hahwul/urx/blob/main/CONTRIBUTING.md\"\u003e\u003cimg src=\"https://img.shields.io/badge/CONTRIBUTIONS-WELCOME-000000?style=for-the-badge\u0026labelColor=000000\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://rust-lang.org\"\u003e\u003cimg src=\"https://img.shields.io/badge/Rust-000000?style=for-the-badge\u0026logo=rust\u0026logoColor=white\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nUrx is a command-line tool designed for collecting URLs from OSINT archives, such as the Wayback Machine and Common Crawl. Built with Rust for efficiency, it leverages asynchronous processing to rapidly query multiple data sources. This tool simplifies the process of gathering URL information for a specified domain, providing a comprehensive dataset that can be used for various purposes, including security testing and analysis.\n\n## Features\n\n* Fetch URLs from multiple sources in parallel (Wayback Machine, Common Crawl, OTX, Arquivo.pt)\n* Keyless by default: Wayback, Common Crawl, OTX, Arquivo.pt, and URLScan (anonymous) all work without an API key\n* API key rotation support for VirusTotal and URLScan providers to mitigate rate limits\n* Filter results by file extensions, patterns, or predefined presets (e.g., \"no-image\" to exclude images)\n* URL normalization and deduplication: Sort query parameters, remove trailing slashes, and merge semantically identical URLs\n* Support for multiple output formats: plain text, JSON, CSV\n* Direct file input support: Read URLs directly from WARC files, URLTeam compressed files, and text files\n* Output results to the console or a file, or stream via stdin for pipeline integration\n* URL Testing:\n  * Filter and validate URLs based on HTTP status codes and patterns.\n  * Extract additional links from collected URLs\n* Caching and Incremental Scanning:\n  * Local SQLite or remote Redis caching to avoid re-scanning domains\n  * Incremental mode to discover only new URLs since last scan\n  * Configurable cache TTL and automatic cleanup of expired entries\n\n![Preview](https://raw.githubusercontent.com/hahwul/urx/refs/heads/main/docs/static/images/preview.jpg)\n\n## Installation\n\n### From Cargo\n\n```bash\n# https://crates.io/crates/urx\ncargo install urx\n```\n\n### From Homebrew\n\n```bash\n# https://formulae.brew.sh/formula/urx\nbrew install urx\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/hahwul/urx.git\ncd urx\ncargo build --release\n```\n\nThe compiled binary will be available at `target/release/urx`.\n\n### From Docker\n\n[ghcr.io/hahwul/urx](https://github.com/hahwul/urx/pkgs/container/urx)\n\n## Usage\n\n### Basic Usage\n\n```bash\n# Scan a single domain\nurx example.com\n\n# Scan multiple domains\nurx example.com example.org\n\n# Scan domains from a file\ncat domains.txt | urx\n```\n\n### Options\n\n```\nUsage: urx [OPTIONS] [DOMAINS]...\n\nArguments:\n  [DOMAINS]...  Domains to fetch URLs for\n\nOptions:\n  -c, --config \u003cCONFIG\u003e           Config file to load\n      --provider-config \u003cPATH\u003e    Separate provider config file holding only API keys (default: $XDG_CONFIG_HOME/urx/provider-config.toml). CLI/env \u003e provider-config \u003e main config.\n  -h, --help             Print help\n  -V, --version          Print version\n\nInput Options:\n      --files \u003cFILES\u003e...        Read URLs directly from files (supports WARC, URLTeam compressed, and text files)\n      --domain-list \u003cPATH\u003e      File of newline-separated domains to scan (repeatable; merged with positional DOMAINS and stdin; `#` comments allowed)\n\nOutput Options:\n  -o, --output \u003cOUTPUT\u003e          Output file to write results\n      --output-dir \u003cPATH\u003e        Write one file per domain into this directory (extension matches --format). Coexists with --output / stdout.\n  -f, --format \u003cFORMAT\u003e          Output format (e.g., \"plain\", \"json\", \"csv\") [default: plain]\n      --merge-endpoint   Merge endpoints with the same path and merge URL parameters\n      --normalize-url    Normalize URLs for better deduplication (sorts query parameters, removes trailing slashes)\n\nProvider Options:\n      --providers \u003cPROVIDERS\u003e\n          Providers to use (comma-separated, e.g., \"wayback,cc,otx,arquivo,vt,urlscan\") [default: wayback,cc,otx]\n      --exclude-providers \u003cEXCLUDE_PROVIDERS\u003e\n          Providers to exclude (comma-separated). Wins on conflict with --providers / --all-providers.\n      --all-providers\n          Enable every supported provider. API-keyed providers only activate when a key is available.\n      --list-providers\n          List every supported provider then exit.\n      --subs\n          Include subdomains when searching\n      --cc-index \u003cCC_INDEX\u003e\n          Common Crawl index to use; accepts comma-separated list to query multiple indexes in parallel (e.g. `CC-MAIN-2026-17,CC-MAIN-2025-51`). `latest` (the default) resolves the newest via collinfo.json. [default: latest]\n      --wayback-from \u003cDATE\u003e\n          Restrict Wayback Machine results to snapshots at or after DATE (YYYY/YYYYMM/YYYYMMDD/YYYYMMDDhhmmss)\n      --wayback-to \u003cDATE\u003e\n          Restrict Wayback Machine results to snapshots at or before DATE (same format as --wayback-from)\n      --vt-api-key \u003cVT_API_KEY\u003e\n          API key for VirusTotal (can be used multiple times for rotation, can also use URX_VT_API_KEY environment variable with comma-separated keys)\n      --urlscan-api-key \u003cURLSCAN_API_KEY\u003e\n          Optional API key for Urlscan; the provider also works anonymously (rate-limited ~30 req/min per IP). Can be used multiple times for rotation, or via URX_URLSCAN_API_KEY (comma-separated keys)\n      --github-api-key \u003cGITHUB_API_KEY\u003e\n          Personal access token for the GitHub Code Search provider (also reads URX_GITHUB_API_KEY, comma-separated for rotation)\n\nDiscovery Options:\n      --exclude-robots   Exclude robots.txt discovery\n      --exclude-sitemap  Exclude sitemap.xml discovery\n\nDisplay Options:\n  -v, --verbose       Show verbose output\n      --silent        Silent mode (no output)\n      --no-progress   No progress bar\n      --show-sources  Annotate output URLs with the providers that returned them\n      --stats         Print a per-provider summary to stderr at end of run\n\nFilter Options:\n  -p, --preset \u003cPRESET\u003e\n          Filter Presets (e.g., \"no-resources,no-images,only-js,only-style\")\n  -e, --extensions \u003cEXTENSIONS\u003e\n          Filter URLs to only include those with specific extensions (comma-separated, e.g., \"js,php,aspx\")\n      --exclude-extensions \u003cEXCLUDE_EXTENSIONS\u003e\n          Filter URLs to exclude those with specific extensions (comma-separated, e.g., \"html,txt\")\n      --patterns \u003cPATTERNS\u003e\n          Filter URLs to only include those containing specific patterns (comma-separated)\n      --exclude-patterns \u003cEXCLUDE_PATTERNS\u003e\n          Filter URLs to exclude those containing specific patterns (comma-separated)\n      --show-only-host\n          Only show the host part of the URLs\n      --show-only-path\n          Only show the path part of the URLs\n      --show-only-param\n          Only show the parameters part of the URLs\n      --min-length \u003cMIN_LENGTH\u003e\n          Minimum URL length to include\n      --max-length \u003cMAX_LENGTH\u003e\n          Maximum URL length to include\n      --strict\n          Enforce exact host validation (default)\n\nNetwork Options:\n      --network-scope \u003cNETWORK_SCOPE\u003e  Control which components network settings apply to (all, providers, testers, or providers,testers) [default: all]\n      --proxy \u003cPROXY\u003e                  Use proxy for HTTP requests (format: \u003chttp://proxy.example.com:8080\u003e)\n      --proxy-auth \u003cPROXY_AUTH\u003e        Proxy authentication credentials (format: username:password)\n      --insecure                       Skip SSL certificate verification (accept self-signed certs)\n      --random-agent                   Use a random User-Agent for HTTP requests\n      --timeout \u003cTIMEOUT\u003e              Request timeout in seconds [default: 120]\n      --retries \u003cRETRIES\u003e              Number of retries for failed requests [default: 2]\n      --parallel \u003cPARALLEL\u003e            Maximum domains fetched concurrently per provider (and concurrent URL tests); a provider's --rate-limit is shared across them [default: 5]\n      --rate-limit \u003cRATE_LIMIT\u003e        Rate limit (requests per second)\n      --rate-limit-by \u003cPAIRS\u003e          Per-provider rate overrides (e.g. `vt=1,wayback=10`); falls back to --rate-limit for unlisted providers\n      --max-time \u003cMAX_TIME\u003e            Global ceiling on provider enumeration time in seconds (0 = unlimited) [default: 0]\n\nTesting Options:\n      --check-status\n          Check HTTP status code of collected URLs [aliases: ----cs]\n      --include-status \u003cINCLUDE_STATUS\u003e\n          Include URLs with specific HTTP status codes or patterns (e.g., --is=200,30x) [aliases: ----is]\n      --exclude-status \u003cEXCLUDE_STATUS\u003e\n          Exclude URLs with specific HTTP status codes or patterns (e.g., --es=404,50x,5xx) [aliases: ----es]\n      --extract-links\n          Extract additional links from collected URLs (requires HTTP requests)\n```\n\n### Examples\n\n```bash\n# Save results to a file\nurx example.com -o results.txt\n\n# Output in JSON format\nurx example.com -f json -o results.json\n\n# Filter for JavaScript files only\nurx example.com -e js\n\n# Exclude HTML and text files\nurx example.com --exclude-extensions html,txt\n\n# Filter for API endpoints\nurx example.com --patterns api,v1,graphql\n\n# Exclude specific patterns\nurx example.com --exclude-patterns static,images\n\n# Use Fileter Preset (similar to --exclude-extensions=png,jpg,.....)\nurx example.com -p no-images\n\n# Use specific providers\nurx example.com --providers wayback,otx\n\n# Add the keyless Arquivo.pt (Portuguese web archive) provider\nurx example.com --providers wayback,cc,otx,arquivo\n\n# URLScan works without a key (anonymous, rate-limited); a key just raises limits\nurx example.com --providers urlscan\n\n# Using VirusTotal and URLScan providers\n# 1. Explicitly add to providers (with API keys via command line)\nurx example.com --providers=vt,urlscan --vt-api-key=*** --urlscan-api-key=***\n\n# 2. Using environment variables for API keys\nURX_VT_API_KEY=*** URX_URLSCAN_API_KEY=*** urx example.com --providers=vt,urlscan\n\n# 3. Auto-enabling: providers are automatically added when API keys are provided\nurx example.com --vt-api-key=*** --urlscan-api-key=*** # No need to specify in --providers\n\n# 4. Multiple API key rotation (to mitigate rate limits)\n# Using repeated flags for multiple keys\nurx example.com --vt-api-key=key1 --vt-api-key=key2 --vt-api-key=key3\n\n# Using environment variables with comma-separated keys\nURX_VT_API_KEY=key1,key2,key3 URX_URLSCAN_API_KEY=ukey1,ukey2 urx example.com\n\n# Combining CLI flags and environment variables (CLI keys are used first)\nURX_VT_API_KEY=env_key1,env_key2 urx example.com --vt-api-key=cli_key1 --vt-api-key=cli_key2\n\n# URLs from robots.txt and sitemap.xml are included by default\n\n# Exclude URLs from robots.txt files\nurx example.com --exclude-robots\n\n# Exclude URLs from sitemap\nurx example.com --exclude-sitemap\n\n# Include subdomains\nurx example.com --subs\n\n# Check status of collected URLs\nurx example.com --check-status\n\n# Read URLs directly from a text file\nurx --files urls.txt\n\n# Combine file input with filtering\nurx --files urls.txt --patterns api,admin -f json\n\n# Extract additional links from collected URLs\nurx example.com --extract-links\n\n# Network configuration\nurx example.com --proxy http://localhost:8080 --timeout 60 --parallel 10 --insecure\n\n# Advanced filtering\nurx example.com -e js,php --patterns admin,login --exclude-patterns logout,static --min-length 20\n\n# HTTP Status code based filtering\nurx example.com --include-status 200,30x,405 --exclude-status 20x\n\n# Disable host validation\nurx example.com --strict false\n\n# URL normalization and deduplication\n# Normalize URLs by sorting query parameters and removing trailing slashes\nurx example.com --normalize-url\n\n# Combine normalization with endpoint merging for comprehensive deduplication\nurx example.com --normalize-url --merge-endpoint\n\n# URL normalization with file input\nurx --files urls.txt --normalize-url\n```\n\n### Caching and Incremental Scanning\n\nUrx supports caching to improve performance for repeated scans and incremental scanning to discover only new URLs.\n\n```bash\n# Enable caching with SQLite (default)\nurx example.com --cache-type sqlite --cache-path ~/.urx/cache.db\n\n# Use Redis for distributed caching\nurx example.com --cache-type redis --redis-url redis://localhost:6379\n\n# Incremental scanning - only show new URLs since last scan\nurx example.com --incremental\n\n# Set cache TTL (time-to-live) to 12 hours\nurx example.com --cache-ttl 43200\n\n# Disable caching entirely\nurx example.com --no-cache\n\n# Combine incremental scanning with filters\nurx example.com --incremental -e js,php --patterns api\n\n# Configuration file with caching settings\nurx -c example/config.toml example.com\n```\n\n#### Caching Use Cases\n\n```bash\n# Daily monitoring - only alert on new URLs\nurx target.com --incremental --silent | notify-tool\n\n# Efficient domain lists processing\ncat domains.txt | urx --incremental --cache-ttl 3600 \u003e new_urls.txt\n\n# Distributed team scanning with Redis\nurx example.com --cache-type redis --redis-url redis://shared-cache:6379\n\n# Fast re-scans during development\nurx test-domain.com --cache-ttl 300  # 5-minute cache for rapid iterations\n```\n\n## Integration with Other Tools\n\nUrx works well in pipelines with other security and reconnaissance tools:\n\n```bash\n# Find domains, then discover URLs\necho \"example.com\" | urx | grep \"login\" \u003e potential_targets.txt\n\n# Combine with other tools\ncat domains.txt | urx --patterns api | other-tool\n```\n\n## Inspiration\n\nUrx was inspired by [gau (GetAllUrls)](https://github.com/lc/gau), a tool that fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. While sharing similar core functionality, Urx was built from the ground up in Rust with a focus on performance, concurrency, and expanded filtering capabilities.\n\n## Contribute\n\nUrx is open-source project and made it with ❤️\nif you want contribute this project, please see [CONTRIBUTING.md](./CONTRIBUTING.md) and Pull-Request with cool your contents.\n\n[![](https://raw.githubusercontent.com/hahwul/urx/refs/heads/main/CONTRIBUTORS.svg)](https://github.com/hahwul/urx/graphs/contributors)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahwul%2Furx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhahwul%2Furx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhahwul%2Furx/lists"}