{"id":21743399,"url":"https://github.com/synacktraa/urify","last_synced_at":"2025-10-07T15:30:47.179Z","repository":{"id":202866666,"uuid":"708046609","full_name":"synacktraa/urify","owner":"synacktraa","description":"Dissect and filter URLs provided on stdin.","archived":false,"fork":false,"pushed_at":"2023-10-23T21:06:15.000Z","size":44,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-01T12:03:56.659Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/synacktraa.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-21T11:10:42.000Z","updated_at":"2024-05-10T13:12:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"f3e8ae50-55e2-4f34-8da5-9ae423a193b2","html_url":"https://github.com/synacktraa/urify","commit_stats":null,"previous_names":["synacktraa/urify"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/synacktraa/urify","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synacktraa%2Furify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synacktraa%2Furify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synacktraa%2Furify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synacktraa%2Furify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/synacktraa","download_url":"https://codeload.github.com/synacktraa/urify/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synacktraa%2Furify/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278799015,"owners_count":26048018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T07:07:50.378Z","updated_at":"2025-10-07T15:30:47.174Z","avatar_url":"https://github.com/synacktraa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Urify 🫧\n\nStreamlining URLs for security testing - No more duplicates, just streamlined precision\n\n---\n\n### 🚀 Installation\n\n1.  Installation using PIP\n\n    ```sh\n    pip install urify\n    ```\n\n2. Installation using Docker\n\n    ```\n    docker pull synacktra/urify\n    ```\n\n---\n\n### 🛠 Commands:\n\nOh, you thought this was just a basic tool? Think again!\n\n#### 🔍 Basic URL Dissections:\n\n- `keys`: Want to know what keys are in your query string? I got you!\n- `values`: Retrieve values from the query string. Yes, like treasure hunting but more nerdy.\n- `params`: Get those key=value pairs, one at a time, like peeling an onion.\n- `apex`: Get the apex domain. It's like climbing a mountain, but lazier.\n- `fqdn`: Retrieve the fully qualified domain name. Fancy words, right?\n- `json`: Encode your dissected URL into JSON. Because... JSON!\n\n#### 🎛 Advanced Filtering:\n\n- `filter`: Refine your URLs using a myriad of component filters:\n\n  - `-scheme`: Filter by request schemes. Yes, we're getting technical here.\n  - `-sub`: Look for specific subdomains. It's like Where's Waldo but for domains.\n  - `-domain`: Specify domains. Because we all have favorites.\n  - `-tld`: Filter by top-level domains. It's all about the hierarchy.\n  - `-ext`: Filter by file extensions. The grand finale of URLs.\n  - `-port`: Specify ports. No, not the wine. The number.\n  - `-apex`: Get URLs by apex domain. It's like VIP access.\n  - `-fqdn`: Filter by fully qualified domain names. The whole shebang!\n  - `-inverse`: Want to be rebellious? Use filters as a deny-list.\n  - `-strict`: Validate all filters. No half measures here!\n  - `-dissect`: Once filtered, dissect the URL for a specific component. Pick and choose!\n\n---\n\n### 📖 Examples:\n\n#### 1. Basic URL Dissections:\n\n```\n$ urify -help\nDissect and filter URLs provided on stdin.\n\nUsage: urify [-help] [mode] ...\n\nOptions:\n  -help     show this help message\n\nModes:\n    keys    Retrieve keys from the query string, one per line.\n    values  Retrieve values from the query string, one per line.\n    params  Key=value pairs from the query string (one per line)\n    path    Retrieve the path (e.g., /users/me).\n    apex    Retrieve the apex domain (e.g., github.com).\n    fqdn    Retrieve the fully qualified domain name (e.g., api.github.com).\n    json    JSON encode the dissected URL object.\n    filter  Refine URLs using component filters.\n```\n\n- **keys**:\n  ```\n  $ cat urls.txt | urify keys\n  search\n  id\n  product-id\n  name\n  ```\n\n- **values**:\n  ```\n  $ cat urls.txt | urify values\n  query\n  123\n  29\n  powder\n  ```\n\n- **params**:\n  ```\n  $ cat urls.txt | urify params\n  search=query\n  id=123\n  product-id=29\n  name=powder\n  ```\n\n- **apex**:\n  ```\n  $ cat urls.txt | urify apex\n  example.com\n  spaghetti.com\n  example.org\n  cartel.net\n  ```\n\n- **fqdn**:\n  ```\n  $ cat urls.txt | urify fqdn\n  sub.example.com\n  mom.spaghetti.com\n  blog.example.org\n  shop.cartel.net\n  ```\n\n- **json**:\n  ```\n  echo \"http://bob:secret@sub.example.com:80/file%20name.pdf?page=10\" | urify json\n  ```\n\n  ```  \n  {\n    \"scheme\": \"http\",\n    \"username\": \"bob\",\n    \"password\": \"secret\",\n    \"subdomain\": \"sub\",\n    \"domain\": \"example\",\n    \"tld\": \"com\",\n    \"port\": \"80\",\n    \"path\": \"/file%20name.pdf\",\n    \"raw_query\": \"page=10\",\n    \"query\": [\n      {\n        \"key\": \"page\",\n        \"value\": \"10\"\n      }\n    ],\n    \"fragment\": \"\",\n    \"apex\": \"example.com\",\n    \"fqdn\": \"sub.example.com\"\n  }\n  ```\n\n#### 2. Advanced Filtering:\n\n```\n$ urify filter -help\n\nRefine URLs using component filters.\nBy default, the command treats provided filters as an allow-list,\nevaluating just one parameter from a specified field.\n\nTo assess all components, add the `-strict` flag.\nFor a deny-list approach, incorporate the `-inverse` flag.\nAfter evaluation, the default result is the URL.\nFor specific URL dissected object, pair the `-dissect` option with any one of:\n`keys` | `values` | `params` | `apex` | `fqdn` | `json`\nIf `-dissect` is not provided, `filter` command aims to print \ndissimilar urls using pattern matching\n\nUsage: urify filter [-help] [-scheme SCHEME [...]] [-sub SUB [...]] [-domain DOMAIN [...]] [-tld TLD [...]] [-ext EXT [...]]     \n                      [-port PORT [...]] [-apex APEX [...]] [-fqdn FQDN [...]] [-inverse] [-strict] [-dissect MODE]\n\nOptions:\n  -help                 show this help message\n  -scheme SCHEME [...]  The request schemes (e.g. http, https)\n  -sub SUB [...]        The subdomains (e.g. abc, abc.xyz)\n  -domain DOMAIN [...]  The domains (e.g. github, youtube)\n  -tld TLD [...]        The top level domains (e.g. in, com)\n  -ext EXT [...]        The file extensions (e.g. pdf, html)\n  -port PORT [...]      The ports (e.g. 22, 8080)\n  -apex APEX [...]      The apex domains (e.g. github.com, youtube.com)\n  -fqdn FQDN [...]      The fully qualified domain names (e.g. api.github.com, app.example.com)\n  -inverse              Process filters as deny-list\n  -strict               Validate all filter checks\n  -dissect MODE         Dissect url and retrieve mode after filtering\n\nModes:\n  keys|values|params|path|apex|fqdn|json\n```\n\n- **filter dissimilar urls**\n  ```\n  $ cat similar_urls.txt | urify filter\n  http://blog.example.net/post/12\n  http://info.example.net/tag/14?id=133\n  ```\n\n- **filter by scheme**:\n  ```\n  $ cat urls.txt | urify filter -scheme https\n  https://sub.example.com/literally%20me.jpg#top\n  ```\n\n- **filter by domain**:\n  ```\n  $ cat urls.txt | urify filter -domain example\n  https://sub.example.com/literally%20me.jpg#top\n  http://blog.example.org:8443/new%20blog?id=123\n  ```\n\n- **filter and dissect**:\n  ```\n  $ cat urls.txt | urify filter -ext pdf jpg -dissect apex\n  example.com\n  spaghetti.com\n  ```\n\n- **filters as deny-list**:\n  ```\n  $ cat urls.txt | urify filter -ext html -inverse\n  https://sub.example.com/literally%20me.jpg#top\n  http://mom.spaghetti.com:8080/cookbook.pdf?search=query\n  http://blog.example.org:8443/new%20blog?id=123\n  ```\n\n- **validate all filters**:\n  ```\n  $ cat urls.txt | urify filter -tld com -port 8080 -strict\n  http://mom.spaghetti.com:8080/cookbook.pdf?search=query\n  ```\n\n---\n\n### 📝 Notes:\n\n- By default, the `filter` command treats the provided filters as an allow-list. But if you're feeling a little naughty, use the `-inverse` flag for a deny-list approach.\n- If you're the meticulous type and want to validate every single filter parameter, add the `-strict` flag to your command. No stone unturned!\n- After all the filtering shenanigans, if you just want the URL, we'll give it to you. But if you're in the mood for something special, pair the `-dissect` option with any one of the listed choices.\n\n---\n\n## 🧠 Custom CLI Parser (Skip this part):\n\nEver felt that built-in command-line parsers are just... meh? I did too. That's why urify is powered by its very own custom CLI parser. Why settle for the ordinary when you can have the extraordinary? \n\n### 🧐 Features:\n\n- **Command-centric**: Create commands as easily as adding a decorator with `@cli.command(...)`. Seriously, it's like sprinkling magic dust on your functions.\n- **Options Galore**: Add options to your commands using the `@cli.option(...)` decorator. Define data types, set multiple values, and even specify valid choices. It's like an all-you-can-eat buffet, but for command-line arguments.\n\n### 📖 Behind the Scenes:\n\n#### Command Class:\n\nThe `Command` class encapsulates each command. It's like a VIP pass for your functions:\n\n- `name`: The unique identifier for your command.\n- `callback`: The function to be invoked when the command is called.\n- `parser`: The argument parser associated with this command. \n\n#### CLI Class:\n\nThe heart and soul of the custom parser. The `Cli` class:\n\n- **Initialization**: Set up your CLI with descriptions, help flags, and more. Feel like a movie director but for commands.\n- **Command Registration**: Register functions as commands with the `@cli.command(...)` decorator. It's like enlisting soldiers for battle.\n- **Option Handling**: Define options with the `@cli.option(...)` decorator. Like adding extra toppings to your pizza.\n- **Execution**: Finally, the `cli.run()` method brings everything together by parsing the arguments and invoking the right command. Lights, camera, action!\n\n### 🤓 Technical Deep Dive:\n\nThe magic happens with the use of the `_ArgumentParser` class (a modified version of `argparse.ArgumentParser`). This allows for a more intuitive and concise definition of command-line interfaces. The beauty of this parser is that it draws inspiration from renowned libraries like `click` and `typer` while maintaining the unique aspects of `argparse`.\n\nOverall, this custom CLI parser provides a seamless and powerful way to interact with command-line tools without the usual hassles.\n\n---\n\n*Disclaimer: No URLs were harmed in the making of this tool.*\n\n---\n\nHappy dissecting! 🎈","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynacktraa%2Furify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsynacktraa%2Furify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynacktraa%2Furify/lists"}