{"id":29663224,"url":"https://github.com/l0g0rhythm/url-refiner","last_synced_at":"2025-07-22T11:32:58.108Z","repository":{"id":303705369,"uuid":"1016402425","full_name":"L0g0rhythm/URL-Refiner","owner":"L0g0rhythm","description":"A Python tool to efficiently process, modify, and deduplicate URL lists. Ideal for security professionals, analysts, and developers, with both CLI and GUI support.","archived":false,"fork":false,"pushed_at":"2025-07-09T01:09:37.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-09T02:24:46.690Z","etag":null,"topics":["automation","bug-bounty","cli","data-processing","deduplication","gui","python","security","security-tools","url","url-manipulation","url-parser"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/L0g0rhythm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-09T01:02:56.000Z","updated_at":"2025-07-09T01:09:40.000Z","dependencies_parsed_at":"2025-07-09T02:25:09.453Z","dependency_job_id":"0ec3c4a2-c31f-487a-8e1a-6f94b2c8cd4d","html_url":"https://github.com/L0g0rhythm/URL-Refiner","commit_stats":null,"previous_names":["l0g0rhythm/url-refiner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/L0g0rhythm/URL-Refiner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/L0g0rhythm%2FURL-Refiner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/L0g0rhythm%2FURL-Refiner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/L0g0rhythm%2FURL-Refiner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/L0g0rhythm%2FURL-Refiner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/L0g0rhythm","download_url":"https://codeload.github.com/L0g0rhythm/URL-Refiner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/L0g0rhythm%2FURL-Refiner/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266483486,"owners_count":23936348,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","bug-bounty","cli","data-processing","deduplication","gui","python","security","security-tools","url","url-manipulation","url-parser"],"created_at":"2025-07-22T11:30:51.741Z","updated_at":"2025-07-22T11:32:58.085Z","avatar_url":"https://github.com/L0g0rhythm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# URL Refiner\r\n\r\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\r\n[![Python Version](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\r\n\r\nA robust and elegant tool to process, modify, and deduplicate URL lists. Designed for security professionals, data analysts, and developers who need to manipulate large volumes of URL parameters efficiently.\r\n\r\n---\r\n\r\n## 📚 Index\r\n\r\n- [✨ Features](#-features)\r\n- [🛠️ Technology Stack](#️-technology-stack)\r\n- [🚀 Installation](#-installation)\r\n- [🖥️ Usage](#-usage)\r\n  - [🔧 CLI (Command-Line Interface)](#-cli-command-line-interface)\r\n  - [🖱️ GUI (Graphical User Interface)](#️-gui-graphical-user-interface)\r\n- [✅ Tests](#-tests)\r\n- [📄 License](#-license)\r\n- [📬 Contact](#-contact)\r\n\r\n---\r\n\r\n## ✨ Features\r\n\r\n- **Parameter Manipulation**: Replace or append values to query parameters in thousands of URLs.\r\n- **Smart Deduplication**: Removes duplicate URLs based on domain, path (optional), and parameter names.\r\n- **Flexible I/O**: Reads from files or stdin—ideal for script pipelines.\r\n- **Organized Output**: Automatically saves processed lists to the `output/` directory with timestamp.\r\n- **Enhanced UX**: Rich progress bar and colorful, informative feedback.\r\n- **Optional GUI**: Simple and intuitive web-based interface.\r\n\r\n---\r\n\r\n## 🛠️ Technology Stack\r\n\r\n| Layer         | Technologies             |\r\n| ------------- | ------------------------ |\r\n| Backend \u0026 CLI | Python 3, Typer, Rich    |\r\n| GUI           | Eel, HTML5, Tailwind CSS |\r\n| Testing       | Pytest                   |\r\n\r\n---\r\n\r\n## 🚀 Installation\r\n\r\n### Method 1: End Users (via `pipx`) — **Recommended**\r\n\r\n1. Install `pipx` (if not already installed):\r\n\r\n   ```sh\r\n   pip install pipx\r\n   ```\r\n\r\n2. Ensure `pipx` is in your system's PATH:\r\n\r\n   ```sh\r\n   pipx ensurepath\r\n   ```\r\n\r\n3. Install directly from GitHub:\r\n   - **CLI + GUI**:\r\n\r\n     ```sh\r\n     pipx install \"git+https://github.com/L0g0rhythm/url-refiner.git#egg=url-refiner[gui]\"\r\n     ```\r\n\r\n   - **CLI only**:\r\n\r\n     ```sh\r\n     pipx install \"git+https://github.com/L0g0rhythm/url-refiner.git\"\r\n     ```\r\n\r\nAfter installation, the commands `url-refiner` and `url-refiner-gui` will be available globally.\r\n\r\n---\r\n\r\n### Method 2: Developers\r\n\r\n1. Clone the repository:\r\n\r\n   ```sh\r\n   git clone https://github.com/L0g0rhythm/url-refiner.git\r\n   cd url-refiner\r\n   ```\r\n\r\n2. Create and activate a virtual environment:\r\n   - **Windows**:\r\n\r\n     ```sh\r\n     python -m venv .venv\r\n     .\\.venv\\Scripts\\activate\r\n     ```\r\n\r\n   - **macOS/Linux**:\r\n\r\n     ```sh\r\n     python3 -m venv .venv\r\n     source .venv/bin/activate\r\n     ```\r\n\r\n3. Install with development and GUI dependencies:\r\n\r\n   ```sh\r\n   pip install -e \".[gui,dev]\"\r\n   ```\r\n\r\n---\r\n\r\n## 🖥️ Usage\r\n\r\n### 🔧 CLI (Command-Line Interface)\r\n\r\nPlace your input files inside the `Inputs/` directory.\r\n\r\n- **Read file and print to console**:\r\n\r\n  ```sh\r\n  url-refiner --input urls.txt\r\n  ```\r\n\r\n- **Save output to a timestamped file**:\r\n\r\n  ```sh\r\n  url-refiner --input urls.txt --output\r\n  ```\r\n\r\n- **Use stdin piping**:\r\n  - **Windows**:\r\n\r\n    ```sh\r\n    type Inputs\\urls.txt | url-refiner\r\n    ```\r\n\r\n  - **macOS/Linux**:\r\n\r\n    ```sh\r\n    cat Inputs/urls.txt | url-refiner\r\n    ```\r\n\r\n- **Advanced options**:\r\n  - Append instead of replacing:\r\n\r\n    ```sh\r\n    url-refiner --input urls.txt --mode append\r\n    ```\r\n\r\n  - Use a custom value:\r\n\r\n    ```sh\r\n    url-refiner --input urls.txt --value \"L0g0rhythm\"\r\n    ```\r\n\r\n  - Exclude specific parameters:\r\n\r\n    ```sh\r\n    url-refiner --input urls.txt --exclude id --exclude token\r\n    ```\r\n\r\n  - Ignore URL path for strict deduplication:\r\n\r\n    ```sh\r\n    url-refiner --input urls.txt --ignore-path\r\n    ```\r\n\r\n---\r\n\r\n### 🖱️ GUI (Graphical User Interface)\r\n\r\nFor an interactive experience:\r\n\r\n1. Launch the GUI:\r\n\r\n   ```sh\r\n   url-refiner-gui\r\n   ```\r\n\r\n2. A browser window will open automatically.\r\n\r\n3. Use the interface to:\r\n   - Paste your list of URLs on the left.\r\n   - Configure options like mode, value, exclusions.\r\n   - Click **Process** to see results and stats.\r\n\r\n---\r\n\r\n## ✅ Tests\r\n\r\nRun the full test suite from the root directory:\r\n\r\n```sh\r\npytest\r\n```\r\n\r\n---\r\n\r\n## 📄 License\r\n\r\nThis project is licensed under the MIT License. See the `LICENSE` file for full details.\r\n\r\n---\r\n\r\n## 📬 Contact\r\n\r\n**Victor Oliveira (L0g0rhythm)**\r\n🔗 Website: [l0g0rhythm.com.br](https://l0g0rhythm.com.br)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fl0g0rhythm%2Furl-refiner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fl0g0rhythm%2Furl-refiner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fl0g0rhythm%2Furl-refiner/lists"}