{"id":26171797,"url":"https://github.com/zebbern/reconx","last_synced_at":"2025-07-03T21:38:30.846Z","repository":{"id":275915949,"uuid":"927602553","full_name":"zebbern/ReconX","owner":"zebbern","description":"🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!","archived":false,"fork":false,"pushed_at":"2025-02-20T16:36:54.000Z","size":59,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T16:51:09.370Z","etag":null,"topics":["crawler","hacking","information-gathering","information-retrieval","information-security","livedata","opsec","osint","osint-tool","pentest","python","python-crawler","search-engine","security","security-tools","website","website-crawler","website-scraper","website-security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zebbern.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-05T08:26:51.000Z","updated_at":"2025-02-20T16:36:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"66886ed7-2ed3-4817-9b4e-f7ff5c8d0536","html_url":"https://github.com/zebbern/ReconX","commit_stats":null,"previous_names":["zebbern/reconx"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebbern%2FReconX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebbern%2FReconX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebbern%2FReconX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zebbern%2FReconX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zebbern","download_url":"https://codeload.github.com/zebbern/ReconX/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248952344,"owners_count":21188427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","hacking","information-gathering","information-retrieval","information-security","livedata","opsec","osint","osint-tool","pentest","python","python-crawler","search-engine","security","security-tools","website","website-crawler","website-scraper","website-security"],"created_at":"2025-03-11T19:51:24.064Z","updated_at":"2025-04-14T20:21:18.761Z","avatar_url":"https://github.com/zebbern.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n    \n\u003cimg src=\"https://github.com/user-attachments/assets/b41f4195-fd20-4005-90d6-78598c5a79f8\" style=\"width:45%;\"\u003e\n\n![Python](https://img.shields.io/badge/Python-3.x-blue)\n![Status](https://img.shields.io/badge/Status-Active-green)\n![License](https://img.shields.io/badge/License-MIT-brightgreen)\n\n\n**ReconX is tool for crawling websites to gather OSINT data and identify potential vulnerabilities using customizable text-based patterns. It leverages a breadth-first search (BFS) strategy with multithreading for fast and efficient crawling.**\n\n [Features](#features)\n \n [Setup](#setup)\n \n [Usage](#usage)\n \n [Showcase](#showcase)\n \n [Python 3.12+ Pip Fix](#python-312-pip-fix)\n\n---\n\n\u003c/div\u003e\n\n## Features\n\n- **OSINT Gathering:** Extracts emails, phone numbers, subdomains, social links, and more.\n- **Vulnerability Scanning:** Detects potential vulnerabilities by matching text patterns.\n- **Customizable Patterns:** Uses external `.txt` files (in a `wordlists` folder) for scanning.\n- **Concurrent Crawling:** Uses multithreading to speed up the crawling process.\n- **Optional Screenshots:** Capture page screenshots using Selenium and Chrome WebDriver.\n\n---\n\n## Setup\n\n1. **Clone or Download** `git clone https://github.com/zebbern/ReconX`\n2. **Install requirements** `pip install -r requirements.txt` or `pip install requests beautifulsoup4 pyyaml selenium`\n3. **Configuration:**\n   - (Optional) Change the `config.yaml` in the directory with settings you want or keep them default.\n   - (Optional) Adjust crawler settings like user agents, SSL verification, and screenshot options.\n4. **Pattern Files:**\n   - Ensure the `wordlists` directory contains the following files:\n     - `query_params.txt`\n     - `api_wss.txt`\n     - `common_misconfig.txt`\n     - `injection_points.txt`\n     - `sensitive_data.txt`\n     - `exposed_files.txt`\n     - `js_vulnerabilities.txt`\n   - customise info in the .txt found in `/wordlists` folder.\n  \n\n## Usage\n\n**Run the crawler by providing the base URL:**\n\n    python crawler.py -u https://example.com\n\n### Terminal Options\n\n- `-u`, `--url`  \n  **Description:** Base URL to start crawling (required).\n\n- `-t`, `--threads`  \n  **Description:** Number of concurrent threads to use (default: 10).\n\n- `-ws`  \n  **Description:** Enable screenshot capture for the crawled pages.\n\n**For all options on you can run**\n\n    #Example:\n    python crawler.py -u https://example.com -ws -t 10 \n    \n**Explanation:**\n- `-u https://example.com`: Specifies the base URL to crawl.\n- `-ws`: Enables screenshots of fetched pages.\n- `-t 10`: Uses 10 concurrent threads for faster crawling.\n---\n\n### Output\n\nUpon completion, the tool generates two JSON files:\n\n- **recon_data.json:** Contains collected OSINT data (e.g., URLs, emails, subdomains).\n- **Vuln.json:** Contains details of detected vulnerabilities based on the pattern matches.\n\nIf screenshots are enabled (`-ws`), they will be saved in the directory specified by `config.yaml -\u003e output_path: \"\"` (default: `webshots`).\n\n---\n\n## Showcase\n`Command Used`: python ReconX.py -u https://monolittenil.no | Use `-ws` if you want to include pictures of each site its crawled can also edit `config.yaml` to crawl deeper.\n\n\u003cimg width=\"60%\" src=\"https://github.com/user-attachments/assets/b929cf1c-6d77-443f-b8c6-5c7abdd66168\"\u003e\n\n\n\u003chr\u003e\n\n## Python 3.12+ Pip Fix:\n### Create and Activate a Virtual Environment\n#### For Linux/macOS:\n```\npython3 -m venv venv \u0026\u0026 source venv/bin/activate\n```\n#### For Windows:\n```\npython -m venv venv \u0026\u0026 .\\venv\\Scripts\\activate\n```\n\n\n\u003e [!WARNING]  \n\u003e These is intended for educational and ethical hacking purposes only. It should only be used to test systems you own or have explicit permission to test. Unauthorized use of third-party websites or systems without consent is illegal and unethical.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebbern%2Freconx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzebbern%2Freconx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzebbern%2Freconx/lists"}