{"id":13845615,"url":"https://github.com/EONRaider/Web-Probe","last_synced_at":"2025-07-12T03:31:25.956Z","repository":{"id":48077940,"uuid":"391050768","full_name":"EONRaider/Web-Probe","owner":"EONRaider","description":"Web Probe is a Python 3 asynchronous port scanner with the purpose of checking live web servers","archived":false,"fork":false,"pushed_at":"2023-02-16T04:14:26.000Z","size":9793,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-14T13:14:59.793Z","etag":null,"topics":["ethical-hacking","hacking","network-programming","pentesting-tools","port-scanner","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EONRaider.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-30T12:07:08.000Z","updated_at":"2024-05-31T02:15:56.000Z","dependencies_parsed_at":"2022-08-12T18:10:16.086Z","dependency_job_id":null,"html_url":"https://github.com/EONRaider/Web-Probe","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EONRaider%2FWeb-Probe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EONRaider%2FWeb-Probe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EONRaider%2FWeb-Probe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EONRaider%2FWeb-Probe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EONRaider","download_url":"https://codeload.github.com/EONRaider/Web-Probe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225791368,"owners_count":17524771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ethical-hacking","hacking","network-programming","pentesting-tools","port-scanner","security-tools"],"created_at":"2024-08-04T17:03:30.354Z","updated_at":"2024-11-21T19:30:22.555Z","avatar_url":"https://github.com/EONRaider.png","language":"Python","funding_links":["https://paypal.com","http://paypal.com"],"categories":["Python"],"sub_categories":[],"readme":"# Web Probe\n\n![Python Version](https://img.shields.io/badge/python-3.7+-blue?style=for-the-badge\u0026logo=python)\n![OS](https://img.shields.io/badge/GNU%2FLinux-red?style=for-the-badge\u0026logo=linux)\n![OS](https://img.shields.io/badge/mac%20OS-gray?style=for-the-badge\u0026logo=apple)\n![OS](https://img.shields.io/badge/Windows-blue?style=for-the-badge\u0026logo=windows)\n\n![GitHub Workflow Status](https://img.shields.io/github/workflow/status/eonraider/web-probe/run-pytest?label=TESTS\u0026style=for-the-badge)\n![GitHub Workflow Status](https://img.shields.io/github/workflow/status/eonraider/Web-Probe/ci-to-dockerhub?style=for-the-badge)\n[![CodeFactor Grade](https://img.shields.io/codefactor/grade/github/EONRaider/web-probe?style=for-the-badge)](https://www.codefactor.io/repository/github/EONRaider/web-probe)\n[![License](https://img.shields.io/github/license/EONRaider/Packet-Sniffer?style=for-the-badge)](https://github.com/EONRaider/Packet-Sniffer/blob/master/LICENSE)\n\n[![Reddit](https://img.shields.io/badge/Reddit-EONRaider-FF4500?style=flat-square\u0026logo=reddit)](https://www.reddit.com/user/eonraider)\n[![Discord](https://img.shields.io/badge/Discord-EONRaider-7289DA?style=flat-square\u0026logo=discord)](https://discord.gg/KVjWBptv)\n[![Twitter](https://img.shields.io/badge/Twitter-eon__raider-38A1F3?style=flat-square\u0026logo=twitter)](https://twitter.com/intent/follow?screen_name=eon_raider)\n\nWeb Probe is a Python 3 asynchronous port scanner with the purpose of \nchecking live web servers. It uses Python's Standard Library `asyncio` \nframework to create TCP connections to an arbitrary number of ports on target IP \naddresses and/or domain names. It can probe an unlimited number of hosts\nsimultaneously, **effectively scanning thousands of hosts within just a few\nseconds.**\n\nFurthermore, WebProbe optionally fetches response headers from each target\nand parses them all, writing a file that displays a frequency analysis for \nthose headers. This is useful for finding *unusual headers* that may \nindicate the presence of a given system, infrastructure signature or \nvulnerability within a large set of targets.\n\nThis application can be run by any Python v3.7+ interpreter, by a system \nrunning **Docker** or, alternatively, as a **standalone executable** on \n64-bit GNU/Linux, Apple macOS or Microsoft Windows. In fact, you don't \neven need Python installed on your local environment to run WebProbe.\n\n## Installation\n\n### From a standalone, multi-platform executable\nWebProbe can be run as a **multi-platform** executable thanks to \n[PyInstaller](https://github.com/pyinstaller/pyinstaller).\n\nSimply [click here](https://github.com/EONRaider/Web-Probe/raw/master/dist/webprobe)\nto download the `webprobe` file from the \n[dist directory](https://github.com/EONRaider/Web-Probe/blob/master/dist/webprobe).\nThen just grant it permissions to execute in the local context and run it as \ndescribed in the following [Usage](#usage) section.\n\n- **Use it as a command**: On GNU/Linux or macOS, either \ndownload the `webprobe` executable or create a symbolic\nlink to it in a convenient directory listed in the `$PATH` environment \nvariable to have `WebProbe` set up as a command in your local system. Take\na look [here](https://stackoverflow.com/a/29235240) if you need help setting\nthis up.\n\n### From a Docker image\nPull the image from DockerHub and check the help prompt with a single\ncommand:\n```\nuser@host:~$ docker run -it eonraider/webprobe --help\n```\n\n- **Use it as a command**: There are several ways of doing this on GNU/Linux,\nbut one of the quickest is to set up the `docker run` command as a shell alias:\n    ```\n    user@host:~$ echo \"alias webprobe='docker run -it eonraider/webprobe'\" \u003e\u003e /home/$USER/.bash_aliases\n    user@host:~$ source ~/.bashrc\n    user@host:~$ webprobe --help\n    (...will download and run the image from DockerHub...)\n    ```\n\n### From a Virtual Environment\nBetter suited for development and eventual contributions to the project:\n```\nuser@host:~$ git clone https://github.com/EONRaider/Web-Probe\nuser@host:~$ cd Web-Probe\nuser@host:~/Web-Probe$ python3 -m venv venv\n(venv) user@host:~/Web-Probe$ source venv/bin/activate\n(venv) user@host:~/Web-Probe$ pip install -r dev-requirements.txt\n```\n\n## Usage\n```\nusage: webprobe.py [-h] (-t ADDRESSES | --version) [-p PORTS]\n                   [--timeout SECONDS] [--prefer-https] [--rebind MAP]\n                   [--silent] [-o FILE_PATH] [--headers DIR_PATH]\n                   [--header-analysis FILE_PATH]\n\nWebProbe: Asynchronous TCP port scanner for live web hosts\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -t ADDRESSES, --targets ADDRESSES\n                        An absolute path to a valid file with line-separated targets, a single target name or a comma-separated sequence of targets to probe, e.g., '45.33.32.156,65.61.137.117,testphp.vulnweb.com'\n  --version             Display the current version for WebProbe.\n  -p PORTS, --ports PORTS\n                        A comma-separated sequence of port numbers and/or port ranges to scan on each target specified, e.g., '20-25,53,80,443'.\n  --timeout SECONDS     Time to wait for a response from a target before closing a connection (defaults to 5 seconds).\n  --prefer-https        Omit performing requests with the HTTP URI scheme for those servers that also respond with HTTPS (defaults to False).\n  --rebind MAP          Allows ports other than 80 and 443 to be assigned to HTTP and HTTPS, respectively. Takes input with the syntax '8080:http' or '8080:http,9900:https'. Defaults to standard port bindings 80:HTTP and 443:HTTPS.\n  --silent              Suppress displaying results to STDOUT.\n  -o FILE_PATH, --output FILE_PATH\n                        Absolute path to a file in which to write results of probing each web host.\n  --headers DIR_PATH    Absolute path to a directory in which to write files with the response headers for each probed URL.\n  --header-analysis FILE_PATH\n                        Absolute path to a file in which to write all fetched headers in ascending order of frequency.\n\nUsage examples:\n\t1. python3 webprobe.py -t google.com\n\t2. python3 webprobe.py -t 45.33.32.156,demo.testfire.net,18.192.172.30 -p 443\n\t3. python3 webprobe.py --prefer-https -t uber.com,paypal.com\n\t4. python3 webprobe.py -t unusual-domain.xyz --rebind 1337:https\n\t5. python3 webprobe.py -t /path/to/domains/file.txt\n```\n\n## Why analyse response headers?\n\nA good answer to this question can be found in\n[this livestream](https://youtu.be/SYExiynPEKM?t=940) from\n[Nahamsec](https://www.youtube.com/channel/UCCZDt7MuC3Hzs6IH4xODLBw)\nYouTube channel, but basically an analysis of response headers allows you\nto find unusual response patterns when performing recon on a target\n(especially in a bug bounty hunting environment). Take a look at the\n[sample header analysis](https://github.com/EONRaider/Web-Probe/blob/master/tests/support_files/webprobe-uber.com.head.analysis.txt)\nfor the uber.com domain included in this project's support files.\n\n\n## Usage Examples\n\n\u003cdetails\u003e\n\u003csummary\u003eView the help prompt\u003c/summary\u003e\n\n```\nuser@host:~$ webprobe --help\nusage: webprobe.py [-h] -t ADDRESSES [-p PORTS] [--timeout SECONDS]\n                   [--prefer-https] [--rebind MAP] [--silent] [-o FILE_PATH]\n                   [--headers DIR_PATH] [--header-analysis FILE_PATH]\n                   \nWebProbe: Asynchronous TCP port scanner for live web hosts\n(...snip...)\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eProbe a single domain\u003c/summary\u003e\n\n```\nuser@host:~$ webprobe --targets google.com\nhttps://google.com\nhttp://google.com\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eProbe multiple domains on a single port number from the CLI\u003c/summary\u003e\n\n```\nuser@host:~$ webprobe -t facebook.com,scanme.nmap.org,instagram.com -p 443\nhttps://facebook.com\nhttps://instagram.com\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eProbe multiple domains from a text file\u003c/summary\u003e\n\n```\nuser@host:~$ cat domains.txt\ngoogle.com\nuber.com\npaypal.com\n\nuser@host:~$ webprobe -t domains.txt\nhttp://google.com\nhttps://google.com\nhttp://uber.com\nhttps://uber.com\nhttps://paypal.com\nhttp://paypal.com\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eProbe a domain and fetch the response headers\u003c/summary\u003e\n\n```\nuser@host:~$ webprobe --targets google.com --headers .\nhttp://google.com\nhttps://google.com\n\nuser@host:~$ cat google.com.head\nhttp://google.com\n    Date: Wed, 04 Aug 2021 20:22:07 GMT\n    Expires: -1\n    Cache-Control: private, max-age=0\n    Content-Type: text/html; charset=ISO-8859-1\n    P3P: CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"\n    Content-Encoding: gzip\n    Server: gws\n    Content-Length: 6144\n    X-XSS-Protection: 0\n    X-Frame-Options: SAMEORIGIN\n    Set-Cookie: 1P_JAR=2021-08-04-20; expires=Fri, 03-Sep-2021 20:22:07 GMT; path=/; domain=.google.com; Secure\n\nhttps://google.com\n    Date: Wed, 04 Aug 2021 20:22:07 GMT\n    Expires: -1\n    Cache-Control: private, max-age=0\n    Content-Type: text/html; charset=ISO-8859-1\n    P3P: CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\"\n    Content-Encoding: gzip\n    Server: gws\n    X-XSS-Protection: 0\n    X-Frame-Options: SAMEORIGIN\n    Set-Cookie: 1P_JAR=2021-08-04-20; expires=Fri, 03-Sep-2021 20:22:07 GMT; path=/; domain=.google.com; Secure\n    Alt-Svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"\n    Transfer-Encoding: chunked\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eProbe 500+ domains from a text file and analyse their\nresponse headers in just 8.5 seconds\u003c/summary\u003e\n\n```\n# Using the tests file from this repository\nuser@host:~$ wc -l tests/support_files/amass-uber.com.txt\n557 tests/support_files/amass-uber.com.txt \u003c-- Number of domains to probe \n\nuser@host:~$ time dist/webprobe --targets tests/support_files/amass-uber.com.txt \\\n--header-analysis ~/Desktop/header-analysis.txt\n\nhttps://get.uber.com\nhttps://wallet.uber.com\nhttps://beta.uber.com\n(...snip...)\nhttps://safetycenter-staging.uber.com\nhttps://dba.usuppliers.uber.com\nhttp://sao2.uber.com\n\nreal\t0m8,558s \u003c-- Total time elapsed\nuser\t0m2,809s\nsys\t0m0,345s\n\nuser@host:~$ cat ~/Desktop/header-analysis.txt\n[CF-Ray]\n\t-\u003e http://investor.uber.com \u003e CF-Ray: 67aa80f63f6df758-GRU\n\n[CF-Cache-Status]\n\t-\u003e http://investor.uber.com \u003e CF-Cache-Status: REVALIDATED\n\n[Expect-CT]\n\t-\u003e http://investor.uber.com \u003e Expect-CT: max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"\n(...snip...)\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eUse WebProbe from Docker image\u003c/summary\u003e\n\nRunning a command for WebProbe using docker is as simple as using \n`docker run -it eonraider/webprobe` followed by the standard arguments \ndescribed in the help prompt. The best way to extract output files \nrelies on creating a volume binding a local system directory to a \nuser-writable directory inside the container, such as `/tmp`.\n```\n# Simple probe from the CLI with results to STDOUT\nuser@host:~$ docker run -it eonraider/webprobe --targets paypal.com\nhttp://paypal.com\nhttps://paypal.com\n\n# Using a volume to extract an output file from the Docker container\nuser@host:~$ docker run -v ~/Desktop:/tmp -it eonraider/webprobe \\\n--targets paypal.com -o /tmp/webprobe-paypal.com.txt\nhttp://paypal.com\nhttps://paypal.com\n\nuser@host:~$ cat ~/Desktop/webprobe-paypal.com.txt\nhttp://paypal.com\nhttps://paypal.com\n```\n\u003c/details\u003e\n\n## Legal Disclaimer\n\nThe use of code contained in this repository, either in part or in its totality,\nfor engaging targets without prior mutual consent is illegal. **It is\nthe end user's responsibility to obey all applicable local, state and \nfederal laws.**\n\nDevelopers assume **no liability** and are not\nresponsible for misuses or damages caused by any code contained\nin this repository in any event that, accidentally or otherwise, it comes to\nbe utilized by a threat agent or unauthorized entity as a means to compromise\nthe security, privacy, confidentiality, integrity, and/or availability of\nsystems and their associated resources by leveraging the exploitation of known\nor unknown vulnerabilities present in said systems, including, but not limited\nto, the implementation of security controls, human- or electronically-enabled.\n\nThe use of this code is **only** endorsed by the developers in those\ncircumstances directly related to **educational environments** or\n**authorized penetration testing engagements** whose declared purpose is that\nof finding and mitigating vulnerabilities in systems, limiting their exposure\nto compromises and exploits employed by malicious agents as defined in their\nrespective threat models.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEONRaider%2FWeb-Probe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEONRaider%2FWeb-Probe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEONRaider%2FWeb-Probe/lists"}