{"id":21972535,"url":"https://github.com/byt3exec/witnessme","last_synced_at":"2025-03-22T23:12:42.317Z","repository":{"id":256373422,"uuid":"855101205","full_name":"byt3exec/WitnessMe","owner":"byt3exec","description":"Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells \u0026 whistles to make life easier. ","archived":false,"fork":false,"pushed_at":"2024-09-10T10:07:38.000Z","size":409,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-28T02:42:16.071Z","etag":null,"topics":["offensive-security","osint","penetration-testing","screenshot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/byt3exec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"byt3bl33d3r","patreon":"byt3bl33d3r","ko_fi":"byt3bl33d3r"}},"created_at":"2024-09-10T10:03:04.000Z","updated_at":"2024-09-10T10:07:41.000Z","dependencies_parsed_at":"2024-09-10T11:34:18.852Z","dependency_job_id":"1b431c53-2254-484a-91ff-cf2865638bb5","html_url":"https://github.com/byt3exec/WitnessMe","commit_stats":null,"previous_names":["emonilo/witnessme","byt3exec/witnessme"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byt3exec%2FWitnessMe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byt3exec%2FWitnessMe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byt3exec%2FWitnessMe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/byt3exec%2FWitnessMe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/byt3exec","download_url":"https://codeload.github.com/byt3exec/WitnessMe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245031517,"owners_count":20549925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["offensive-security","osint","penetration-testing","screenshot"],"created_at":"2024-11-29T15:04:04.046Z","updated_at":"2025-03-22T23:12:42.297Z","avatar_url":"https://github.com/byt3exec.png","language":"Python","funding_links":["https://github.com/sponsors/byt3bl33d3r","https://patreon.com/byt3bl33d3r","https://ko-fi.com/byt3bl33d3r"],"categories":[],"sub_categories":[],"readme":"# WitnessMe\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/5151193/60783062-1f637c00-a106-11e9-83de-83ef88115f74.gif\" alt=\"WitnessMe\"/\u003e\n\u003c/p\u003e\n\nWitnessMe is primarily a Web Inventory tool inspired by [Eyewitness](https://github.com/FortyNorthSecurity/EyeWitness), its also written to be extensible allowing you to create custom functionality that can take advantage of the headless browser it drives in the back-end.\n\nWitnessMe uses the [Pyppeteer](https://github.com/pyppeteer/pyppeteer) library to drive Headless Chromium.\n\n## Table of Contents\n\n- [WitnessMe](#witnessme)\n  * [Motivation](#motivation)\n  * [Official Discord Channel](#official-discord-channel)\n  * [Installation](#Installation)\n    + [Docker](#docker)\n    + [Python Package](#python-package)\n    + [Development Install](#development-install)\n  * [Quick starts](#quick-starts)\n    + [Finding F5 Load Balancers Vulnerable to CVE-2020-5902](#finding-f5-load-balancers-vulnerable-to-cve-2020-5902)\n    + [Scraping Javascript Heavy Webpages](#scraping-javascript-heavy-webpages)\n  * [RESTful API](#restful-api)\n  * [Deploying to the Cloud](#deploying-to-the-cloud-)\n    + [GCP Cloud Run](#gcp-cloud-run)\n    + [AWS ElasticBeanstalk](#aws-elasticbeanstalk)\n  * [Usage and Examples](#usage-and-examples)\n    + [Modes of Operation](#modes-of-operation)\n      * [Screenshot Mode](#screenshot-mode)\n      * [Grab Mode](#grab-mode)\n    + [Interacting with the Scan Database](#interacting-with-the-scan-database)\n    + [Generating Reports](#generating-reports)\n    + [Proxying](#Proxying)\n    + [Previewing Screenshots Directly in the Terminal](#preview-screenshots-directly-in-the-terminal)\n  * [Creating Signatures](#call-for-signatures)\n\n## Motivation\n\nAre there are a bunch of other tools that do this? Absolutely. See the following projects for alternatives (I'm sure there are more, these are the ones I've personally tried):\n\n- [Eyewitness](https://github.com/FortyNorthSecurity/EyeWitness)\n- [GoWitness](https://github.com/sensepost/gowitness)\n- [Aquatone](https://github.com/michenriksen/aquatone)\n\nThe reason why I wrote WitnessMe was that none of these projects had all of the features I wanted/needed in order for them to work well within my workflow. Additionally, some of them are prone to a decent amount of installation/dependency hell.\n\nHere are some of the main features that make WitnessMe \"stand out\":\n\n- Written in Python 3.7+\n- Ability to parse extremely large Nessus and NMap XML files\n- Docker compatible\n- No installation/dependency hell\n- Full test suite! Everything is less prone to bugs\n- CSV \u0026 HTML reporting\n- HTTP Proxy Support\n- Provides a RESTful API! Scan stuff remotely!\n- CLI interface to view and search scan results without having to view the reports.\n- Signature scanning (Signatures use YAML files)\n- Preview screenshots directly in the terminal (On MacOSX/ITerm2 and some Nix terminals)\n- Extensibly written, allowing you to add functionality that can take advantage of headless chromium.\n- Built to be deployed to the Clouds (e.g. GCP Cloud Run , AWS ElasticBeanstalk etc...)\n\n## Installation\n\n### Docker\n\nRunning WitnessMe from a Docker container is fully supported and is the easiest/recommended way of using the tool.\n\n**Note: it is highly recommended to give the Docker container at least 4GB of RAM during large scans as Chromium can be a resource hog. If you keep running into \"Page Crash\" errors, it's because your container does not have enough memory. On Mac/Windows you can change this by clicking the Docker Task Bar Icon -\u003e Preferences -\u003e Resources. For Linux, refer to Docker's documentation**\n\nPull the image from Docker Hub:\n\n```console\ndocker pull emonilo/witnessme\n```\n\nYou can then spin up a docker container, run it like the main `witnessme` script and pass it the same arguments:\n\n```console\ndocker run --rm -ti $IMAGE_ID screenshot https://google.com 192.168.0.1/24\n```\n\nAlternatively, you can drop into a shell within the container and run the tools that way. This also allows you to execute the `wmdb` and `wmapi` scripts.\n\n```console\ndocker run --rm -ti --entrypoint=/bin/sh $IMAGE_ID\n```\n\n### Python Package\n\nWitnessMe is also available as a Python package (Python 3.7 or above is required). If you do install it this way it is extremely recommended to use [pipx](https://github.com/pipxproject/pipx) as it takes care of installing everything in isolated environments for you in a seamless manner.\n\nRun the following commands:\n\n```console\npython3 -m pip install --user pipx\npipx install witnessme\n```\n\nAll of the WitnessMe scripts should now be in your PATH and ready to go.\n\n### Development Install\n\nYou really should only install WitnessMe this way if you intend to hack on the source code. You're going to Python 3.7+ and [Poetry](https://python-poetry.org/): please refer to the Poetry installation documentation in order to install it.\n\n```console\ngit clone https://github.com/emonilo/WitnessMe \u0026\u0026 cd WitnessMe\npoetry install\n```\n\n## Quick Starts\n\n### Finding F5 Load Balancers Vulnerable to CVE-2020-5902\n\nInstall WitnessMe using Docker:\n\n```console\ndocker pull emonilo/witnessme\n```\n\nGet the `$IMAGE_ID` from the `docker images` command output, then run the following command to drop into a shell inside the container. Additionally, specify the `-v` flag to mount the current directory inside the container at the path `/transfer` in order to copy the scan results back to your host machine (if so desired):\n\n```console\ndocker run -it --entrypoint=/bin/sh -v $(pwd):/transfer $IMAGE_ID\n```\n\nScan your network using WitnessMe, it can accept multiple .Nessus files, Nmap XMLs, IP ranges/CIDRs. Example:\n\n```console\nwitnessme screenshot 10.0.1.0/24 192.168.0.1-20 ~/my_nessus_scan.nessus ~/my_nmap_scan.xml\n```\n\nAfter the scan is finished, a folder will have been created in the current directory with the results. Access the results using the `wmdb` command line utility:\n\n```console\nwmdb scan_2020_$TIME/\n```\n\nTo quickly identify F5 load balancers, first perform a signature scan using the `scan` command. Then search for \"BIG-IP\" or \"F5\" using the `servers` command (this will search for the \"BIG-IP\" and \"F5\" string in the signature name, page title and server header):\n\n![image](https://user-images.githubusercontent.com/5151193/86619581-43fc6900-bf91-11ea-9a01-ba8ce09c3f3b.png)\n\nAdditionally, you can generate an HTML or CSV report using the following commands:\n```console\nWMDB ≫ generate_report html\n```\n```console\nWMDB ≫ generate_report csv\n```\n\nYou can then copy the entire scan folder which will contain all of the reports and results to your host machine by copying it to the `/transfer` folder.\n\n### Scraping Javascript Heavy Webpages\n\nAs of v1.5.0, WitnessMe has a `grab` command which allows you to quickly scrape Javascript heavy webpages by rendering the page first with Headless Chromium and then parsing the resulting HTML using the specified XPath (see [here](https://devhints.io/xpath) for an XPath cheatsheet).\n\nBelow are a few examples to get your started.\n\nThis grabs a list of all advertised domains on the `144.161.160.0/23` subnet from [Hurricane Electric's BGP Toolkit](https://bgp.he.net/):\n```console\nwitnessme -d grab -x '//div[@id=\"dns\"]/table//tr/td[2]/a/text()' https://bgp.he.net/net/144.161.160.0/23#_dns\n```\n\n## RESTful API\n\nAs of version 1.0, WitnessMe has a RESTful API which allows you to interact with the tool remotely.\n\n**Note: Currently, the API does not implement any authentication mechanisms. Make sure to allow/deny access at the transport level**\n\nTo start the RESTful API for testing/development purposes run :\n```console\nwmapi\n```\n\nThe API documentation will then be available at http://127.0.0.1:8000/docs\n\n[Uvicorn](https://www.uvicorn.org/) should be used to enable SSL and run the API in production. See [this dockerfile](https://github.com/byt3bl33d3r/WitnessMe/blob/master/dockerfiles/Dockerfile.selfhosted) for an example.\n\n## Deploying to the Cloud (™)\n\nSince WitnessMe has a RESTful API now, you can deploy it to the magical cloud and perform scanning from there. This would have a number of benefits, including giving you a fresh external IP on every scan (More OPSEC safe when assessing attack surface on Red Teams).\n\nThere are a number of ways of doing this, you can obviously do it the traditional way (e.g. spin up a machine, install docker etc..).\n\nRecently cloud service providers started offering ways of running Docker containers directly in a fully managed environment. Think of it as serverless functions (e.g. AWS Lambdas) only with Docker containers.\n\nThis would technically allow you to really quickly deploy and run WitnessMe (or really anything in a Docker container) without having to worry about underlying infrastructure and removes a lot of the security concerns that come with that.\n\nBelow are some of the ones I've tried along with the steps necessary to get it going and any issues I encountered.\n\n### GCP Cloud Run\n\n**Unfortunately, it seems like Cloud Run doesn't allow outbound internet access to containers, if anybody knows of a way to get around this please get in touch**\n\nCloud Run is by far the easiest of these services to work with.\n\nThis repository includes the `cloudbuild.yaml` file necessary to get this setup and running.\n\nFrom the repositories root folder (after you authenticated and setup a project), these two commands will automatically build the Docker image, publish it to the Gcloud Container Registry and deploy a working container to Cloud Run:\n\n```bash\ngcloud builds submit --config cloudbuild.yaml\ngcloud run deploy --image gcr.io/$PROJECT_ID/witnessme --platform managed\n```\n\nThe output will give you a HTTPS url to invoke the WitnessMe RESTful API from :)\n\nWhen you're done:\n\n```bash\ngcloud run services delete witnessme\ngcloud container images delete gcr.io/$PROJECT_ID/witnessme\n```\n\n### AWS ElasticBeanstalk\n\nTO DO\n\n## Usage\n\nThere are 3 main utilities:\n\n- `witnessme`: is the main CLI interface.\n- `wmdb`: allows you to browse the database (created on each scan) to view results and generate reports.\n- `wmapi`: provides a RESTful API to schedule, start, stop and monitor scans.\n\n### Modes of Operations\n\nAs of v1.5.0 there are two main modes (commands) that the `witnessme` utility Supports:\n\n- The `screenshot` command, you guessed it, screenshots webpages. This is the main functionality.\n- The `grab` command allows you to scrape pages and quickly grab server headers.\n\n```\nusage: witnessme [-h] [--threads THREADS] [--timeout TIMEOUT] [-d] [-v] {screenshot,grab} ...\n\nWitnessMe!\n\npositional arguments:\n  {screenshot,grab}\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --threads THREADS  Number of concurrent browser tab(s) to open\n                     [WARNING: This can cause huge RAM consumption if set to high values] (default: 15)\n  --timeout TIMEOUT  Timeout for each connection attempt in seconds (default: 15)\n  -d, --debug        Enable debug output (default: False)\n  -v, --version      show program's version number and exit\n```\n\n#### Screenshot Mode\n\n```console\n$ witnessme screenshot --help\nusage: witnessme screenshot [-h] [-p PORTS [PORTS ...]] target [target ...]\n\npositional arguments:\n  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -p PORTS [PORTS ...], --ports PORTS [PORTS ...]\n                        Ports to scan if IP Range/CIDR is provided\n```\n\nCan accept a mix of .Nessus file(s), Nmap XML file(s), files containing URLs and/or IPs, IP addresses/ranges/CIDRs and URLs or alternatively read from stdin.\n\n*Note: WitnessMe detects .Nessus and NMap files by their extension so make sure Nessus files have a `.nessus` extension and NMap scans have a `.xml` extension*\n\nLong story short, should be able to handle anything you throw at it:\n\n```console\nwitnessme screenshot 192.168.1.0/24 192.168.1.10-20 https://bing.com ~/my_nessus_scan.nessus ~/my_nmap_scan.xml ~/myfilewithURLSandIPs\n```\n\n```console\n$ cat my_domain_list.txt | witnessme screenshot -\n```\n\nIf an IP address/range/CIDR is specified as a target, WitnessMe will attempt to screenshot HTTP \u0026 HTTPS pages on ports 80, 8080, 443, 8443 by default. This is customizable with the `--port` argument.\n\nOnce a scan is completed, a folder with all the screenshots and a database will be in the current directory, point `wmdb` to the folder in order to see the results.\n\n```console\nwmdb scan_2019_11_05_021237/\n```\n#### Grab Mode\n\n```console\n$ witnessme grab --help\nusage: witnessme grab [-h] [-x XPATH | -l] target [target ...]\n\npositional arguments:\n  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -x XPATH, --xpath XPATH\n                        XPath to use\n  -l, --links           Get all links\n```\n\nThe `grab` subcommand allows you to render Javascript heavy webpages and scrape their content using XPaths. See this [section](#scraping-javascript-heavy-webpages) for some examples.\n\n### Interacting with the Scan Database\n\nOnce a scan is completed (using the `screenshot` mode), a folder with all the screenshots and a database will be in the current directory, point `wmdb` to the folder in order to see the results.\n\n```console\nwmdb scan_2019_11_05_021237/\n```\nThis will drop you into the WMDB CLI menu.\n\nPressing tab will show you the available commands and a help menu:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/5151193/88490790-725bdb80-cf74-11ea-8ecd-1300cf1ad534.png\" alt=\"Tab \"/\u003e\n\u003c/p\u003e\n\nThe `servers` and `hosts` commands in the `wmdb` CLI accept 1 argument. WMCLI is smart enough to know what you're trying to do with that argument\n\n#### Server Command\n\nNo arguments will show all discovered servers. Passing it an argument will search the `title` and `server` columns for that pattern (it's case insensitive).\n\nFor example if you wanted to search for all discovered Apache Tomcat servers:\n- `servers tomcat` or `servers 'apache tomcat'`\n\nSimilarly if you wanted to find servers with a 'login' in the title:\n- `servers login`\n\n#### Hosts Command\n\nNo arguments will show all discovered hosts. Passing it an argument will search the `IP` and `Hostname` columns for that pattern (it's case insensitive). If the value corresponds to a Host ID it will show you the host information and all of the servers discovered on that host which is extremely useful for reporting purposes and/or when targeting specific hosts.\n\n#### Signature Scan\n\nYou can perform a signature scan on all discovered services using the `scan` command.\n\n### Generating Reports\n\nYou can use the `generate_report` command in the `wmdb` cli to generate reports in HTML or CSV format. To generate a HTML report simply run `generate_report` without any arguments. Here's an example of what it'll look like:\n\n![image](https://user-images.githubusercontent.com/5151193/86676611-2c44d500-bfd1-11ea-87fd-faf874a2dcf2.png)\n\nTo generate a CSV report:\n\n```console\nWMDB ≫ generate_report csv\n```\n\nThe reports will then be available in the scan folder.\n\n### Proxying\n\nAs of v1.5 WitnessMe supports proxying all of its traffic through an HTTP proxy. Specify a `HTTP_PROXY` environment variable to force the underlying headless browser to proxy its traffic through the desired host:\n\n```console\nHTTP_PROXY=http://127.0.0.1:8080 witnessme screenshot ~/my_targets.txt\n```\n\n```console\nHTTP_PROXY=http://127.0.0.1:8080 witnessme grab https://www.google.com\n```\n\n### Preview Screenshots Directly in the Terminal\n\n**Note: this feature will only work if you're on MacOSX and using ITerm2**\n\nYou can preview screenshots directly in the terminal using the `show` command:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/5151193/68194496-5e012a00-ff72-11e9-9ccd-6a50aa384f3e.png\" alt=\"ScreenPreview\"/\u003e\n\u003c/p\u003e\n\n## Writing Signatures\n\nIf you run into a new webapp write a signature for it! It's beyond simple and they're all in YAML!\n\nDon't believe me? Here's the AirOS signature (you can find them all in the [signatures directory](https://github.com/byt3bl33d3r/WitnessMe/tree/master/witnessme/signatures)):\n\n```yaml\ncredentials:\n- password: ubnt\n  username: ubnt\nname: AirOS\nsignatures:\n- airos_logo.png\n- form enctype=\"multipart/form-data\" id=\"loginform\" method=\"post\"\n- align=\"center\" class=\"loginsubtable\"\n- function onLangChange()\n# AirOS ubnt/ubnt\n```\n\nYup that's it. Just plop it in the signatures folder and POW! Done.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyt3exec%2Fwitnessme","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbyt3exec%2Fwitnessme","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyt3exec%2Fwitnessme/lists"}