{"id":13530695,"url":"https://github.com/spectralops/netz","last_synced_at":"2025-04-06T22:10:59.492Z","repository":{"id":108930950,"uuid":"344230507","full_name":"SpectralOps/netz","owner":"SpectralOps","description":"Discover internet-wide misconfigurations while drinking coffee","archived":false,"fork":false,"pushed_at":"2021-05-11T13:00:24.000Z","size":92,"stargazers_count":391,"open_issues_count":1,"forks_count":46,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-30T21:08:33.445Z","etag":null,"topics":["cybersecurity","go","golang","osint","scanner"],"latest_commit_sha":null,"homepage":"https://github.com/spectralops/netz","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpectralOps.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-03-03T18:47:21.000Z","updated_at":"2025-03-24T07:35:23.000Z","dependencies_parsed_at":"2023-07-09T11:47:22.924Z","dependency_job_id":null,"html_url":"https://github.com/SpectralOps/netz","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpectralOps%2Fnetz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpectralOps%2Fnetz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpectralOps%2Fnetz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpectralOps%2Fnetz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpectralOps","download_url":"https://codeload.github.com/SpectralOps/netz/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247557767,"owners_count":20958047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","go","golang","osint","scanner"],"created_at":"2024-08-01T07:00:53.672Z","updated_at":"2025-04-06T22:10:59.474Z","avatar_url":"https://github.com/SpectralOps.png","language":"Go","readme":"\u003cp align=\"center\"\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n   \u003cimg src=\"media/netz-logo.png\" width=\"330\"/\u003e\n\u003cbr/\u003e\n\u003cbr/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cb\u003e:bomb: Discover internet-wide misconfigurations\u003c/b\u003e\n\u003cbr/\u003e\n\u003cb\u003e:ok_hand: Verify your assets are not blindly open\u003c/b\u003e\n\u003cbr/\u003e\n\u003chr/\u003e\n\u003c/p\u003e\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)\n\n\n# netz :globe_with_meridians::eagle:\n\nThe purpose of this project is to discover an internet-wide misconfiguration of network components like web-servers/databases/cache-services and more.\nThe basic use-case for such misconfiguration - a service that is publicly exposed to the world without a credentials `¯\\_(ツ)_/¯`     \n\nYou probably familiar with tools like [Shodan](https://www.shodan.io/), [Censys](https://censys.io/), [ZoomEye](https://www.zoomeye.org/) to query such wide internet components,  \nbut here we are going to do it in a fun way :: by hands :D  \n\nThe tools we are going to use are [masscan](https://github.com/robertdavidgraham/masscan), and [zgrab2](https://github.com/zmap/zgrab2) from [ZMap](https://zmap.io/) project. For the first phase of port scanning, we will use [masscan](https://github.com/robertdavidgraham/masscan), then for the second phase, we will run [zgrab2](https://github.com/zmap/zgrab2) to check applicative access for those ports.\n\n[ZMap](https://github.com/zmap/zmap) is also internet-wide scanner, so why [masscan](https://github.com/robertdavidgraham/masscan) and not [ZMap](https://github.com/zmap/zmap)..?\nbecause we want to go wild and use kernel module [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) to get blazing fast packets-per-second to scan the entire internet in minutes, \nand [ZMap](https://github.com/zmap/zmap) basically does support it in the past, but now [ZMap](https://github.com/zmap/zmap) doesn't compatible with the latest [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/).\n\nNote that [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) requires a license per MAC/NIC (you can run 5 minutes in demo before it will kill the flow), and you need a special NIC from Intel (don't worry, the public cloud has such) so you can go without this module, and pay on time to wait for results.\n\nThere are few options to run this project:\n\n1. Use netz cloud runner tool - this tool automate the full pipeline, including infrastructure on top of AWS\n2. Run by yourself using docker \n3. For [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) run by yourself the infrastructure and using [pf_ring setup](pf_ring/configure_pf_ring.sh)\n\nIf you want to read more about it, you can found it here: [Scan the whole internet while drinking coffee](https://cmpxchg16.medium.com/scan-the-whole-internet-while-drinking-coffee-9c4085539594)\n\n## TL;DR\n\nIn [discover.sh](docker/discover.sh) you will find a test for [Elasticsearch](https://www.elastic.co/).  \nThe flow is: \n* run [masscan](https://github.com/robertdavidgraham/masscan) on the entire internet for port 9200 ([Elasticsearch](https://www.elastic.co/) port)\n* pipe ip list from step 1 into [zgrab2](https://github.com/zmap/zgrab2) (you can change with `ZGRAB2_ENDPOINT` environment variable for any [Elasticsearch](https://www.elastic.co/) API Endpoint, for instance: `/_cat/indices`  \n* extract with [jq](https://stedolan.github.io/jq/) just those ip's that return HTTP 200 OK and include `lucene_version`  \n\nThis flow result is ips' that has internet access to [Elasticsearch](https://www.elastic.co/) without credentials.    \n\nThis test flow demonstrates [Elasticsearch](https://www.elastic.co/) scan. You can run such scans on any port (service port) you wish and on any supported protocol by [zgrab2 modules](https://github.com/zmap/zgrab2/tree/master/modules). Environment variables can modify more control:      \n`PORT_TO_SCAN`  \n`SUBNET_TO_SCAN`  \n`ZGRAB2_ENDPOINT`\n\nIn case you wish to add a missing protocol, you can extend [zgrab2](https://github.com/zmap/zgrab2) by [adding new protocols](https://github.com/zmap/zgrab2#adding-new-protocols.)  \n\n\nWe will go through a setup to be faster and faster (decreasing the time to wait).\n\n## Let's Go :rocket:\n## 1. netz cloud runner tool\nThis is the easiest option as it automates everything in AWS on top of Elastic Container Service (ECS).  \nWhat it does:  \n\n* Create IAM role for the pipeline\n* Put IAM Policy\n* Create Instance Profile\n* Associate IAM role to Instance Profile\n* Create Temporary ECS Cluster\n* Create EC2 instance (instance type based on user input `--instance-type`)\n* Create a number of Network Interfaces (number based on user input `--number-of-nic`)\n* Create Public Elastic IP (number based on user input `--number-of-nic`)\n* Associate Elastic IP with Network Interface (for each user input `--number-of-nic`)\n* Run ECS task with the scanning pipeline\n* Create CloudWatch log group and stream the pipeline docker output into the user terminal\n* Destroying all AWS resources\n* Done\n\n## How to run  \nConfigure AWS credentials, you can do it by `~/.aws/credentials`,  \nor by settings environment variables:  \n`AWS_REGION`  \n`AWS_ACCESS_KEY_ID`  \n`AWS_SECRET_ACCESS_KEY`  \n\n[Install Golang 1.14 +](https://golang.org/dl/)  \n\n```\n$ netz\nNAME:\n   netz - netz cloud runner\n\nUSAGE:\n   netz [options]\n\nCOMMANDS:\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --debug                        Show debugging information (default: false)\n   --file value                   Task definition file in JSON or YAML\n   --cluster value                ECS cluster name (default: \"netz\")\n   --log-group value              Cloudwatch Log Group Name to write logs to (default: \"netz-runner\")\n   --security-group value         Security groups to launch task. Can be specified multiple times\n   --subnet value                 Subnet to launch task.\n   --region value                 AWS Region\n   --number-of-nic value          Number of network interfaces to create and attach to instance. (default: 0)\n   --instance-type value          Instance type.\n   --instance-key-name value      Instance key name to for ssh.\n   --role-name value              Role name for netz. (default: \"netzRole\")\n   --role-policy-name value       Role policy name for netz. (default: \"netzPolicy\")\n   --instance-profile-name value  Instance profile name to attach to instance. (default: \"netzInstanceProfile\")\n   --task-timeout value           Task timeout (in minutes), stop everything after that. (default: 120)\n   --skip-destroy                 Skip destroy of cloud resources when done. (default: false)\n   --help, -h                     show help (default: false)\nRequired flags \"file, security-group, subnet, region, number-of-nic, instance-type, instance-key-name\"\n```\n\n### Example\n```\n$ netz --file taskdefinition.json --security-group sg-XXXXXXXXXXXXXXXXXX --subnet subnet-XXXXXXXX --region us-west-1 --debug --number-of-nic 5 --instance-type c4.8xlarge --instance-key-name XXXXXXXXX\n```\n\n:warning:    \n**Because masscan meltdown the network, SSH mostly will not be available, also CloudWatch logs will be deferred, so the tailed logs in user terminal will take some time.**  \n\nNote that [taskdefinition.json](taskdefinition.json) is related to running with the automated way with AWS ECS.  \nIn that file, you will be able to change the subnet \u0026 port to scan, also the application endpoint.  \nIn this file, you can also control the CPU \u0026 RAM you allocate to the task. This test assumed c4.8xlarge, so the config is `60 x cpu` and `36 GB RAM`.  \n\n### Result\nOn AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS =\u003e took 25 minutes  \n\n## 2. Run by yourself using docker\n\n### 2.1 Basic\n#### Run with Docker on basic computer/NIC\n##### Steps\n```bash\n$ git clone https://github.com/SpectralOps/netz\n$ cd netz/docker/\n$ docker build -t netz .\n$ docker run -e PORT_TO_SCAN='80' -e SUBNET_TO_SCAN='216.239.38.21/32' -e ZGRAB2_ENDPOINT='/' -e TASK_DEFINITION='docker' -v /tmp/:/opt/out --network=host -it netz\n```\n:warning:    \n**The time to scrape the entire internet with simple hardware and simple internet backbone could take days**\n\n\n### 3. Faster :zap:\n#### Run with Docker on Cloud with one 10gbps NIC\n\nRun instance with one 10gbps NIC (e.g. in AWS c4.8xlarge [already configured with])  \n\nSteps are the same as [2.1 Basic](https://github.com/SpectralOps/netz#21-basic).\n\n### Result\nOn AWS with c4.8xlarge ~ 700k ~ 950k PPS =\u003e took 2.5 hours.\n\n### 4. Faster++ :zap::dizzy:\n#### Run with Docker on Cloud with multiple 10gbps NIC (e.g. in AWS c4.8xlarge 10gbps NIC )\n* Run in AWS c4.8xlarge Ubuntu 18.04 and connect multiple NIC ([ENI's](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html))\n* For each NIC you need to configure the OS to see those new NIC's.\n\nEdit the netplan file: \n`vim /etc/netplan/50-cloud-init.yaml`\n\nNow it has one NIC:\n\n```yaml\nnetwork:\n    version: 2\n    ethernets:\n        ens3:\n            dhcp4: true\n            match:\n                macaddress: 06:XX:XX:XX:XX:XX\n            set-name: ens3\n```\n\nYou need to add the second, the third and so on...\n```yaml\nnetwork:\n    version: 2\n    ethernets:\n        ens3:\n            dhcp4: true\n            match:\n                macaddress: 03:XX:XX:XX:XX:XX\n            set-name: ens3\n        ens4:\n            dhcp4: true\n            match:\n                macaddress: 04:XX:XX:XX:XX:XX\n            set-name: ens4\n        ens5:\n            dhcp4: true\n            match:\n                macaddress: 05:XX:XX:XX:XX:XX\n            set-name: ens5\n        ens6:\n            dhcp4: true\n            match:\n                macaddress: 06:XX:XX:XX:XX:XX\n            set-name: ens6\n        ens7:\n            dhcp4: true\n            match:\n                macaddress: 07:XX:XX:XX:XX:XX\n            set-name: ens7\n```\n\nApply network configuration: `sudo netplan --debug apply`  \n\nSteps are the same as [2.1 Basic](https://github.com/SpectralOps/netz#21-basic).\n\nNote that now with multiple NICs, the [masscan](https://github.com/robertdavidgraham/masscan) configuration that will be created in `docker run` will contain all NICs:  \n\n\ne.g masscan.conf:  \n\n```bash\nadapter[0] = ens3\nrouter-mac[0] = 06:XX:XX:XX:XX:XX\nadapter-ip[0] = 172.31.8.167\nadapter-mac[0] = 06:YY:YY:YY:YY:YY\nadapter[1] = ens4\nrouter-mac[1] = 06:XX:XX:XX:XX:XX\nadapter-ip[1] = 172.31.8.76\nadapter-mac[1] = 06:YY:YY:YY:YY:YY\nadapter[2] = ens5\nrouter-mac[2] = 06:XX:XX:XX:XX:XX\nadapter-ip[2] = 172.31.1.233\nadapter-mac[2] = 06:YY:YY:YY:YY:YY\n```\n\n### Result\nOn AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS =\u003e took 35 minutes\n\n\n### 5. Faster++++ :zap::dizzy::tornado:\n#### Run on Cloud with 10gbps NIC with [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/)\nIn case you want to scrape the internet in a few minutes with [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/), you will need to run a machine that supports the kernel device drivers and a machine that has 10gbps NIC.  \n\nNotes:\n* Because [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) bypasses the TCP stack, so in case you have just one NIC ens3 and you will open it with **zc**:enc3, you will lose SSH access. If you still want SSH access, you will need another NIC, e.g. ens4, then open ens4 with **zc**, so it will be **zc**:ens4, so ens3 will continue as management NIC for SSH.\n* If you run a machine with 1gbps NIC, it will still be fast, but it will take **x10** more time you could `¯\\_(ツ)_/¯`\n* You don't have to run such a machine like c4.8xlarge, you can run each machine that supports the ixgbevf  \nfrom: [enhanced networking with the Intel 82599 VF interface](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sriov-networking.html)\n\n\n#### Steps\n\n```bash\n$ git clone https://github.com/SpectralOps/netz\n$ cd netz\n```\n\nEdit masscan.conf -- **important** look that now the adapter prefix is **zc:**:\n\n```bash\nadapter[0] = zc:ens4\nrouter-mac[0] = 06:XX:XX:XX:XX:XX\nadapter-ip[0] = 172.31.8.167\nadapter-mac[0] = 06:YY:YY:YY:YY:YY\n```\n\nThe `adapter-ip` and `adapter-mac` you can get from the command: `ifconfig`  \nThe `adapter-mac` you can get from the command: `arp -a`  \n\nRun [configure_pf_ring.sh](configure_pf_ring.sh)  \n\n**Before** the kernel module kicked in - this should be the state:\n\n```bash\n$ sudo pf_ringcfg --list-interfaces\nName: ens3                 Driver: ixgbevf    [Supported by ZC]\nName: docker0              Driver: bridge\nName: ens6                 Driver: ixgbevf    [Supported by ZC]\nName: ens5                 Driver: ixgbevf    [Supported by ZC]\nName: ens4                 Driver: ixgbevf    [Supported by ZC]\n```\n\n**After** the kernel module kicked in - this should be the state:\n\n```bash\n$ sudo pf_ringcfg --list-interfaces\nName: ens3                 Driver: ixgbevf    [Running ZC]\nName: docker0              Driver: bridge\nName: ens6                 Driver: ixgbevf    [Running ZC]\nName: ens5                 Driver: ixgbevf    [Running ZC]\nName: ens4                 Driver: ixgbevf    [Running ZC]\n```\n\n### Run scan:\n\n```bash\nPORT_TO_SCAN='9200' SUBNET_TO_SCAN='0.0.0.0/0' ZGRAB2_ENDPOINT='/' TASK_DEFINITION='docker' bash -x discover.sh\n```\n\n\n### Result\nOn AWS with c4.8xlarge with 4 x NIC ~ 10.5M ~ 12M PPS =\u003e took 10 minutes\n\n\n# Disclaimer\nOur main drive in life is to make the world a better and safer place. If you would like to use this information to harm someone, you are doing the opposite, and at your own risk.    \n\n\n# Copyright\n\nCopyright (c) 2020 [Uri Shamay](http://cmpxchg16.me) [@cmpxchg16](http://twitter.com/cmpxchg16). See [LICENSE](LICENSE.txt) for further details.\n","funding_links":[],"categories":["Tools","Network","2. [↑](#-content) Pentesting"],"sub_categories":["Binary files examination and editing","Scanning / Pentesting","Dynamic Analysis","2.6 [↑](#-content) Network","Network Vulnerability Scanners"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspectralops%2Fnetz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspectralops%2Fnetz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspectralops%2Fnetz/lists"}