Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/m8sec/subscraper
Subdomain and target enumeration tool built for offensive security testing
https://github.com/m8sec/subscraper
bugbounty enumeration osint penetration-testing pentest pentest-tool python3 subdomain-brute subdomain-enumeration subdomain-scanner subdomain-takeover
Last synced: about 1 month ago
JSON representation
Subdomain and target enumeration tool built for offensive security testing
- Host: GitHub
- URL: https://github.com/m8sec/subscraper
- Owner: m8sec
- License: gpl-3.0
- Created: 2018-09-27T18:00:51.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-01-28T13:44:26.000Z (7 months ago)
- Last Synced: 2024-05-20T23:34:28.499Z (4 months ago)
- Topics: bugbounty, enumeration, osint, penetration-testing, pentest, pentest-tool, python3, subdomain-brute, subdomain-enumeration, subdomain-scanner, subdomain-takeover
- Language: Python
- Homepage:
- Size: 96.7 KB
- Stars: 744
- Watchers: 28
- Forks: 94
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- awesome-hacking-lists - m8sec/subscraper - Perform subdomain enumeration through various techniques and retrieve detailed output to aid in further testing. (Python)
README
# SubScraper
Overview
:small_blue_diamond:
Usage
:small_blue_diamond:
Contribute
SubScraper is a subdomain enumeration tool that uses a variety of techniques to find subdomains of a given target. Subdomain enumeration is especially helpful during penetration testing and bug bounty hunting to uncover an organization's attack surface.
Depending on the CMD arguments applied, SubScraper can resolve DNS names, request HTTP(S) information, and perform CNAME lookups for takeover opportunities during the enumeration process. This can help identify next steps and discover patterns for exploitation.
#### Key Features
- Modular design makes it easy to add new techniques/sources.
- Various levels of enumeration for additional data gathering.
- Allows for multiple target inputs, reading from `.txt` or STDIN.
- Windows CLI compatibility.
- Generate output files in `.txt` or `.csv` format.
## Installation
### Python
The following can be used to install SubScraper on Windows, Linux, & MacOS:
```bash
git clone https://github.com/m8sec/subscraper
cd subscraper
pip3 install -r requirements.txt
```
### Poetry
Install and run SubScraper using [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer):
```bash
git clone https://github.com/m8sec/subscraper
cd subscraper
poetry install
poetry run subscraper -h
```
### Docker
You can build a docker image and run subscraper from Docker:
```
git clone https://github.com/m8sec/subscraper
cd subscraper
docker build -t m8sec/subscraper .
# Display help
docker run --rm m8sec/subscraper
# Example scanning a site
docker run --rm m8sec/subscraper -d example.com
```
## Updates:
Use the configuration file at `~/.config/subscraper/config.json` to store API keys for easy reuse.
If updating to a newer version after v4.0.0, use the `-update` argument to pull a new copy of the config file and ensure
compatibility - *Note: This will remove any existing key entries.*
### Modules
A full list of modules can be found using the `-ls` command line argument:
```
bevigil - BeVigil OSINT API for scraping mobile application for subdomains (API Key Req)
crt.sh - Subdomains enumeration using cert.sh.
virustotal - Lookup subdomain on VirusTotal (API Key Req)
dnsrepo - Parse dnsrepo.noc.org without an API key - 150 result limit
chaos - Project Discovery's Chaos (API Key Req)
certspotter - Use Certspotter API to collect subdomains
bufferover - Query Bufferover.run API (API Key Req)
alienvault - Find subdomains using AlienVault OTX
redhuntlabs - RedHunt Labs recon API (API Key Req)
archive - Use archive.org to find subdomains.
dnsdumpster - Use DNS dumpster to enumerate subdomains.
censys.io - Gather subdomains through censys.io SSL cert Lookups. (API Key Req)
shodan - Get subdomains with Shodan (API Key Req)
```
## Usage
### Command Line Args
```
SubScraper Options:
-debug Enable debug logging
-update Update config file (Will remove existing entries)
-config CONFIG Override default config location
-silent Show subdomains only in output
-threads THREADS, -T THREADS Max threads for enumeration (65*).
-t TIMEOUT set connection timeouts (3*)
-d DOMAIN, --domain DOMAIN Target domain input (domain, .txt, STDIN, etc.
Module Options:
-ls List SubScraper enumeration modules.
-m MODULES Execute module(s) by name or group (all*).
-module-only Execute modules only not brute force
Bruteforce Options:
-w WORDLIST Custom wordlist for DNS brute force.
-ns NS Comma separated nameservers to use
Enumeration Options:
-r, -resolve Resolve IP address for each subdomain identified.
-c, -cname Perform CNAME lookup for subdomain takeover checks
-http Probe for active HTTP services.
-http-port HTTP_PORT HTTP ports to check, comma separated (80,443*)
Output Options:
-nc, -no-color Disable color output
-active Only report active subdomains with resolved IP
-csv Create CSV output report
-o REPORT Output file
```
### Example Inputs
```
python3 subscraper.py -d example.com -resolve -http -module-only
python3 subscraper.py -d example.com -cname -m none -o sub_report.csv -csv
cat domains.txt | python3 subscraper.py -active -silent
```
## Contribute
Contribute to the project by:
* Like and share the tool!
* Create an issue to report new enumeration techniques
* OR, better yet, develop a module and initiate a PR.