{"id":13474959,"url":"https://github.com/m8sec/subscraper","last_synced_at":"2025-10-25T18:17:51.739Z","repository":{"id":46102121,"uuid":"150627712","full_name":"m8sec/subscraper","owner":"m8sec","description":"Subdomain and target enumeration tool built for offensive security testing","archived":false,"fork":false,"pushed_at":"2024-06-19T14:26:13.000Z","size":104,"stargazers_count":864,"open_issues_count":0,"forks_count":99,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-04-12T17:46:23.345Z","etag":null,"topics":["bugbounty","enumeration","osint","penetration-testing","pentest","pentest-tool","python3","subdomain-brute","subdomain-enumeration","subdomain-scanner","subdomain-takeover"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m8sec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2018-09-27T18:00:51.000Z","updated_at":"2025-04-11T06:43:30.000Z","dependencies_parsed_at":"2023-10-16T22:39:13.912Z","dependency_job_id":"d837f94a-659d-49cf-9853-c59e8f5180f1","html_url":"https://github.com/m8sec/subscraper","commit_stats":null,"previous_names":["m8r0wn/subscraper"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m8sec%2Fsubscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m8sec%2Fsubscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m8sec%2Fsubscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m8sec%2Fsubscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m8sec","download_url":"https://codeload.github.com/m8sec/subscraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254592367,"owners_count":22097010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","enumeration","osint","penetration-testing","pentest","pentest-tool","python3","subdomain-brute","subdomain-enumeration","subdomain-scanner","subdomain-takeover"],"created_at":"2024-07-31T16:01:16.248Z","updated_at":"2025-10-25T18:17:51.650Z","avatar_url":"https://github.com/m8sec.png","language":"Python","funding_links":[],"categories":["Uncategorized","Python"],"sub_categories":["Uncategorized"],"readme":"# SubScraper\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.twitter.com/m8sec\"\u003e\u003cimg src=\"https://img.shields.io/badge/Twitter-@m8sec-blue?style=plastic\u0026logo=twitter\"/\u003e\u003c/a\u003e\u0026nbsp;\u0026nbsp;\n    \u003ca href=\"/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-GPL%20v3.0-green.svg?style=plastic\"/\u003e\u003c/a\u003e \u0026nbsp;\u0026nbsp;\n  \u003cbr\u003e\n    \u003ca href=\"https://github.com/m8sec/subscraper#subscraper\"\u003eOverview\u003c/a\u003e\n    \u0026nbsp;\u0026nbsp; :small_blue_diamond: \u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://github.com/m8sec/subscraper#usage\"\u003eUsage\u003c/a\u003e\n    \u0026nbsp;\u0026nbsp; :small_blue_diamond: \u0026nbsp;\u0026nbsp;\n    \u003ca href=\"https://github.com/m8sec/subscraper#contribute\"\u003eContribute\u003c/a\u003e\n  \u003cbr\u003e\n\u003c/p\u003e\n\nSubScraper is a subdomain enumeration tool that uses a variety of techniques to find subdomains of a given target. Subdomain enumeration is especially helpful during penetration testing and bug bounty hunting to uncover an organization's attack surface.\n\nDepending on the CMD arguments applied, SubScraper can resolve DNS names, request HTTP(S) information, and perform CNAME lookups for takeover opportunities during the enumeration process. This can help identify next steps and discover patterns for exploitation.  \n\n#### Key Features\n\n- Modular design makes it easy to add new techniques/sources.\n- Various levels of enumeration for additional data gathering.\n- Allows for multiple target inputs, reading from `.txt` or STDIN.\n- Windows CLI compatibility. \n- Generate output files in `.txt` or `.csv` format.\n\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"942\" alt=\"demo\" src=\"https://github.com/m8sec/subscraper/assets/13889819/c8503198-7759-4123-b921-28a74b773e7b\"\u003e\n\u003c/p\u003e\n\n\n## Installation\n### Python\nThe following can be used to install SubScraper on Windows, Linux, \u0026 MacOS:\n\n```bash\ngit clone https://github.com/m8sec/subscraper\ncd subscraper\npip3 install -r requirements.txt\n```\n\n### Poetry\nInstall and run SubScraper using [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer):\n```bash\ngit clone https://github.com/m8sec/subscraper\ncd subscraper\npoetry install\npoetry run subscraper -h\n```\n\n### Docker\nYou can build a docker image and run subscraper from Docker:\n```\ngit clone https://github.com/m8sec/subscraper\ncd subscraper\ndocker build -t m8sec/subscraper .\n\n# Display help\ndocker run --rm m8sec/subscraper\n\n# Example scanning a site\ndocker run --rm m8sec/subscraper -d example.com\n```\n\n## Updates:\nUse the configuration file at `~/.config/subscraper/config.json` to store API keys for easy reuse. \n\nIf updating to a newer version after v4.0.0, use the `-update` argument to pull a new copy of the config file and ensure \ncompatibility - *Note: This will remove any existing key entries.*\n\n\n### Modules\nA full list of modules can be found using the `-ls` command line argument:\n```\nbevigil              - BeVigil OSINT API for scraping mobile application for subdomains (API Key Req)\ncrt.sh               - Subdomains enumeration using cert.sh.\nvirustotal           - Lookup subdomain on VirusTotal (API Key Req)\ndnsrepo              - Parse dnsrepo.noc.org without an API key - 150 result limit\nchaos                - Project Discovery's Chaos (API Key Req)\ncertspotter          - Use Certspotter API to collect subdomains\nbufferover           - Query Bufferover.run API (API Key Req)\nalienvault           - Find subdomains using AlienVault OTX\nredhuntlabs          - RedHunt Labs recon API (API Key Req)\narchive              - Use archive.org to find subdomains.\ndnsdumpster          - Use DNS dumpster to enumerate subdomains.\ncensys.io            - Gather subdomains through censys.io SSL cert Lookups. (API Key Req)\nshodan               - Get subdomains with Shodan (API Key Req)\n```\n\n## Usage\n### Command Line Args\n```\nSubScraper Options:\n  -debug                Enable debug logging\n  -update               Update config file (Will remove existing entries)\n  -config CONFIG        Override default config location\n  -silent               Show subdomains only in output\n  -threads THREADS, -T THREADS    Max threads for enumeration (65*).\n  -t TIMEOUT                      set connection timeouts (3*)\n  -d DOMAIN, --domain DOMAIN      Target domain input (domain, .txt, STDIN, etc.\n\nModule Options:\n  -ls                   List SubScraper enumeration modules.\n  -m MODULES            Execute module(s) by name or group (all*).\n  -module-only          Execute modules only not brute force\n\nBruteforce Options:\n  -w WORDLIST           Custom wordlist for DNS brute force.\n  -ns NS                Comma separated nameservers to use\n\nEnumeration Options:\n  -r, -resolve          Resolve IP address for each subdomain identified.\n  -c, -cname            Perform CNAME lookup for subdomain takeover checks\n  -http                 Probe for active HTTP services.\n  -http-port HTTP_PORT  HTTP ports to check, comma separated (80,443*)\n\nOutput Options:\n  -nc, -no-color        Disable color output\n  -active               Only report active subdomains with resolved IP\n  -csv                  Create CSV output report\n  -o REPORT             Output file\n```\n\n### Example Inputs\n```\npython3 subscraper.py -d example.com -resolve -http -module-only\npython3 subscraper.py -d example.com -cname -m none -o sub_report.csv -csv\ncat domains.txt | python3 subscraper.py -active -silent\n```\n\n## Contribute\nContribute to the project by:\n* Like and share the tool!\n* Create an issue to report new enumeration techniques\n* OR, better yet, develop a module and initiate a PR.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm8sec%2Fsubscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm8sec%2Fsubscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm8sec%2Fsubscraper/lists"}