{"id":13841190,"url":"https://github.com/nscuro/fdnssearch","last_synced_at":"2025-07-11T12:30:30.746Z","repository":{"id":37029044,"uuid":"277367551","full_name":"nscuro/fdnssearch","owner":"nscuro","description":"Swiftly search FDNS datasets from Rapid7 Open Data","archived":true,"fork":false,"pushed_at":"2022-11-20T19:49:20.000Z","size":148,"stargazers_count":21,"open_issues_count":9,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-05T17:26:54.557Z","etag":null,"topics":["bugbounty","dns","fdns","golang","opendata","rapid7","subdomains"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nscuro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-05T19:15:03.000Z","updated_at":"2024-04-22T04:05:32.000Z","dependencies_parsed_at":"2023-01-20T10:47:50.180Z","dependency_job_id":null,"html_url":"https://github.com/nscuro/fdnssearch","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nscuro%2Ffdnssearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nscuro%2Ffdnssearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nscuro%2Ffdnssearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nscuro%2Ffdnssearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nscuro","download_url":"https://codeload.github.com/nscuro/fdnssearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225720397,"owners_count":17513596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","dns","fdns","golang","opendata","rapid7","subdomains"],"created_at":"2024-08-04T17:01:04.067Z","updated_at":"2024-11-21T11:30:22.393Z","avatar_url":"https://github.com/nscuro.png","language":"Go","funding_links":[],"categories":["Go (531)","Go"],"sub_categories":[],"readme":"# fdnssearch\n\n![Build Status](https://github.com/nscuro/fdnssearch/workflows/Continuous%20Integration/badge.svg?branch=master)\n\n*Swiftly search [FDNS](ttps://github.com/rapid7/sonar/wiki/Forward-DNS) datasets from Rapid7 Open Data*\n\n**Disclaimer**: You can do most of what *fdnssearch* does with [`bash`, `curl`, `pigz` and `jq`](https://github.com/rapid7/sonar/wiki/Analyzing-Datasets). This is nothing revolutionary. *fdnssearch* simply offers a [nicer UX](#usage) and some [QoL features](#interoparability). If you prefer a server-based solution, take a look at the [amazing Crobat project](https://github.com/Cgboal/SonarSearch).\n\n## Installation\n\n`go install github.com/nscuro/fdnssearch/cmd/fdnssearch@latest`\n\nAlternatively, clone this repo and run `make install`. Make sure `$GOPATH/bin` is in your `$PATH`.\n\n*fdnssearch* requires Go \u003e= 1.17\n\nPrebuilt binaries are available [as well](https://github.com/nscuro/fdnssearch/releases/).\n\n### Docker\n\nClone this repository, `cd` into it and run `make docker`.  \nThe image can then be used as follows: `docker -it --rm nscuro/fdnssearch -h`\n\n## Usage\n\n```                                                  \nUsage:\n  fdnssearch [flags]\n\nFlags:\n      --amass-config string    Amass config to load domains from\n  -a, --any                    Additionally search ANY dataset (ignored when -f is set)\n      --any-only               Only search ANY dataset (ignored when -f is set)\n  -d, --domains stringArray    Domains to search for\n  -e, --excludes stringArray   Domains to exclude from search\n  -f, --files stringArray      Dataset files\n  -h, --help                   help for fdnssearch\n  -o, --output string          Output file\n      --plain                  Disable colored output\n  -q, --quiet                  Only print results, no errors or log messages\n      --timeout int            Timeout in seconds\n  -t, --types stringArray      Record types to search for (a, aaaa, cname, txt, mx) (default [a])\n  -v, --version                Show version\n```\n\nErrors and log messages are written to `STDERR`, search results to `STDOUT`. This allows for easy piping without the need to use `--quiet`. \nWhen piping results to other commands, make sure to disable colored output with `--plain`.\n\n### Examples\n\nSearching for `A` and `CNAME` records of subdomains of `example.de` and `example.com`:\n\n```bash\n$ fdnssearch -d example.de -d example.com -t a -t cname\n```\n\nSearching for `AAAA` and `TXT` records of subdomains of `example.com`, disabling colored output and writing results to `results.txt`:\n\n```bash\n$ fdnssearch -d example.com -t aaaa -t txt --plain | tee results.txt\n```\n\nSearching for `A` records of subdomains of `example.com`, excluding `(*.)acme.example.com` and writing results to `results.txt`:\n\n```bash\n$ fdnssearch -d example.com -e acme.example.com -o results.txt\n```\n\n### Remote Datasets\n\nWhen no local dataset files are provided using `-f` / `--files`, *fdnssearch* will fetch the current datasets from Rapid7's website. It will search all datasets that match the record types provided with `-t` / `--types`. \n\nThis requires a fairly good internet connection, but doesn't pollute your storage with huge files that get outdated quickly. The slower your connection, the fewer search workers are required.\n\nRapid7 provides a dataset with `ANY` records in addition to the specific datasets:\n\n\u003e Until early November 2017, all of these were for the 'ANY' record with a fallback A and AAAA request if neccessary. After that, the ANY study represents only the responses to ANY requests, and dedicated studies were created for the A, AAAA, CNAME and TXT record lookups with appropriately named files.\n\nIf you want your search to include this dataset as well, use the `--any` flag. Be aware that you **will** get a lot of duplicate results this way. Be sure to [deduplicate](#deduplication) your results. If you want to save time however, additionally pass the `--any-only` flag. *fdnssearch* will then exclusively search search the `ANY` dataset. While this may yield results faster, you may not get as many results as without `--any-only`.\n\n### Local Datasets\n\nIt is possible to search local dataset files as well:\n\n```bash\n$ fdnssearch -f /path/to/datasets/2020-05-23-1590208726-fdns_a.json.gz -d example.com\n```\n\n### Performance\n\n*fdnssearch* uses *klauspost*'s [`pgzip`](https://github.com/klauspost/pgzip) for performant decompression of the datasets.\nThanks to `pgzip`, the performance of *fdnssearch* is pretty much on par with the `pigz`, `grep` and `jq` approach:\n\n```bash\n$ time pigz -dc /path/to/datasets/2020-06-28-1593366733-fdns_cname.json.gz \\\n    | grep 'google\\.com' \\\n    | jq '. | select(.name | endswith(\".google.com\")) | select(.type == \"cname\") | .name' \\\n    \u003e /dev/null\npigz -dc /path/to/datasets/2020-06-28-1593366733-fdns_cname.json.gz  57.22s user 41.11s system 212% cpu 46.269 total\ngrep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn,.idea,.tox}   18.78s user 6.91s system 55% cpu 46.268 total\njq  \u003e /dev/null  2.59s user 0.07s system 5% cpu 46.268 total\n```\n\n```bash\n$ time fdnssearch -d google.com -t cname --quiet \\\n    -f /path/to/datasets/2020-06-28-1593366733-fdns_cname.json.gz \\\n    \u003e /dev/null\nfdnssearch -d google.com -t cname -f  --quiet \u003e /dev/null  64.58s user 0.85s system 144% cpu 45.266 total\n```\n\nThis is with an [Intel i7 8700K](https://ark.intel.com/content/www/us/en/ark/products/126684/intel-core-i7-8700k-processor-12m-cache-up-to-4-70-ghz.html) and a [Samsung 970 EVO NVMe M.2 SSD](https://www.samsung.com/us/computing/memory-storage/solid-state-drives/ssd-970-evo-nvme-m2-500gb-mz-v7e500bw/) on Windows 10 in WSL 2.  \nYour mileage may vary.\n\n### Deduplication\n\n*fdnssearch* will not perform deduplication in order to provide search results as quickly and efficiently as possible. \nUse tools like `uniq` or `sort` for this.\n\nGiven a file `results.txt` which only contains record names, deduplication can be achieved with:\n\n```bash\n$ sort --unique -o results.txt results.txt\n```\n\n### Interoparability\n\n#### Amass\n\n*fdnssearch* can parse target domains and exclusions from [Amass config files](https://github.com/OWASP/Amass/blob/master/examples/config.ini):\n\n```bash\n$ grep -A 3 \"\\[domains\\]\" amass.ini\n[domains]\ndomain = example.com\ndomain = example.de\ndomain = example.fr\n\n$ grep -A 1 \"\\[blacklisted\\]\" amass.ini\n[blacklisted]\nsubdomain = acme.example.com\n\n$ fdnssearch --amass-config amass.ini\n```\n\nThis is equivalent to\n\n```bash\n$ fdnssearch -d example.com -d example.de -d example.fr -e acme.example.com\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnscuro%2Ffdnssearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnscuro%2Ffdnssearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnscuro%2Ffdnssearch/lists"}