{"id":16162073,"url":"https://github.com/peterdavehello/nrd-list-downloader","last_synced_at":"2025-03-18T22:30:54.158Z","repository":{"id":70808184,"uuid":"542716313","full_name":"PeterDaveHello/nrd-list-downloader","owner":"PeterDaveHello","description":"Shell Script to download NRD(Newly Registered Domain) list for free","archived":false,"fork":false,"pushed_at":"2023-06-15T15:48:19.000Z","size":22,"stargazers_count":48,"open_issues_count":0,"forks_count":20,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-17T00:03:40.525Z","etag":null,"topics":["adguard-blocklist","bash","cybersecurity","domain","hacktoberfest","infosec","malicious","malware","nrd","osint","phishing","security","shell","shellscript","threat-intelligence","threats"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PeterDaveHello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"PeterDaveHello","open_collective":"peterdavehello","ko_fi":"peterdavehello","liberapay":"PeterDaveHello","issuehunt":"peterdavehello"}},"created_at":"2022-09-28T17:39:23.000Z","updated_at":"2025-03-10T15:09:40.000Z","dependencies_parsed_at":"2024-10-21T14:36:58.114Z","dependency_job_id":null,"html_url":"https://github.com/PeterDaveHello/nrd-list-downloader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterDaveHello%2Fnrd-list-downloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterDaveHello%2Fnrd-list-downloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterDaveHello%2Fnrd-list-downloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterDaveHello%2Fnrd-list-downloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PeterDaveHello","download_url":"https://codeload.github.com/PeterDaveHello/nrd-list-downloader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244318417,"owners_count":20433898,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adguard-blocklist","bash","cybersecurity","domain","hacktoberfest","infosec","malicious","malware","nrd","osint","phishing","security","shell","shellscript","threat-intelligence","threats"],"created_at":"2024-10-10T02:28:59.723Z","updated_at":"2025-03-18T22:30:53.856Z","avatar_url":"https://github.com/PeterDaveHello.png","language":"Shell","funding_links":["https://github.com/sponsors/PeterDaveHello","https://opencollective.com/peterdavehello","https://ko-fi.com/peterdavehello","https://liberapay.com/PeterDaveHello","https://issuehunt.io/r/peterdavehello"],"categories":[],"sub_categories":[],"readme":"# nrd-list-downloader\n\nnrd-list-downloader is a shell script that automatically downloads, decompresses, and aggregates Newly Registered Domain (NRD) lists from [WhoisDS.com](https://www.whoisds.com/newly-registered-domains). The aggregated NRD lists can be easily used with popular domain blocking tools like [Pi-Hole][Pi-Hole], [AdGuard Home][AdGuard Home], and [Blocky][Blocky] to enhance your network security.\n\n## What are Newly Registered Domains (NRDs)?\n\nNewly Registered Domains (NRDs) are domain names that have been registered very recently. They often pose risks as they may not yet be operational and are frequently used for malicious purposes. For instance, NRDs can be [parked domains][Domain Parking Wikipedia] that contain advertisements or are held for sale, provided by default by domain registrars or resellers. Hackers commonly utilize NRDs to spread malware, host phishing sites, and engage in other malicious activities. Understanding the potential threats posed by NRDs is crucial for strengthening cybersecurity measures.\n\n## Why we need NRD list?\n\nNRD lists can be utilized with DNS blocking tools such as Pi-Hole, AdGuard Home, or Blocky. By integrating NRD lists, these tools can reinforce your cybersecurity measures in conjunction with other [threat-hostlist][threat-hostlist].\n\nNRDs are sometimes risky, as hackers like to use them to spread malware, phishing sites, cybercrime, or engage in other malicious activities. Many NRDs used for harmful purposes are short-lived, making traditional security intelligence-based protection less effective against these threats. Though there are also legitimate websites using NRDs, blocking all NRDs might cause false positives. However, the blocking of benign NRDs will only be a short while. Considering the pros and cons, it's worthy to do so.\n\nSome well-known DNS providers like Cisco Umbrella, Akamai ETP(Enterprise Threat Protector), and NextDNS also provide the option to block NRD on their services.\n\n## Dependencies\n\nThis script uses several common command-line tools. Ensure you have the following installed:\n\n- `mkdir`: for creating directories\n- `wc`: for counting lines and words\n- `base64`: for encoding and decoding Base64 data\n- `curl`: for downloading content from remote URLs\n- `cat`: for concatenating and displaying file contents\n- `zcat`: for reading gzip compressed data\n- `mktemp`: for creating temporary files\n- `date`: for displaying date and time in the system\n- `tr`: for character translation, such as deleting characters\n- `realpath`: for normalizing file path names\n- `dirname`: for extracting the directory name of a file\n\nIf any of these tools are missing, you can usually install them using your system's package manager (e.g., `apt`, `yum`, or `pacman`).\n\n## Usage\n\nTo get started, clone this repository or download the `nrd-list-downloader.sh` script and grant it execute permission. Then, run the script in the terminal:\n\n```sh\n~/nrd-list-downloader $ ./nrd-list-downloader.sh\n```\n\nBy default, the script downloads and aggregates free NRD data from the past 7 days, saving each day's data in `daily/free/` and combining it into a single file `nrd-7days-free.txt`:\n\n```sh\n~/nrd-list-downloader $ ./nrd-list-downloader.sh\nYou are using nrd-list-downloader to download NRD(Newly Registered Domain) list ...\nNRD list of the last 7 will be downloaded.\n\nDownload and decompress 2022-09-21 data ...137691 domains found.\nDownload and decompress 2022-09-22 data ...55127 domains found.\nDownload and decompress 2022-09-23 data ...187397 domains found.\nDownload and decompress 2022-09-24 data ...118922 domains found.\nDownload and decompress 2022-09-25 data ...60806 domains found.\nDownload and decompress 2022-09-26 data ...82211 domains found.\nDownload and decompress 2022-09-27 data ...103078 domains found.\n...\nNRD list for the last 7 days saved to nrd-7days-free.txt, 745232 domains found.\n```\n\nTo specify a custom duration of data, set the `$DAY_RANGE` variable, such as `14`:\n\n```sh\n~/nrd-list-downloader $ DAY_RANGE=14 ./nrd-list-downloader.sh\nYou are using nrd-list-downloader to download NRD(Newly Registered Domain) list ...\nNRD list of the last 14 will be downloaded.\n\nDownload and decompress 2022-09-14 data ...141531 domains found.\nDownload and decompress 2022-09-15 data ...139059 domains found.\nDownload and decompress 2022-09-16 data ...129216 domains found.\nDownload and decompress 2022-09-17 data ...119194 domains found.\nDownload and decompress 2022-09-18 data ...105884 domains found.\nDownload and decompress 2022-09-19 data ...86455 domains found.\nDownload and decompress 2022-09-20 data ...106169 domains found.\ndaily/free/2022-09-21 existed with 137691 domains, skip the download and decompress process ...\ndaily/free/2022-09-22 existed with 55127 domains, skip the download and decompress process ...\ndaily/free/2022-09-23 existed with 187397 domains, skip the download and decompress process ...\ndaily/free/2022-09-24 existed with 118922 domains, skip the download and decompress process ...\ndaily/free/2022-09-25 existed with 60806 domains, skip the download and decompress process ...\ndaily/free/2022-09-26 existed with 82211 domains, skip the download and decompress process ...\ndaily/free/2022-09-27 existed with 103078 domains, skip the download and decompress process ...\nNRD list for the last 14 days saved to nrd-14days-free.txt, 1572740 domains found.\n```\n\nThe result will be saved in `nrd-14days-free.txt`. The script will not re-download existing daily data in the `daily/free/` directory, saving time and avoiding redundant downloads.\n\nWe recommend running nrd-list-downloader regularly (daily or weekly) to keep your NRD lists updated and maximize the effectiveness of your domain blocking tools.\n\n### Paid Account Support\n\nThis script also supports paid accounts on WhoisDS.com for accessing premium data. To use a paid account, set the following two variables:\n\n- `PAID_WHOISDS_USERNAME`\n- `PAID_WHOISDS_PASSWORD`\n\nYou can set the username and password by directly modifying the script or by passing them as environment variables when running the script:\n\n```sh\n~/nrd-list-downloader $ PAID_WHOISDS_USERNAME=\"johndoe@unknownexample.net\" PAID_WHOISDS_PASSWORD=\"RAND.mPA$$W0rd\" ./nrd-list-downloader.sh\n```\n\nPaid data will be stored in the `daily/paid/` directory and aggregated into the `nrd-7days-paid.txt` file.\n\nThe script does not include a mechanism for checking the validity of these credentials, so ensure the information provided is accurate.\n\n## NRD Risk Reference\n\nHere are some references that show you why NRDs are risky. You can take a look, and decide to block them or not.\n\n- Newly Registered Domains: Malicious Abuse by Bad Actors - Unit 42 - Palo Alto Networks\n  - \u003chttps://unit42.paloaltonetworks.com/newly-registered-domains-malicious-abuse-by-bad-actors/\u003e\n- Block newly-registered domains to reduce security threats - The State of Security\n  - \u003chttps://www.tripwire.com/state-of-security/featured/block-newly-registered-domains-to-reduce-security-threats-in-your-organisation/\u003e\n- Should We Blocklist Newly Registered Domains? - Uptycs\n  - \u003chttps://www.uptycs.com/blog/should-we-blocklist-newly-registered-domains\u003e\n- Why blocking NRDs ? - The Newly Registered Domains category | Artica Wiki\n  - \u003chttps://wiki.articatech.com/filtering-service/newly-registered-domains\u003e\n\nExplore these articles and others to better understand the risks associated with NRDs and determine the most appropriate approach for your organization.\n\n## Credits\n\nSpecial thanks to WhoisDS.com, although they do not require any mentions in their license. They kindly provide those data for free, even for commercial use, without restrictions.\n\nQuoted from \u003chttps://www.whoisds.com/newly-registered-domains\u003e:\n\n\u003e The data provided below is daily list of Newly Registered Domains without whois database downloaded free of charge; except where otherwise stated, they may be reused, including for commercial purposes, without a license and without any payment.\n\n## License\n\nGPL-3.0 (GNU GENERAL PUBLIC LICENSE Version 3)\n\n[Pi-Hole]: https://pi-hole.net/\n[AdGuard Home]: https://github.com/AdguardTeam/AdguardHome\n[Blocky]: https://0xerr0r.github.io/blocky\n[threat-hostlist]: https://github.com/PeterDaveHello/threat-hostlist\n[Domain Parking Wikipedia]: (https://en.wikipedia.org/wiki/Domain_parking)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterdavehello%2Fnrd-list-downloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterdavehello%2Fnrd-list-downloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterdavehello%2Fnrd-list-downloader/lists"}