{"id":29385800,"url":"https://github.com/phani-kb/dns-toolkit","last_synced_at":"2025-07-19T18:02:20.733Z","repository":{"id":303479402,"uuid":"919767769","full_name":"phani-kb/dns-toolkit","owner":"phani-kb","description":"A Go CLI for collecting, resolving, analyzing, and generating DNS blocklists and allowlists — including DNS-to-IP resolution, reverse lookups, and overlap detection — for use in DNS sinkholes and firewalls.","archived":false,"fork":false,"pushed_at":"2025-07-16T03:31:55.000Z","size":26564,"stargazers_count":0,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-16T07:53:11.893Z","etag":null,"topics":["adblock","adguard","ads","allowlist","blocklist","cobra-cli","consolidate","dns","domain","filterlist","golang","host","ip","malware","overlap","phishing","pi-hole","privacy","scam","sinkhole"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phani-kb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-21T01:15:35.000Z","updated_at":"2025-07-16T03:25:13.000Z","dependencies_parsed_at":"2025-07-07T23:07:22.127Z","dependency_job_id":"474afc60-66d4-4f7f-a4a6-f4994aa83f3b","html_url":"https://github.com/phani-kb/dns-toolkit","commit_stats":null,"previous_names":["phani-kb/dns-toolkit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/phani-kb/dns-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phani-kb%2Fdns-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phani-kb%2Fdns-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phani-kb%2Fdns-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phani-kb%2Fdns-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phani-kb","download_url":"https://codeload.github.com/phani-kb/dns-toolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phani-kb%2Fdns-toolkit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265981982,"owners_count":23859562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adblock","adguard","ads","allowlist","blocklist","cobra-cli","consolidate","dns","domain","filterlist","golang","host","ip","malware","overlap","phishing","pi-hole","privacy","scam","sinkhole"],"created_at":"2025-07-10T06:00:26.484Z","updated_at":"2025-07-19T18:02:20.711Z","avatar_url":"https://github.com/phani-kb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DNS Toolkit\n\n[![Go CI Workflow](https://github.com/phani-kb/dns-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/phani-kb/dns-toolkit/actions/workflows/ci.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/phani-kb/dns-toolkit)](https://goreportcard.com/report/github.com/phani-kb/dns-toolkit)\n[![codecov](https://codecov.io/gh/phani-kb/dns-toolkit/branch/main/graph/badge.svg)](https://codecov.io/gh/phani-kb/dns-toolkit)\n[![GoDoc](https://godoc.org/github.com/phani-kb/dns-toolkit?status.svg)](https://godoc.org/github.com/phani-kb/dns-toolkit)\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n[![Go Version](https://img.shields.io/badge/Go-1.23-blue.svg)](https://golang.org/doc/go1.23)\n[![GitHub commit activity](https://img.shields.io/github/commit-activity/y/phani-kb/dns-toolkit)](https://github.com/phani-kb/dns-toolkit/graphs/commit-activity)\n[![GitHub repo size](https://img.shields.io/github/repo-size/phani-kb/dns-toolkit)](https://github.com/phani-kb/dns-toolkit)\n\nA Go-based command-line utility for downloading, processing, resolving, and consolidating DNS blocklists and allowlists from multiple sources. Performs DNS-to-IP resolution, reverse lookups, and overlap detection. Generates ready-to-use lists for DNS sinkholes, Pi-hole, AdGuard Home, and other network security tools.\n\n---\n\n## Source Configuration (Important!)\n\nAll source details—including URLs, categories, licenses, update frequency, and website links—are managed in the configuration files located in the `data/config` folder (`sources*.json`).\n\nSources are configured in `data/config/sources*.json` files. Each source specifies:\n\n- Download URL and frequency\n- Source type (domain, IPv4, IPv6, AdGuard, etc.)\n- Categories (advertising, malware, privacy, etc.)\n- License and website information\n\n[View and edit source configuration files in `data/config/`](https://github.com/phani-kb/dns-toolkit/tree/main/data/config)\n\n\u003e **To add, modify, or review sources, always refer to the files in `data/config/`.**\n\n## Special Note on Top Domains (tranco-list.eu)\n\nTop domains sourced from the tranco-list.eu list (`domain_top` type) are treated as an allowlist. You can configure the `count_to_consider` value in the relevant config file (`data/config/sources_domain_top.json`) to increase the number of top domains included in the allowlist. This is useful for fine-tuning the strictness or permissiveness of your DNS filtering setup.\n\n## Published Outputs\n\n**Ready-to-use blocklist files are published daily to the [`output`](https://github.com/phani-kb/dns-toolkit/tree/output) branch:**\n\n- Domain and IP blocklists/allowlists compatible with Pi-hole, pfBlockerNG, AdGuard Home\n- Lists organized by size (mini, lite, normal, big) and category (advertising, malware, privacy)\n- Top entries based on source frequency for high-confidence blocking\n\n**Usage:** Add `https://raw.githubusercontent.com/phani-kb/dns-toolkit/output/[filename]` to your DNS filtering tool.\n\n**[View Detailed Overlap Analysis →](https://github.com/phani-kb/dns-toolkit/blob/output/overlap.md)** - Comprehensive analysis showing how entries are shared across different DNS sources.\n\n**Processing summaries and metadata are archived in the [`summaries`](https://github.com/phani-kb/dns-toolkit/tree/summaries) branch with 1-year retention.**\n\n\u003c!-- BRANCH_SIZES_START --\u003e\n## Branch Sizes\n\n**Note:** The repo size badge above only reflects the default branch (`main`).\n\n- **Output branch size:** 202.77 MB\n- **Summaries branch size:** 0.44 MB\n\n\u003c!-- BRANCH_SIZES_END --\u003e\n\n---\n\n\u003c!-- STATS_START --\u003e\n## Source Statistics\n\n*Automatically generated statistics from source configuration files*\n\n| Metric | Count | Details |\n|--------|-------|---------|\n| **Total Sources** | 134 | 90 enabled, 44 disabled |\n| **Blocklist Sources** | 126 | Sources providing blocking rules |\n| **Allowlist Sources** | 20 | Sources providing exception rules |\n| **Categories** | 24 | ads, adult, annoyance, anonymizer, botnet, dns, doh, exploit, fakenews, gambling, malicious, malware, mobile, others, phishing, privacy, proxy, ransomware, scam, security, spam, threat, trackers, trojan |\n| **Source Types** | 25 | adguard, cidr_ipv4, domain, domain_adguard, domain_comment, domain_csv_http_url_find, domain_custom_csv_blackbook, domain_custom_csv_maltrail, domain_custom_html_ccam, domain_custom_html_puppyscams, domain_http_url, domain_top, domain_url, domain_with_comment_suffix, hostname, ipv4, ipv4_cidr_expand, ipv4_custom_html_ccam, ipv4_find, ipv4_http_url, ipv4_range_expand, ipv4_url, ipv6, ipv6_find, ipv6_htaccess |\n| **Geographic Coverage** | 21 countries | CN, CZ, DE, ES, FI, FR, HU, ID, IL, IT, KR, LV, MY, NL, RO, RU, SA, SK, UA, US, VN |\n| **Last Updated** | 2025-07-17 21:08:35 UTC | Statistics generation time |\n\n\u003c!-- STATS_END --\u003e\n\n## Installation\n\n```bash\ngit clone https://github.com/phani-kb/dns-toolkit.git\ncd dns-toolkit\ngo build -o bin/dns-toolkit main.go\n```\n\n## Quick Start\n\n```bash\n# Download and process all sources\ndns-toolkit download\ndns-toolkit process\ndns-toolkit consolidate\ndns-toolkit generate output\n\n# Search for a domain\ndns-toolkit search example.com\n\n# Analyze overlaps between sources\ndns-toolkit overlap\n\n# Find top entries across sources\ndns-toolkit top\n```\n\n## Key Commands\n\n```text\nDNS Toolkit\n\nUsage:\n  dns-toolkit [command]\n\nAvailable Commands:\n  archive          Archive DNS toolkit data\n  consolidate      Consolidate processed files\n  download         Download enabled sources\n  generate         Generate different types of outputs\n  help             Help about any command\n  overlap          Find overlap between source files\n  process          Process downloaded files\n  search           Search for a domain or IP in the processed files\n  sts              Prints the source types summary\n  top              Find top entry(s) in each generic source type\n  validate-sources Validate the sources configuration\n  version          Print the version number of DNS Toolkit\n\nFlags:\n  -h, --help   help for dns-toolkit\n\nUse \"dns-toolkit [command] --help\" for more information about a command.\n```\n\n## Output Structure\n\n```text\ndata/output/\n├── *_blocklist.txt            # Blocklists for various source types (adguard, domain, ipv4, ipv6, cidr)\n├── *_allowlist.txt            # Allowlists for various source types (adguard, domain, etc.)\n├── categories/                # Lists by category (ads, malware, privacy, etc.)\n├── groups/                    # Lists by size (mini, lite, normal, big)\n├── top/                       # Top entries based on source frequency\n├── ignored/                   # Entries filtered by allowlists\n└── summaries/                 # Processing metadata and statistics\n```\n\n## Credits\n\nThis project is made possible by the following blocklist and allowlist sources:\n\n1Hosts (Lite), abpvn_hosts, Adaway, AdBlockID, AdGuard Base filter, AdGuard DNS filter, AlienVault_Reputation, AntiAdBlockFilters, anudeepND_adservers, anudeepND_Allowlist, bigdargon_hostsVN, BinaryDefense_Banlist, Blackhole_Today, BlockListDE_Brute, BlockListDE_Strong, Blocklists UT1 Cryptojacking, Blocklists UT1 Malware, Blocklists UT1 Publicite, Borestad_AbuseIPDB, BruteforceBlocker, Cameleon, CINSScore_BadGuys_Army, CJX Annoyance, CybercrimeTracker_All, CybercrimeTracker_CCAM, CybercrimeTracker_CCPMGate, cyberhost_malware-blocklist, DandelionSprout-Anti-Malware-List, Dan Pollock's List, DoH_IP_blocklists, DoH_IP_list, DShield, EasyList, Easy Privacy, EmergingThreats_CompromisedIPs, ET_fwip, fabriziosalmi_allowlist, fabriziosalmi_blocklists, FabrizioSalmi_DNS, FadeMind_2o7Net, FakeWebshopListHUN, Firehol_abusers_30d, Firehol_BitcoinNodes_1d, Firehol_Botscout_1d, Firehol_GPF_Comics, Firehol_level1, Firehol_level2, Firehol_level3, Firehol_SocksProxy_7d, Firehol_SSLProxies_1d, Frogeye trackers, GetAdmiral Domains Filter List, GlobalAntiScamOrg-blocklist-domains, GlobalAntiScamOrg-blocklist-ips, Greensnow, HaGeZi Amazon Tracker, HaGeZi Apple Tracker, HaGeZi DNS TIF Mini, HaGeZi_DoH, HaGeZi Encrypted DNS Servers, HaGeZi Gambling Only Domains, HaGeZi Microsoft Tracker, HaGeZi Most Abused TLDs, HaGeZi Normal, HaGeZi Pro, HaGeZi's Pro Blocklist, HaGeZi_TIF, HaGeZi Xiaomi Tracker, Hestat_Minerchk, Hostsfile, hufilter, iam-py-test_my-filters-001-antitypo, jarelllama_Scam-Blocklist, kadantiscam, Local Allowlist (AdGuard), Local Allowlist (Domain), Local Blocklist (AdGuard), Malicious URL Blocklist (URLHaus), Maltrail_StaticTrails, malware-filter_phishing-filter, MyIP_MS_Blocklist, nrd-14day-mini, OISD Blocklist Big, OpenPhish_Feed, Peter Lowe's Blocklist, phishing_army, Policeman_SimpleDomainsBlocklist, Public_DNS4, PuppyScams, quidsup_notrack-annoyance, quidsup_notrack-malware, quidsup_notrack-tracker, RedDragonWebDesign_block-everything, RPiList_specials-malware, RPiList_specials-phishing, Rutgers_DROP, Sblam_Blocklist, ScriptzTeam_BadIPS, Sentinel_Greylist, ShadowWhisperer_Allowlist, ShadowWhisperer_BlockLists Ads, ShadowWhisperer_BlockLists Adult, ShadowWhisperer_BlockLists Malware, ShadowWhisperer_BlockLists Scam, ShadowWhisperer's Dating List, Spam404, spamhaus_drop, Stamparm_Blackbook, StevenBlack_Adhoc_list, StevenBlack_Fake_Gambling_Porn, T145_allowlist-domains, T145_allowlist-ips, T145_black-mirror, T145_blocklist, tranco, Ukrainian Ad Filter, Ukrainian Annoyance Filter, Ukrainian Privacy Filter, Ukrainian Security Filter, UncheckyAds, URLHaus (Abuse.ch), URLHaus_Text, USOM-Blocklists-domains, USOM-Blocklists-ips, Viriback_Dump, VXVault_URLList, WaLLy3K, WindowsSpyBlocker_Hosts_spy, Winhelp2002, YousList, YousList-AdGuard, youtube_GoodbyeAds, Yoyo Adservers-Hosts, Yoyo AdServers-IPList\n\n**Note:** Detailed information about each source including URLs, licenses, categories, and website links can be found in the configuration files located in `data/config/sources*.json`.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes with tests\n4. **Validate commit messages** before pushing. All commit message must reference a GitHub issue.\n5. Submit a pull request\n\n## License\n\nThis project is licensed under the terms specified in the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphani-kb%2Fdns-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphani-kb%2Fdns-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphani-kb%2Fdns-toolkit/lists"}