Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/bongochong/combinedprivacyblocklists

Ad-blocking hosts files, IP block lists, PAC filters, ABP / uBO / ADG subscriptions, and a whole lot more. All merged from multiple reputable sources, combined with my own research. Also, script-based utilities to help you create such things yourself. Updated at least once every two weeks, often more frequently.
https://github.com/bongochong/combinedprivacyblocklists
ad-blocking ancient-truths bash bittorrent blocklists bromite cygwin dns file-sharing foss freedom gnu hosts linux mac p2p pihole privacy security shell-scripting
Last synced: 5 days ago
JSON representation
Host: GitHub
URL: https://github.com/bongochong/combinedprivacyblocklists
Owner: bongochong
License: other
Created: 2017-05-04T20:27:57.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2024-10-28T21:08:55.000Z (about 2 months ago)
Last Synced: 2024-10-29T22:36:27.449Z (about 2 months ago)
Topics: ad-blocking, ancient-truths, bash, bittorrent, blocklists, bromite, cygwin, dns, file-sharing, foss, freedom, gnu, hosts, linux, mac, p2p, pihole, privacy, security, shell-scripting
Language: Shell
Homepage:
Size: 389 MB
Stars: 232
Watchers: 10
Forks: 25
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

        # Combined Privacy Block Lists  



  

  



Ad & malware-blocking hosts files, IP block lists, PAC filters, ABP / uBO / ADG subscriptions and more, all compiled from multiple reputable sources, along with my own research. The generation of these lists is automated via my homemade scripts and utilities, friendler versions of which are available in a [subdirectory of this repository](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/BLT) and regularly updated to reflect backend improvements. Said utilities can be installed with a simple [script](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/cpbl-installer.sh) or [one-liner](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/README.md), for ease of use. Most content in this repo is approved by Richard Stallman himself (theoretically). Use as you see fit, but do continue reading.  

| TLD-Optimized |

| - |

| [**ABP**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/cpbl-abp-list.txt) ǁ [**DNSMasq**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/cpbl-dnsmasq.txt) ǁ [**Domains**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/cpbl-ctld.txt) ǁ [**Wildcards**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/cpbl-wildcard-blacklist.txt) ǁ [**Bromite**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/cpbl-bromite.dat) ǁ [**PAC**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/pac-done.js) |

| Traditional |

| - |

| [**Standard Hosts**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/newhosts-final.hosts) ǁ [**Windows Hosts**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/optimized-win.hosts) ǁ [**Domains**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/BlacklistedDomains.txt) |

| IP Addresses & Ranges | 

| - |

| [**CIDR**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final.cidr) ǁ [**IPFilter (DAT)**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final-win.dat) ǁ [**IPFilter (P2P)**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final.p2p) |

My lists block malicious and harmfully deceptive content, like advertising, tracking, telemetry, scams, and malware. The lists **do not** block porn, social media, or so-called "fake news" sites, except for the advertising, tracking, telemetry, scam, and malware servers associated with them. False positives and invalid entries are [rigorously](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/NoFormatting/WhitelistedDomains.txt) [removed](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/parsing/tld-filter.dat) from all lists, and those which contain domains (like hosts files) are processed by an IDN-to-Punycode conversion routine, meaning that they can be utilized on *all* operating systems. In short, these block lists are both comprehensive and universally compatible, while remaining reasonable in size + scope, and free of censorship. If you find yourself in need of slimmer or somewhat less aggressive lists, I generate [Mini or *Light* variants](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/MiniLists) with every update.  

Updates to the lists usually occur every two weeks, but often more frequently. If there is a wave of rapid updates in a single day, this means that there are significant improvements under way. Please consider [donating to the project](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/DONATE.md) if you find it useful. It should also be noted that my lists are not *just* relevant to those living in English-speaking countries, as I make efforts to block a good amount of malicious "international" content as well. Additionally, I would like to point out that I manually review updates before making final commits, as I rely on this data myself. Some of my work is included OOTB or as an option in the wonderful [AdBlock Plus](https://adblockplus.org/subscriptions#type_other), [ReThink DNS](https://rethinkdns.com/configure), [Blokada](https://community.blokada.org/tag/blocklist), and [DNSWarden](https://dnswarden.com/customfilter.html) projects, along with [adblock](https://github.com/openwrt/packages/blob/master/net/adblock/files/adblock.sources) + [simple-adblock](https://github.com/openwrt/packages/blob/master/net/adblock-fast/files/etc/config/adblock-fast) for OpenWrt, all of which offer important utilities that enable users to block undesirable content across a wide variety of platforms.  

**Descriptions (*mind the [footnotes](#footnotes)*)**

- [**newhosts-final.hosts**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/newhosts-final.hosts): Is a comprehensive hosts file which blocks known ad, exploit, malware, and tracking servers. It is pulled from PGL Yoyo^*01*, URL Haus^*02*, Updated Immortal Malware Domains^*03*, several AdGuard^*04* and EasyList sources^*05*, along with a wide variety of supplementary lists for increased protection against telemetry, additions for mobile platforms, and emerging threat prevention.^*06* It is then merged, sorted and deduped. This list is compatible with any operating systems that make use of a hosts file, and works great with all manner of DNS filtering setups, including local VPN-based solutions for mobile platforms. A notably enhanced experience with the world wide web awaits you. _An [IPv6 version](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/newhosts-final-IPv6.hosts), a [version for both IPv4 and IPv6](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/newhosts-final-Dual.hosts), a [plain list of blocked domains](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/BlacklistedDomains.txt), an [optimized wildcard list](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/cpbl-wildcard-blacklist.txt), and a [dnsmasq blacklist](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/cpbl-dnsmasq.txt) are available as well._

* [**cpbl-abp-list.txt**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/cpbl-abp-list.txt): Is a filter subscription for AdBlock Plus, uBlock Origin, AdGuard and other similar products, derived from my main hosts file, with some *mostly* cosmetic filters to remove overlooked web annoyances. It is much smaller than my hosts file while blocking the same amount of content and more (thanks to wildcard matching and flexible ABP syntax). It is quite effective, and an especially good choice for those who do not have access to the hosts file or configurable DNS-based blocking options. I recommend using this list with AdGuard's Base + Tracking + Annoyances filters^*07*, or EasyList + EasyPrivacy + FanBoy's Annoyance filters^*08*, due to their additional element hiding and exception rule-sets. This list is also available - albeit a bit trimmed down - as a [custom ad-block filter for **Bromite**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/cpbl-bromite.dat).

+ [**pac-done.js**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/pac-done.js): Is a Proxy AutoConfig file^*09* meant to block advertising, tracking, malware servers and such. This file is generated from a slimmed down list of the same reputable sources which my main hosts file uses, and takes advantage of the - rather limited - wildcard matching capabilities which PAC files offer. If you are working under a desktop operating system, this PAC file should serve you well, but those on mobile platforms and older devices will probably fare better with my [MiniList PAC file](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/MiniLists/mini-pac-done.js).

- [**combined-final-win.dat**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final-win.dat) and [**combined-final.p2p**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final.p2p): Are IP block lists suitable for use in torrent clients like Halite for Windows^*10*, or qBitTorrent^*11* and Transmission^*12* under any OS. These lists adhere to [*ipfilter.dat*](http://wiki.amule.org/wiki/Ipfilter.dat_file) and [*PeerGuardian P2P*](https://sourceforge.net/p/peerguardian/wiki/dev-blocklist-format-p2p/) conventions, and will come in especially handy on public trackers (though you can use them confidently on private ones as well). The discerning user may be interested to know that all iBlockList sources have been abandoned for these lists, meaning that the increasingly outdated entries associated with that service, are now a thing of the past. The lists will mitigate a wide gamut of nefarious traffic, being merged from [the same sources](https://gist.github.com/bongochong/46a6b7ac6450d8df4b447432ee4159ad) as the CIDR list described below.

+ [**combined-final.cidr**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/combined-final.cidr): Is a CIDR-format IP block list suitable for use with the IPSet utility, Linux-based routers, a variety of firewalls, file-sharing clients and so on. Basically, it works with anything which accepts CIDR lists. It is compiled from the ad-blocking IP list provided by PGL Yoyo^*13*, a collection of smaller lists from BlockList.de^*14*, FeodoTracker abuse lists^*15*, high confidence Emerging Threats lists^*16*, along with DShield's Top Twenty^*17*, SpamHaus Drop + eDrop^*18*, and a conservative full bogons list.^*19* It also incorporates [a small list of IPs](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/personal-entries.cidr) I've observed malicious activity from, via running public Shoutcast and Icecast servers for extended periods of time. It is likewise deduped + sorted (with overlapping ranges resolved). This list is also available with `deny from` prepended to each line, [**here**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/combined-denied.cidr).

- [**optimized-win.hosts**](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/optimized-win.hosts): Is a hosts file specifically tailored for Windows users, wherein the list is arranged to have seven hosts per line, thus saving space, and preventing the lag that some users experience when they have a large hosts file on that platform. It is culled from the same sources as my main hosts file. While the main hosts file I distribute is meant for all platforms, some Windows users experience a delay when attempting to connect to sites after starting up with a large hosts file in place. This optimized list will resolve that issue for said users. For those who wish to experiment with IPv6 additions, the [IPv4+IPv6 version is here](https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/optimized-win-Dual.hosts).  

***Please feel free to subscribe to or use these lists as is. I use them on all of my personal devices.*** The aforementioned lists also fall under a separate license, the [WTFPL](http://www.wtfpl.net/txt/copying/).

**Finally**, the user-friendly versions of the scripts and utilities that I use to generate these lists, can be set up in one shot [by following the instructions here](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/README.md).  

They include the following:  

+ [**Hosts File Updater**](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-hosts.sh) - Automated and easy-to-configure bash script for updating your hosts file from multiple sources. Pre-configured to work with included templates. Runs properly out of the box. GNU/Linux and other Unix-like operating systems only. There is also a version that updates your hosts file with **both** IPv4 and IPv6 entries [here](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-hosts-dual.sh), and one for IPv6-only [here](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-hosts-six.sh).

* [**IP Block List Updater**](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-ipfilt.sh) - Bash script for automated generation of IP block lists (**CIDR**, **DAT** and **P2P**) from multiple sources, for firewalls, torrent clients etc... Pre-configured and runs out of the box. GNU/Linux and other Unix-like operating systems only. This tool essentially deprecates the legacy [p2p](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-btp2p.sh) and [dat](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-btdat.sh) scripts (which were originally crafted to interface with a once-popular service that has been abandoned for too long now).

+ [**PAC Updater**](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/BLT/update-pac.sh) - A bash script that automatically generates, updates and optionally uploads an ad-blocking PAC file to your server of choice. GNU/Linux and other Unix-like operating systems only.  

These tools are modified versions of the ones I run on my GNU/Linux boxes. I test them regularly. Some of them have dependencies, so make sure you inspect their respective readme files, and that your system has the relevant programs if you plan on using my tools. The GNU/Linux scripts should work on OS X or BSD, so long as dependencies are met. There are [editions for Cygwin](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/BLT/Cygwin) as well. All of my tools clean up after themselves, leaving no extraneous or intermediary files behind.  

If you're a Windows user who is not comfortable with Cygwin, then you may be interested in [**My CWP Utilities Repository**](https://github.com/bongochong/CWP-Utilities), which contains simplified Windows ports of the utilities listed above, compatible with both 32 and 64-bit versions of Windows. They are friendlier versions of my GNU/Linux tools, bundled with the necessary dependencies, distributed with an integrated, easy-to-use, and coherent interface.  

_{**Notice**: Due to resource considerations, I may have to re-initialize this repository without a commit history at some point. 7+ years of commits (this repo was initialized in 2017) make for one huge .git directory, and re-basing + pruning + garbage collection has only done so much. Everything important will always be here, but don't be surprised if you arrive here one day and only see two or three commits in the logs. This started as a purely personal project, and an attempt to replicate the functionality of some very useful software titles, but in the form of bash scripts. It has grown into a good deal more than that, and I am very pleased that others find my efforts useful.}  



# 

^{01. [PGL Yoyo Hosts](http://pgl.yoyo.org/adservers/).} 


^{02. [URL Haus](https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-hosts-online.txt).} 


^{03. [Updated Immortal Malware Domains](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/NoFormatting/MD-ID-Fork.txt).} 


^{04. [AdGuard Sources](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/ABP2Hosts).} 


^{05. [EasyList Sources](https://github.com/bongochong/CombinedPrivacyBlockLists/blob/master/ABP2Hosts/easylist_desktop-hosts.txt).} 


^{06. [Additional](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/NoFormatting) | [Supplementary](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/WindowsTelemetryBlockSupplements) | [Sources](https://github.com/bongochong/CombinedPrivacyBlockLists/tree/master/ABP2Hosts).} 


^{07. [AdGuard](https://kb.adguard.com/en/general/adguard-ad-filters#filters).} 


^{08. [EasyList](https://easylist.to/).} 


^{09. [Proxy Auto-Configuration](https://developer.mozilla.org/en-US/docs/Web/HTTP/Proxy_servers_and_tunneling/Proxy_Auto-Configuration_PAC_file).} 


^{10. [Halite](https://www.fosshub.com/Halite.html).} 


^{11. [qBitTorrent](https://www.qbittorrent.org/).} 


^{12. [Transmission](https://transmissionbt.com/).} 


^{13. [PGL Yoyo PeerBlock](https://pgl.yoyo.org/adservers/iplist.php?ipformat=peerblock&showintro=0&mimetype=plaintext).} 


^{14. [BlockList.de Lists](https://www.blocklist.de).} 


^{15. [FeodoTracker Abuse](https://feodotracker.abuse.ch).} 


^{16. [Emerging Threats](https://www.proofpoint.com/us).} 


^{17. [DShield Lists](https://dshield.org/).} 


^{18. [SpamHaus Lists](https://www.spamhaus.org/drop/).} 


^{19. [Cymru Bogon Reference](https://team-cymru.com/community-services/bogon-reference/).}