https://github.com/asifbacchus/adserver-list-update
bash-script to download updated adserver list files, deduplicate them and format the combined output file for use with DNSMASQ external hosts file.
https://github.com/asifbacchus/adserver-list-update
Last synced: 3 months ago
JSON representation
bash-script to download updated adserver list files, deduplicate them and format the combined output file for use with DNSMASQ external hosts file.
- Host: GitHub
- URL: https://github.com/asifbacchus/adserver-list-update
- Owner: asifbacchus
- Created: 2018-05-17T08:15:40.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-05-21T15:52:07.000Z (almost 7 years ago)
- Last Synced: 2024-12-30T10:45:41.307Z (5 months ago)
- Language: Shell
- Size: 13.7 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# adserver-list-update
bash-script to download updated adserver list files, deduplicate them and format
the combined output file for use with DNSMASQ external hosts file.
This script downloads several popular, well maintained, lists of known adservers
then amalagates those lists into one large list, removes duplicate entries and
formats the whole thing to point to configurable IP4 and IP6 addresses in a
format usable as a general hosts file or with DNSMASQ as an external hosts file.
Let me first say this is made for my personal use on my ad-blocking DNSMASQ
server (I'll link to my blog detailing full setup later). I don't script often
and the whole ad-blocking server is my way of better learning linux. That being
said, ANY suggestions (with explanations, please, so I can learn!) would be
appreciated. Otherwise, if this helps you too, then awesome!
Obviously no warranties or guantees are made by me in any form nor do I accept
any responsibility if this destroys your computer, your personal life, your
country or the world in general.
Script overview:
Variables:
IPv4 redirect address ($ipv4addr) - i.e. your webserver or 0.0.0.0
IPv6 redirect address ($ipv6addr) - i.e. your webserver or ::
Script's location ($dir) - this is auto-set to the current working directory
Location for downloaded files ($locationSourcefiles) - deleted after use
Location for working temp files ($locationWorkingfiles) - deleted after use
Location for output file ($listpath) - where final output file should be saved
This script downloads a set of known adserver addresses/hosts files from the
following sources:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://mirror1.malwaredomains.com/files/justdomains
http://sysctl.org/cameleon/hosts
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://hosts-file.net/ad_servers.txt
The files are then parsed individually as needed to remove comments, interface
addresses and target IP addresses. A working copy of just domain names is saved
in the $locationWorkingfiles directory.
All individual lists are combined into one list, sorted alphabetically for ease
of reading and duplicates are removed.
The $ipv4addr is prepended to one copy of the combined list and the $ipv6addr
is prepended to a second copy of the same list. The two lists are then merged
and sorted so that each host has an IP4 entry immediately followed by the
corresponding IP6 entry. This is the final list and is saved in $listpath as
'adblock.dnsmasq'.
The script then deletes the $locationSourcefiles and $locationWorkingfiles
directories and their contents so that future updates start 'clean'.
Finally, DNSMASQ is restarted via systemctl so it reads the updated list file
that was generated by this script.
To Do:
- Make it prettier
- Add checking for root priviledges
- Add entries into syslog? separate logfile?
- Add chron entries to run this script at user-specified interval?
Let me know if you have any suggestions!
Thanks for checking out this script :-)