Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/elliotwutingfeng/inversion-dnsbl-generator

Generate malicious URL blocklists for DNSBL applications like pfBlockerNG or Pi-hole by scanning various public URL sources using the Safe Browsing API from Google and/or Yandex.
https://github.com/elliotwutingfeng/inversion-dnsbl-generator

aiohttp blocklist dnsbl domains-project domcop fasttld firewall google-safe-browsing icann pfblockerng pi-hole pihole python python3 ray safebrowsing sqlite3 top1m trancolist yandex-api

Last synced: about 2 months ago
JSON representation

Generate malicious URL blocklists for DNSBL applications like pfBlockerNG or Pi-hole by scanning various public URL sources using the Safe Browsing API from Google and/or Yandex.

Host: GitHub
URL: https://github.com/elliotwutingfeng/inversion-dnsbl-generator
Owner: elliotwutingfeng
License: bsd-3-clause
Created: 2021-11-24T11:53:29.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2024-08-20T05:48:00.000Z (5 months ago)
Last Synced: 2024-08-20T08:05:46.339Z (5 months ago)
Topics: aiohttp, blocklist, dnsbl, domains-project, domcop, fasttld, firewall, google-safe-browsing, icann, pfblockerng, pi-hole, pihole, python, python3, ray, safebrowsing, sqlite3, top1m, trancolist, yandex-api
Language: Python
Homepage:
Size: 450 KB
Stars: 20
Watchers: 2
Forks: 6
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

Inversion DNSBL (Domain Name System-based blackhole list) Generator

Table of Contents

Blocklists available for download

URL sources

Safe Browsing API vendors

Requirements

Setup instructions

Getting Started
- Download Google Safe Browsing API hashes
- Download and Identify malicious URLs from Tranco TOP1M

Other Examples

Known Issues

Disclaimer

References

---

## Blocklists available for download

![Total Blocklist URLs](https://tokei-rs.onrender.com/b1/github/elliotwutingfeng/Inversion-DNSBL-Blocklists?label=Total%20Blocklist%20URLS&style=for-the-badge)

You may download the blocklists [here](https://github.com/elliotwutingfeng/Inversion-DNSBL-Blocklists#inversion-dnsbl-domain-name-system-based-blackhole-list-blocklists)

## URL sources

## Safe Browsing API vendors

## Requirements

### System (mandatory)

- Linux or macOS
- Python 3.10+
- Multi-core x86-64 CPU; for Python Ray support
- RAM: At least 32GB
- SSD Storage Space: At least 700GB required to process all URL sources

### Safe Browsing API Access (mandatory)

Choose at least one

- Google: [Obtain a Google Developer API key and set it up for the Safe Browsing API](https://developers.google.com/safe-browsing/v4/get-started)
- Yandex: [Obtain a Yandex Developer API key](https://yandex.com/dev/safebrowsing)

### URL feed access (optional)

- ICANN Zone Files: [Sign up for a ICANN CZDS account](https://czds.icann.org)
- Once registered, turn off email notifications in the user settings (otherwise they will send you hundreds of acknowledgement emails),
then select `Create New Request` on the Dashboard to request for zone file access.

### Uploading blocklists to GitHub (optional)

- [Create a GitHub API Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)

### Download limits

- **ICANN CZDS (Centralized Zone Data Service):** Once every 24 hours per zone file
- **Switch.ch:** Once every 24 hours per zone file

## Setup instructions

`git clone` and `cd` into the project directory

### Declare environment variables

```bash
cp .env-dev .env
```

In `.env`, fill in the following variables

```bash
# Mandatory: At least one of the following Safe Browsing API keys
GOOGLE_API_KEY=
YANDEX_API_KEY=

# Optional: ICANN zone file access
ICANN_ACCOUNT_USERNAME=
ICANN_ACCOUNT_PASSWORD=
# Some registrars will not accept your request reason unless you include your Name, Email, IP Address, Physical Address (Building, Street, Postcode etc.), and Phone Number
ICANN_REQUEST_REASON='Detection of potentially malicious domains for cybersecurity research. Name: _ Email: _ IP Address: _ Physical Address: _ Phone Number: _'

# Optional: Upload generated blocklists to your GitHub repository
GITHUB_ACCESS_TOKEN=
BLOCKLIST_REPOSITORY_NAME=
```

### Install dependencies

According to [PEP 668](https://peps.python.org/pep-0668), use of a virtual environment is [strongly recommended](https://packaging.python.org/en/latest/specifications/externally-managed-environments) as of 2023.

```bash
python3 -m venv venv
venv/bin/python3 -m pip install --upgrade pip
venv/bin/python3 -m pip install -r requirements.txt
```

### Download Domains Project URLs (optional)

```bash
# Dataset size ~49Gb
cd ../
git clone https://github.com/tb0hdan/domains.git
cd domains
git lfs install # you will need to install Git LFS first (https://git-lfs.github.com)
```

Edit `unpack.sh` and remove `combine` from the last line, then run:

```bash
./unpack.sh
```

## Getting Started

### Download Google Safe Browsing API hashes

> :warning: As of 4 August 2023, the following command will make around 9000 calls (exact number depends on number of hashes in Google's dataset) to Google Safe Browsing API. As the daily limit is 10,000 calls, `--update-hashes` should be run no more than once every 24 hours.

```bash
venv/bin/python3 main.py --update-hashes --vendors google
```

### Download and Identify malicious URLs from Tranco TOP1M

- :heavy_check_mark: Add Tranco TOP1M URLs to database
- :heavy_check_mark: Identify malicious URLs from database using Safe Browsing API hashes, and generate a blocklist
- :heavy_check_mark: Update database with latest malicious URL statuses
- :memo: Sources: **Tranco TOP1M**
- :shield: Vendors: **Google**

```bash
venv/bin/python3 main.py --fetch-urls --identify-malicious-urls --sources top1m --vendors google
```

## Other Examples

### Download DomCop TOP10M URLs

- :heavy_check_mark: Add DomCop TOP10M URLs to database (no blocklist will be generated)
- :memo: Sources: **DomCop TOP10M**
- :shield: Vendors: **Not Applicable**

```bash
venv/bin/python3 main.py --fetch-urls --sources top10m
```

### Download and Identify malicious URLs from all sources

> :warning: Requires at least 700GB free space.
>
> :information_source: If you have not downloaded any Safe Browsing API hashes yet, add the `--update-hashes` flag to the following command.

- :heavy_check_mark: Add URLs from all sources to database
- :heavy_check_mark: Identify malicious URLs from database using Safe Browsing API hashes, and generate a blocklist
- :heavy_check_mark: Update database with latest malicious URL statuses
- :memo: Sources: Everything
- :shield: Vendors: **Google**

```bash
venv/bin/python3 main.py --fetch-urls --identify-malicious-urls --vendors google
```

### Retrieve URLs marked as malicious from past scans from database

- :heavy_check_mark: Retrieve URLs with malicious statuses (attained from past scans) from database, and generate a blocklist
- :memo: Sources: **DomCop TOP10M**, **Domains Project**
- :shield: Vendors: **Google**

```bash
venv/bin/python3 main.py --retrieve-known-malicious-urls --sources top10m domainsproject --vendors google
```

### Display help message

```bash
venv/bin/python3 main.py --help
```

## Known Issues

- Yandex Safe Browsing Update API appears to be unserviceable. Yandex Technical support has been notified.

## Disclaimer

- This project is not sponsored, endorsed, or otherwise affiliated with Google and/or Yandex.

- Google works to provide the most accurate and up-to-date information about unsafe web resources. However, Google cannot guarantee that its information is comprehensive and error-free: some risky sites may not be identified, and some safe sites may be identified in error.

- URLs detected with the Safe Browsing API usually have a malicious validity period of about 5 minutes. As the blocklists are updated only once every 24 hours, the blocklists must not be used to display user warnings.

**More information on Google Safe Browsing API usage limits:**

## References

-
-
-
-
-