Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/shizunge/blocky-lists-updater

Update lists without restarting blocky DNS.
https://github.com/shizunge/blocky-lists-updater
dns self-hosted updater
Last synced: 2 months ago
JSON representation
Update lists without restarting blocky DNS.
Host: GitHub
URL: https://github.com/shizunge/blocky-lists-updater
Owner: shizunge
License: gpl-3.0
Created: 2024-01-21T04:27:44.000Z (11 months ago)
Default Branch: main
Last Pushed: 2024-10-08T04:24:10.000Z (3 months ago)
Last Synced: 2024-10-12T06:16:29.852Z (3 months ago)
Topics: dns, self-hosted, updater
Language: Shell
Homepage:
Size: 88.9 KB
Stars: 7
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

        # blocky lists updater

[![Release](https://img.shields.io/github/release/shizunge/blocky-lists-updater.svg?label=Release)](https://github.com/shizunge/blocky-lists-updater/releases/latest)

[![License](https://img.shields.io/badge/License-GPLv3-blue)](https://github.com/shizunge/blocky-lists-updater/blob/main/LICENSE)

[![Image Size](https://img.shields.io/docker/image-size/shizunge/blocky-lists-updater/latest.svg?label=Image%20Size)](https://hub.docker.com/r/shizunge/blocky-lists-updater)

[![Docker Pulls](https://img.shields.io/docker/pulls/shizunge/blocky-lists-updater.svg?label=Docker%20Pulls&logo=Docker)](https://hub.docker.com/r/shizunge/blocky-lists-updater)

[![Build](https://img.shields.io/github/actions/workflow/status/shizunge/blocky-lists-updater/on-push.yml?label=Build&branch=main&logo=GitHub)](https://github.com/shizunge/blocky-lists-updater/actions/workflows/on-push.yml)

[![CodeFactor Grade](https://img.shields.io/codefactor/grade/github/shizunge/blocky-lists-updater?label=CodeFactor&logo=CodeFactor)](https://www.codefactor.io/repository/github/shizunge/blocky-lists-updater)

[Blocky lists updater](https://github.com/shizunge/blocky-lists-updater) updates lists without restarting [blocky](https://0xerr0r.github.io/blocky/) DNS.

## Introduction

[blocky](https://github.com/0xERR0R/blocky) is a DNS proxy that blocks DNS queries based on external lists. It reads blocking lists and allowing list from a configuration file. Every time we want to add a list or a domain to the blocking or allowing lists, we must edit the configuration file, then restart blocky. This may lead to DNS down.

[Block lists updater](https://github.com/shizunge/blocky-lists-updater) runs as a companion service, allowing you to edit lists and domains in lists without restarting blocky.

## Usage

You can run *lists updater* via Docker, see the [example](./examples) `docker-compose.yml` files.

It is a set of scripts, so you can also run it on the host as well `.src/entrypoint.sh`. It starts a infinite loop and watches the change of *sources*.

## How it works

* *Lists updater* downloads and concatenates multiple lists into a single file based on a *source* file that contains URLs to tell where to download lists.

* *Lists updater* watches changes of the *source* files and/or runs downloading periodically.

* *Lists updater* also watches files of lists of domains. No downloads in this case.

* *Lists updater* ships with a [static file server](https://github.com/static-web-server/static-web-server), allowing blocky to read the downloaded files.

* *List updater* sends [refresh requests](https://0xerr0r.github.io/blocky/swagger.html#operation--lists-refresh-post) to blocky.

*Lists updater* is a set of shell scripts. The entrypoint is the `main` function inside `entrypoint.sh`. You can run *Lists updater* as a container as well, for example

```

docker run -d -p 8080:8080 --env "BLU_BLOCKY_URL=http://blocky:4000" -v ./my_sources:/sources -v ./my_domains:/web/watch shizunge/blocky-lists-updater

```

### Without *lists updater*

Here is a snippet of a blocky configuration file. We use this as an example to demostarte how to translate an existing blocky configuration to *lists updater*.

```

blocking:

  denylists:

    group-one:

      - https://github.com/StevenBlack/hosts

      - https://github.com/Perflyst/PiHoleBlocklist

    group-two:

      - |

        blocked.example.com

        blocked2.example.com

  clientGroupsBlock:

    default:

      - group-one

      - group-two

```

### Using *lists updater* to manage lists

Create a new *source* file for `group-one`, which contains the lists, for example `/sources/group-one.txt`.

```

# Comments are supported

https://github.com/StevenBlack/hosts

https://github.com/Perflyst/PiHoleBlocklist

```

Then You modify the blocky configuration file for `group-one` to read the aggregate list generated by *Lists updater* via the static file server from *lists updater* instead of two separated lists. The file name of the aggregate list will be same as the *source* file, but it goes to the destination folder e.g. `/web/downloaded`.

```

blocking:

  denylists:

    group-one:

      - http://lists-updater:8080/downloaded/group-one.txt

```

Start *Lists updater*, it reads the *source* file, downloads contents of these two lists, then concatenates them to a single file. The result file will have the same name `group-one.txt`, but it will go to the destination folder e.g. `/web/downloaded`. The static file server will serves files under `/web` for blocky to read.

For now on, if you want to add or remove lists for `group-one`, you no longer need to modify the blocky configuration file. Instead you edit the *source* file `/sources/group-one.txt`. *Lists updater* watches the changes of the *source* files, and starts new downloads following refresh requests to blocky. *Lists updater* can also download the lists periodically.

### Using *lists updater* to manage domains

*Lists updater* also watches files that contain lists of domains besides watching the *source* files. In this case, you maintain local copies of the lists of domains, no download occurs.

Create a new file for lists of domains, e.g. `/web/watch/group-two.txt`

```

blocked.example.com

blocked2.example.com

```

Then You modify the blocky configuration file for `group-two` to read this file via the static file server.

```

blocking:

  denylists:

    group-two:

      - http://lists-updater:8080/watch/group-two.txt

```

When you add or remove domains from `group-two.txt`, *lists updater* watches the changes and sends a refresh request to blocky.

## Configurations

You can configure the most behaviors of *lists updater* via environment variables.

| Environment Variable  | Default |Description |

|-----------------------|---------|------------|

| BLU_BLOCKY_URL            |       | Enable [refresh request](https://0xerr0r.github.io/blocky/swagger.html#operation--lists-refresh-post) to blocky. It should be the base URL of blocky. Use an empty string to disable refresh requests. Limitation: the updater will send requests to a single blocky instance today. |

| BLU_DESTINATION_FOLDER    | /web/downloaded | The location of aggregate lists. This should be under `BLU_WEB_FOLDER` thus blocky can read the files via the static file server. |

| BLU_INITIAL_DELAY_SECONDS | 0     | Delay in seconds before the first download. |

| BLU_INTERVAL_SECONDS      | 86400 | Interval between two downloads. Set to 0 disable periodically downloads, then downloads will occur only at *source* changes. |

| BLU_LOG_LEVEL             | INFO  | Control how many logs generated by Gantry. Valid values are `NONE`, `ERROR`, `WARN`, `INFO`, `DEBUG`. |

| BLU_NODE_NAME             |       | Add node name to logs. |

| BLU_NOTIFICATION_APPRISE_URL |    | Enable notification after each refresh with [apprise](https://github.com/caronc/apprise-api). This must point to the notification endpoint (e.g. `http://apprise:8000/notify`). Use an empty string to disable notification. |

| BLU_POST_DOWNLOAD_CMD     |       | A command or function running after downloading a list. The first argument will be the path to the downloaded file, i.e. your command will be `eval` as `your_command `. The command will apply to all downloaded lists. This can be used to fix problems in the lists before the upstream maintainer fixing it. |

| BLU_SOURCES_FOLDER | /sources   | The location to read sources lists. The files contains URLs where to download the lists of domains. Use an empty string to disable watching. |

| BLU_WATCH_FOLDER   | /web/watch | The location of user defined lists of domains. *lists updater* watches changes in this folder and sends refresh requests to blocky. This should be under `BLU_WEB_FOLDER` thus blocky can read the files via the static file server. Use an empty string to disable watching. |

| BLU_WEB_FOLDER     | /web       | The location that the static file server serves. Use an empty string to disable the file server. |

| BLU_WEB_PORT       | 8080       | The port that static file server listens to. |

## Caveats

* Need a third party file editor.

    Blocky lists updater bases on a set of shell scripts. It does not provide any UI to edit lists. You need an additional service for lists editing. See the [examples](./examples).

* No group management

    You still need to modify the blocky configuration file to edit groups.

## Contacts

If you have any problems or questions, please contact me through a [GitHub issue](https://github.com/shizunge/blocky-lists-updater/issues).

## Related projects

You may also like my [blocky Postgresql dashboard](https://github.com/shizunge/blocky-postgresql).