{"id":15148412,"url":"https://github.com/michalszmidt/hctl","last_synced_at":"2025-08-24T22:25:13.966Z","repository":{"id":176386182,"uuid":"657569951","full_name":"michalszmidt/hctl","owner":"michalszmidt","description":"Ultimate tool to manage (transform, merge, remove duplicates, validate) hostlists written in rust!","archived":false,"fork":false,"pushed_at":"2024-07-22T18:12:57.000Z","size":184,"stargazers_count":1,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-09-27T13:03:32.619Z","etag":null,"topics":["adguard-dns","adguard-home","adguardhome","dns-blocking","dns-blocklists","hostfile","hostfiles","hostlist","pi-hole"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michalszmidt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-06-23T10:57:45.000Z","updated_at":"2023-08-20T23:26:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"3a77ad8a-c019-4898-846e-e4378307f7b8","html_url":"https://github.com/michalszmidt/hctl","commit_stats":null,"previous_names":["michalszmidt/hctl"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michalszmidt%2Fhctl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michalszmidt%2Fhctl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michalszmidt%2Fhctl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michalszmidt%2Fhctl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michalszmidt","download_url":"https://codeload.github.com/michalszmidt/hctl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219866820,"owners_count":16555824,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adguard-dns","adguard-home","adguardhome","dns-blocking","dns-blocklists","hostfile","hostfiles","hostlist","pi-hole"],"created_at":"2024-09-26T13:03:37.676Z","updated_at":"2024-10-10T02:23:25.802Z","avatar_url":"https://github.com/michalszmidt.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n    __            __     __\n   / /_   _____  / /_   / /\n  / __ \\ / ___/ / __/  / / \n / / / // /__  / /_   / /  \n/_/ /_/ \\___/  \\__/  /_/   \n```\n\u003cdiv\u003e\n  \u003cimg alt=\"Repo Size\" src=\"https://img.shields.io/github/repo-size/michalszmidt/hctl\" /\u003e\n  \u003cimg alt=\"Lines of code\" src=\"https://sloc.xyz/github/michalszmidt/hctl?category=code\" /\u003e\n  \u003cimg alt=\"Last Commit\" src=\"https://img.shields.io/github/last-commit/michalszmidt/hctl\" /\u003e\n  \u003cimg alt=\"Assets Downloads\" src=\"https://img.shields.io/github/downloads/michalszmidt/hctl/total\" /\u003e\n\u003c/div\u003e\n\n# About\n\n\nHostlists tools cli `hctl` is app written in rust to manage (merge, transform, clear from comments etc.) your hostlists from diffrent formats to selected by you!\n\n- [Wiki](https://github.com/michalszmidt/hctl/wiki)\n- [Usage](https://github.com/michalszmidt/hctl/wiki/Usage)\n\n# Prerequisites, dependencies\n### `openssl`\n\n- **\\*nix systems (linux, BSD, mac)**: should be already there, if not install it from your package repo like `brew install openssl@3`\n- **windows**: install from `choco` or `vcpkg` or whatever other way that won't welcome viruses.\n\nNote that you can get rid of openssl dependency if you want, by building from source with modified dependencies, see [building from source](https://github.com/michalszmidt/hctl/wiki/Developer#building-from-source).\n\n# Downloads\n## As a package\n\n[![Packaging status](https://repology.org/badge/vertical-allrepos/hctl.svg)](https://repology.org/project/hctl/versions)\n\n![OBS package build status](https://img.shields.io/obs/home%3Amichalszmidt/hctl/openSUSE_Leap_15.6_standard/x86_64?label=OBS%3A%20OpenSUSE%20Leap%2015.6\u0026link=https%3A%2F%2Fsoftware.opensuse.org%2Fdownload%2Fpackage%3Fpackage%3Dhctl%26project%3Dhome%253Amichalszmidt)\n\n![OBS package build status](https://img.shields.io/obs/home%3Amichalszmidt/hctl/openSUSE_Tumbleweed/x86_64?label=OBS%3A%20OpenSUSE%20Tumbleweed\u0026link=https%3A%2F%2Fsoftware.opensuse.org%2Fdownload%2Fpackage%3Fpackage%3Dhctl%26project%3Dhome%253Amichalszmidt)\n\nAvailable:\n- Alpine Linux edge: `apk add hctl --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing/` (from testing repository)\n- OpenSUSE Linux 15.6/Tumbleweed: `opi hctl` (using opi community repo helper)\n- Arch Linux (based): `yay -Su hctl` (using AUR helper)\n\nPackaging progress [issue](https://github.com/michalszmidt/hctl/issues/9)\n\n## [From release page](https://github.com/michalszmidt/hctl/releases/latest)\n\n| :exclamation:  BUILD REGRESSION   |\n|-----------------------------------|\n\nOpenBSD and DragonflyBSD doesn't have rust tier 1 support and their repo version is too old to compile on stable\n\n\u003ctable\u003e\n\u003ctr\u003e\u003cth\u003ePlatform\u003c/th\u003e\u003cth\u003eDownloads\u003c/th\u003e\u003cth\u003eBuild Status\u003c/th\u003e\u003c/tr\u003e\n\u003ctr\u003e\u003ctd\u003e\n\n| OS |\n|----------|\n| Linux glibc |\n| Linux musl  |\n| MacOS X |\n| Windows |\n| FreeBSD |\n| NetBSD |\n| Illumos |\n\n\u003c/td\u003e\u003ctd\u003e\n    \n| amd64 | aarch64 |\n|-------|---------|\n|[app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-linux-glibc-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-linux-glibc-amd64) | [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-linux-glibc-aarch64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-linux-glibc-aarch64) |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-linux-musl-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-linux-musl-amd64) | [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-linux-musl-aarch64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-linux-musl-aarch64) |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-macosx-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-macosx-amd64) | [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-macosx-silicon), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-macosx-silicon) |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-windows-amd64.exe), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-windows-amd64.exe.txt) | [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-windows-aarch64.exe), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-windows-aarch64.exe.txt) |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-freebsd-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-freebsd-amd64) | NSU |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-netbsd-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-netbsd-amd64) | NSU |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-illumos-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-illumos-amd64) | NSU |\n\u003c/td\u003e\u003ctd\u003e\n\n| amd64 | aarch64 |\n|-------|---------|\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_linux_glibc_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | [![Build Status](https://api.cirrus-ci.com/github/michalszmidt/hctl.svg?task=LinuxGlibcAarch64)](https://cirrus-ci.com/github/michalszmidt/hctl) |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_linux_musl_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | [![Build Status](https://api.cirrus-ci.com/github/michalszmidt/hctl.svg?task=LinuxMuslAarch64)](https://cirrus-ci.com/github/michalszmidt/hctl) |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_macosx_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | [![Build Status](https://api.cirrus-ci.com/github/michalszmidt/hctl.svg?task=MacosxSilicon)](https://cirrus-ci.com/github/michalszmidt/hctl) |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_windows_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_windows_aarch64.yml)](https://github.com/michalszmidt/hctl/actions) |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_freebsd_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | NSU |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_netbsd_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | NSU |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_illumos_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | NSU |\n\n\u003c/td\u003e\u003c/tr\u003e \u003c/table\u003e \n\n\u003c!-- \n| DragonflyBSD |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-dragonflybsd-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-dragonflybsd-amd64) | N/A |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_dragonflybsd_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | N/A |\n\n| OpenBSD |\n| [app](https://github.com/michalszmidt/hctl/releases/download/latest/hctl-openbsd-amd64), [sha256](https://github.com/michalszmidt/hctl/releases/download/latest/sha256-hctl-openbsd-amd64) | NSU |\n| [![Build Status](https://img.shields.io/github/actions/workflow/status/michalszmidt/hctl/release_openbsd_amd64.yml)](https://github.com/michalszmidt/hctl/actions) | NSU |\n--\u003e\n\n- NSU - Not Set Up Yet, may (or not) appear in the future\n\n# Features\n\n- [x] Merge hostlist\n- [x] Remove duplicates\n- [x] Remove commented lines (default behaviour)\n- [x] Fetch lists directly from network (https)\n- [x] YAML config\n- [x] Parallel processing\n- [x] Rejected lines may be saved with information about source list\n- [x] Support for Windows encoded text files (line ending with `\\n\\r`)\n- [x] Output to stdout to be used with unix pipe stream control (partial support)\n- [x] Whitelists (enumerated, from external lists file/https)\n- [x] Whitelists including subdomains\n- [x] Record existance validation from DNS servers (unencrypted and over tls)\n- [x] User-defined DNS and DOT Resolvers\n- [x] Could use diffrent input types (like dnsmasq, bind) in yaml config\n- [x] Interacive status indication\n- [x] Settings yaml could be fetched from http(s)\n- [ ] Auto scheme recognition\n- [ ] YAML has full settings (remote sources, whitelist , ~args from cli~)\n- [ ] Subdomains family recognition, save with subdomain order (like adaway does)\n- [ ] Fault-tolerance, better error handling\n- [ ] Deep rejected analysis\n- [ ] Quiet option\n- [ ] Progressbar on save\n\n\nJump [here](https://github.com/michalszmidt/hctl/wiki/Manual#supported-formats) to see supported input and output formats\n\nJump [here](https://github.com/michalszmidt/hctl/issues/1) to feature-request issue.\n\n# [Usage](https://github.com/michalszmidt/hctl/wiki/Usage)\n\n\n```bash\n# hctl -D --help\nManipulate on domains\n\nUsage: hctl {domain|--domain|-D} [OPTIONS]\n\nOptions:\n  -p, --path \u003cpath\u003e          Path to file to be read [path without quotes]\n  -o, --out \u003cout\u003e            Path to the out file [stdout/path without quotes]\n  -z, --optimize \u003coptimize\u003e  Optimize for memory or speed, default: memory [memory/speed]\n  -c, --config \u003cconfig\u003e      Path to config [without qoutes]\n  -m, --mode \u003cmode\u003e          Process single or multiple lists [single/folder/config/url]\n  -i, --intro \u003cintro\u003e        Whether append intro \"Assembled From:...\" to out file from config [true/false]\n  -r, --rejected \u003crejected\u003e  Whether save rejected to ./rejected.txt [true/false]\n  -f, --format \u003cformat\u003e      Type of out format [dnsmasq/loopback/empty/linewise/bind/snort/unbound/machosts/hostperm1/junkbuster/littlesnitch/pdnsd]\n  -t, --validate \u003cvalidate\u003e  (Non-intrusive): Verbose validate your records in single file issuing dns-over-tls query in hardcoded servers [yes/no]\n  -d, --dns \u003cdns\u003e            Validate your processed records issuing dns query in predefined servers, config mode allows custom ones [yes/no]\n  -a, --pattern \u003cpattern\u003e    Input format of list [hosts/dnsmasq]\n  -h, --help                 Print help\n```\n\n# Magic behind the scene\n\nThere is heavy usage of rust iterators to enable lazy-reading\n\nProcessing is done by firstly removing addreses characteristic for host file like 127.0.0.1 and comments after the records. Next tool removes all whitespace characters and non-urls using regex.\nSee [rules.rs](https://github.com/michalszmidt/hctl/blob/master/src/logic/rules.rs) if you want to embed similar solution in your app.\n\n## License\n[BSD-3-clause](https://github.com/michalszmidt/hctl/blob/main/LICENSE)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichalszmidt%2Fhctl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichalszmidt%2Fhctl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichalszmidt%2Fhctl/lists"}