{"id":49227291,"url":"https://github.com/cbuijs/aggrip-go","last_synced_at":"2026-05-03T17:01:31.749Z","repository":{"id":353295516,"uuid":"1218797616","full_name":"cbuijs/aggrip-go","owner":"cbuijs","description":"GO version of the aggrip tools","archived":false,"fork":false,"pushed_at":"2026-04-24T09:27:01.000Z","size":90,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-24T09:48:12.808Z","etag":null,"topics":["allowlist","blacklist","blocklist","cidr","dns","domain","ip","ip-address","tools","whitelist"],"latest_commit_sha":null,"homepage":"https://github.com/cbuijs/aggrip","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cbuijs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-23T08:20:37.000Z","updated_at":"2026-04-24T09:26:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cbuijs/aggrip-go","commit_stats":null,"previous_names":["cbuijs/aggrip-go"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/cbuijs/aggrip-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbuijs%2Faggrip-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbuijs%2Faggrip-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbuijs%2Faggrip-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbuijs%2Faggrip-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cbuijs","download_url":"https://codeload.github.com/cbuijs/aggrip-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbuijs%2Faggrip-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32577126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["allowlist","blacklist","blocklist","cidr","dns","domain","ip","ip-address","tools","whitelist"],"created_at":"2026-04-24T09:02:03.640Z","updated_at":"2026-05-03T17:01:31.734Z","avatar_url":"https://github.com/cbuijs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aggrip-go\n\nHigh-performance, low-latency, and secure Go utilities for optimizing, deduplicating, and managing massive DNS and IP-based blocklists and allowlists. Built for enterprise-grade public network filtering.\n\n## Included Tools\n\n### 1. clean-dom\n\n**Enterprise-grade DNS Blocklist Optimizer** Consolidates multiple DNS blacklists and whitelists, deduplicates subdomains using an O(N log N) reverse-string tree sort, translates Punycode natively, and dynamically routes Adblock rules.\n\n**Key Features:**\n\n* **Concurrent Ingestion:** Streams massive lists via HTTP/HTTPS or local paths using Go concurrency.\n\n* **Format Autodetection:** Upfront heuristic format detection (hosts, adblock, routedns, squid, domain).\n\n* **Adblock Parsing:** Extracts modifiers (e.g., `$denyallow`) and translates Punycode (IDNA) automatically.\n\n* **Native Validation \u0026 Dictionaries:** Fast byte-level payload validation. Strict RFC checking securely drops invalid anomalies (e.g., all-numeric TLDs). Features built-in, zero-dependency validation dictionaries for **IANA** (1400+ TLDs) and **OpenNIC** to strictly guarantee output sanitization. Support for decentralized/unregistered roots (Handshake) is configurable.\n\n* **Tree Deduplication:** Rapid O(N log N) deduplication via reverse string sorting (automatically drops redundant subdomains if a parent is already blocked). *Note: If a TLD like `com` is ingested via the `--allow-tld` flag, it acts as an apex parent and correctly collapses all subdomains (e.g. `example.com`) beneath it.*\n\n* **Multiple Outputs:** Generates ready-to-use configs for `domain`, `hosts`, `adblock`, `dnsmasq`, `unbound`, `rpz`, `routedns`, and `squid`.\n\n* **Hosts Output Compression:** Includes optional `--compress-hosts[=\u003cnum\u003e]` capacity routing to collapse multiple domains securely onto a single IP mapping (e.g., `0.0.0.0 dom1 dom2...`). Defaults to 10 nodes per line when the flag is triggered.\n\n**Usage Example:**\n\n```bash\n# Standard Output Formatting\nclean-dom -b https://example.com/ads.txt -a local-allow.txt -o unbound --out-blocklist unbound-filter.conf --valid-tlds iana,opennic -v\n\n# Compressed HOSTS File Routing (Output mapping up to 15 domains per single IP line)\nclean-dom -b https://example.com/ads.txt -o hosts --compress-hosts=15 --out-blocklist filter.hosts -v\n```\n\n### 2. clean-ip\n\n**Enterprise-grade IP Blocklist Optimizer** High-speed CIDR, IP, and range aggregator. Cross-references against allowlists, collapses redundant subnets into optimal CIDR blocks, and mathematically punches holes for exclusions.\n\n**Key Features:**\n\n* **High-Speed I/O:** Zero-allocation tokenization (`strings.FieldsFunc`) and 1MB buffered I/O stream writers for massive firewall datasets.\n\n* **Mathematical Hole-Punching:** Safely fractures and excludes allowlisted IPs/CIDRs from larger blocked supernets, avoiding firewall bypasses.\n\n* **Range Summarization:** Autoconverts spaced (`1.1.1.1 1.1.1.5`) or dashed (`1.1.1.1-1.1.1.5`) IP ranges into optimal CIDR blocks natively.\n\n* **Subnet Collapsing:** Sorts and aggregates overlapping CIDRs into the largest possible subnets to shrink firewall state tables.\n\n* **Firewall Ready:** Exports to `cidr`, `netmask`, `range`, `cisco`, `iptables`, `mikrotik`, and `padded`.\n\n**Usage Example:**\n\n```bash\nclean-ip -b drop.txt -a allow.txt -o iptables --out-blocklist rules.v4 -v\n```\n\n### 3. aggrip\n\n**High-Speed IP to CIDR Aggregator** A streamlined, high-performance pipeline utility to ingest raw lists of IP addresses and CIDR blocks, outputting a merged and mathematically optimized CIDR list.\n\n**Key Features:**\n\n* **Stream Processing:** Designed for standard UNIX pipes (`STDIN`/`STDOUT`) with optional file flags (`-i`, `-o`) for massive datasets.\n\n* **O(N log N) Compression:** Merges redundant or contiguous CIDR subnets using a high-speed sorting-stack algorithm.\n\n* **Zero-Allocation Parsing:** Utilizes Go's native `net/netip` package for highly efficient, memory-safe IP and prefix evaluation.\n\n* **Strict Boundary Enforcement:** Optional strict mode (`-s`) automatically drops invalid CIDRs instead of implicitly truncating dirty host bits.\n\n* **Dual Stack Independence:** Processes and segregates IPv4 and IPv6 streams mathematically to prevent cross-boundary collisions.\n\n**Usage Example:**\n\n```bash\n# Standard UNIX piping\ncat raw_ips.txt | aggrip \u003e optimized_cidrs.txt\n\n# Direct file I/O with strict boundary enforcement\naggrip -i raw_ips.txt -o optimized_cidrs.txt -s -v\n```\n\n### 4. undup\n\n**Blazing Fast Binary-Level Domain Deduplicator** A highly specialized, low-latency deduplication engine that removes redundant subdomains when their parent domains exist in the same feed.\n\n**Key Features:**\n\n* **Zero-Copy Byte Parsing:** Reads entirely from bytes arrays without heavy string allocations to maximize parsing throughput.\n* **Concurrent Reversals:** Shards string reversal workloads across all available CPU cores automatically for parallel processing.\n* **Strict Validation Bypass:** High-speed inline structural validation completely circumvents slow regex engine overhead.\n* **Flexible I/O:** Supports standard UNIX piping or direct high-speed file streams.\n\n**Usage Example:**\n\n```bash\n# Fast deduplication with UNIX pipes\ncat domains.txt | undup \u003e unique_domains.txt\n\n# File I/O with less-strict parsing\nundup -i mixed_domains.txt -o clean_domains.txt -l -v\n```\n\n## Building from Source\n\n```bash\n# Change to main parrent/root directory\ncd aggrip-go\n\n# Build aggrip\ngo build -ldflags=\"-s -w\" -o aggrip ./aggrip\n\n# Build clean-dom\ngo build -ldflags=\"-s -w\" -o clean-dom ./clean-dom\n\n# Build clean-ip\ngo build -ldflags=\"-s -w\" -o clean-ip ./clean-ip\n\n# Build undup\ngo build -ldflags=\"-s -w\" -o undup ./undup\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbuijs%2Faggrip-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcbuijs%2Faggrip-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbuijs%2Faggrip-go/lists"}