{"id":51052222,"url":"https://github.com/jaldertech/asg","last_synced_at":"2026-06-22T18:01:10.425Z","repository":{"id":342886039,"uuid":"1175526788","full_name":"jaldertech/asg","owner":"jaldertech","description":"Aldertech Storage Governor — Intelligent health monitoring, capacity planning, and scrub scheduling for BTRFS RAID pools with mismatched drives.","archived":false,"fork":false,"pushed_at":"2026-03-07T21:19:01.000Z","size":65,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-10T19:10:18.373Z","etag":null,"topics":["automation","btrfs","data-integrity","homelab","python","raid1","raspberry-pi","self-hosted","server-monitoring","storage-management","sysadmin"],"latest_commit_sha":null,"homepage":"https://aldertech.uk","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaldertech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-07T20:41:36.000Z","updated_at":"2026-03-08T01:36:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jaldertech/asg","commit_stats":null,"previous_names":["jaldertech/asg"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/jaldertech/asg","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaldertech%2Fasg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaldertech%2Fasg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaldertech%2Fasg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaldertech%2Fasg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaldertech","download_url":"https://codeload.github.com/jaldertech/asg/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaldertech%2Fasg/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34659896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","btrfs","data-integrity","homelab","python","raid1","raspberry-pi","self-hosted","server-monitoring","storage-management","sysadmin"],"created_at":"2026-06-22T18:00:40.006Z","updated_at":"2026-06-22T18:01:10.420Z","avatar_url":"https://github.com/jaldertech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/jaldertech/asg/main/assets/logo.png\" alt=\"Aldertech Logo\" width=\"200\"/\u003e\n\u003c/p\u003e\n\n# ASG (Aldertech Storage Governor)\n\n\u003e **\"Why does `df -h` say I have 2.6TB free but BTRFS thinks it's 3.5TB?\"**\n\u003e\n\u003e Because `df` is wrong — and nobody else fixes this.\n\nASG (Aldertech Storage Governor) is a lightweight, zero-dependency Python tool that provides accurate health monitoring, capacity planning, and intelligent scrub scheduling for BTRFS RAID1 pools — especially those with **mismatched drive sizes**.\n\nIf you've ever mixed a 4TB drive with two 1TB drives and wondered why your free space numbers don't add up, ASG is for you.\n\n---\n\n## The Problem\n\nBTRFS RAID1 with mismatched drives is powerful but poorly served by standard tools:\n\n- **`df -h` is mathematically wrong.** It doesn't account for RAID1 pairing constraints on mismatched drives.\n- **`btrfs scrub` is fire-and-forget.** The standard approach (`btrfs scrub start -Bd`) blocks indefinitely with zero awareness of system load — your media server stutters while the scrub runs.\n- **Metadata concentration is invisible.** BTRFS may silently place all metadata chunks on your two smallest drives. If either fails, the entire pool is unrecoverable. No standard tool warns you about this.\n- **No capacity prediction.** Without historical tracking, you won't know your pool is filling up until it's too late.\n\n## The Solution\n\nASG provides five integrated governance modules:\n\n1. **Real Capacity Engine** — Calculates true RAID1 free space using the formula `min(sum/2, sum - largest)`, tracks historical usage, and predicts days-to-full.\n2. **Heuristic Scrub Controller** — Runs scrubs in the background, monitoring `/proc/loadavg` and `/proc/diskstats`. Automatically pauses when your system is busy and resumes when idle.\n3. **Integrity Monitor** — Polls `btrfs device stats` for all drives, detects errors, and sends alerts with deduplication.\n4. **Metadata Concentration Watchdog** — Warns if metadata/system chunks are dangerously concentrated on a subset of drives.\n5. **Snapshot Bridge** — Creates atomic read-only snapshots before your backup software runs, with automatic cleanup.\n\n---\n\n## Features\n\n- **Zero dependencies** — Python standard library only (PyYAML optional for config)\n- **Accurate RAID1 maths** for 2, 3, 4, or more mismatched drives\n- **Load-aware scrub throttling** using `/proc/loadavg` and `/proc/diskstats`\n- **Metadata concentration alerts** — catches the silent killer\n- **Days-to-full prediction** from historical usage data\n- **Pre-backup atomic snapshots** with retention management\n- **Pool UUID validation** before every operation (prevents wrong-pool disasters)\n- **`--dry-run`** on all filesystem-modifying commands\n- **Notifications** — Discord, NTFY, and Gotify (all optional)\n- **Config-driven** — single YAML file, sensible defaults\n\n---\n\n## Requirements\n\n| Requirement | Minimum | Notes |\n|---|---|---|\n| Linux kernel | 4.0+ | Any kernel with BTRFS support |\n| btrfs-progs | 5.0+ | `sudo apt install btrfs-progs` |\n| Python | 3.9+ | |\n| BTRFS profile | RAID1 | RAID1C3/RAID1C4 should also work |\n| sudo | Required | BTRFS commands require root |\n| PyYAML | Optional | For config file; falls back to defaults without it |\n\n---\n\n## Quick Start\n\n### Option 1: Install via pip\n\n```bash\npip3 install aldertech-asg\n```\n\n### Option 2: Install via Git Clone\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/jaldertech/asg.git\ncd asg\n\n# 2. Run the installer (requires root)\nsudo bash setup.sh\n\n# 3. Edit your config\nsudo nano /etc/asg/config.yaml\n\n# 4. Test it\nasg status\n```\n\n### Option 3: Run directly (no install)\n\n```bash\ngit clone https://github.com/jaldertech/asg.git\ncd asg\npython3 -m asg status\n```\n\n---\n\n## Configuration\n\nCopy `config.yaml` to `/etc/asg/config.yaml` and edit it:\n\n```yaml\npool:\n  mount: \"/mnt/my_pool\"\n  # uuid: \"optional-pin-for-safety\"\n\nscrub:\n  load_threshold: 3.0           # Pause scrub above this 1-min load average\n  io_threshold_percent: 40.0    # Pause if any drive exceeds this I/O %\n\ncapacity:\n  chunk_fullness_warn_percent: 90.0\n  free_space_warn_percent: 10.0\n  metadata_min_device_count: 3  # Warn if metadata on fewer devices\n\nsnapshots:\n  directory: \".snapshots\"       # Relative to pool mount\n  retention_days: 3\n\n# Optional — remove if you don't want notifications\nnotifications:\n  discord:\n    webhook_url: \"https://discord.com/api/webhooks/...\"\n```\n\nIf PyYAML is not installed, the tool runs with built-in defaults (mount point: `/mnt/media_pool`).\n\n---\n\n## Usage\n\n```bash\n# Full status overview (integrity + capacity + snapshots)\nasg status\n\n# Real capacity report with days-to-full prediction\nasg capacity\n\n# Run a load-aware scrub (pauses during high I/O)\nasg scrub --dry-run     # Preview first\nasg scrub               # Run for real\n\n# Check all drives for errors\nasg check\n\n# Pre-backup snapshot (cleanup old + create new)\nasg snapshot --dry-run\nasg snapshot\n\n# Clean up expired snapshots only\nasg cleanup\n\n# Use a custom config path\nasg --config /path/to/config.yaml status\n```\n\n### Example Output: Capacity Report\n\n```\nASG Capacity Report — /mnt/media_pool\nTimestamp: 2026-03-07 19:45:57\n=================================================================\n\n  Total Raw Capacity:   7454.1 GiB\n  Largest Drive:        3727.4 GiB\n  RAID1 Usable Ceiling: 3726.7 GiB\n  Data Used:            100.5 GiB\n  Real Free Space:      3625.8 GiB\n  Utilisation:          2.7%\n  Days to Full:         ~342\n\n  Per-Device Allocation:\n  Device           Data       Meta     System      Unalloc      Total\n  ---------------------------------------------------------------\n  /dev/sdc         103.0G     1.000G     0.031G      1761.3G    1863.7G\n  /dev/sdd          30.0G     1.000G     0.008G       900.5G     931.5G\n  /dev/sde          25.0G     1.000G     0.008G       905.5G     931.5G\n  /dev/sdf          50.0G     1.000G     0.031G      3676.2G    3727.4G\n\n  WARNINGS:\n    ! Data chunks are 96.6% full — new chunk allocation imminent.\n=================================================================\n```\n\n---\n\n## Cron Integration\n\nAdd to your crontab (`crontab -e`):\n\n```bash\n# Daily capacity report at 00:20\n20 0 * * * asg capacity \u003e\u003e ~/asg.log 2\u003e\u00261\n\n# Weekly throttled scrub on Sunday at 02:00\n0 2 * * 0 asg scrub \u003e\u003e ~/asg.log 2\u003e\u00261\n\n# Daily pre-backup snapshot at 02:00\n0 2 * * * asg snapshot \u003e\u003e ~/asg.log 2\u003e\u00261\n```\n\n---\n\n## How the RAID1 Maths Works\n\nStandard `df` divides total raw space by the data ratio (2.0 for RAID1), giving `sum / 2`. But this ignores the **pairing constraint**: each RAID1 chunk must fit on two different drives.\n\nFor mismatched drives, the real limit is:\n\n```\nusable = min(sum_of_all_drives / 2, sum_of_all_drives - largest_drive)\n```\n\n**Example:** 4TB + 2TB + 1TB + 1TB = 8TB raw\n\n| Calculation | Result | Why |\n|---|---|---|\n| `sum / 2` | 4 TB | Standard RAID1 formula |\n| `sum - largest` | 4 TB | The 4TB drive can only mirror 4TB of data across the other 3 drives |\n| **Usable ceiling** | **4 TB** | `min(4, 4)` — balanced in this case |\n\nBut with 8TB + 1TB + 1TB = 10TB raw:\n\n| Calculation | Result | Why |\n|---|---|---|\n| `sum / 2` | 5 TB | Looks generous |\n| `sum - largest` | 2 TB | The 8TB drive can only mirror 2TB across the others |\n| **Usable ceiling** | **2 TB** | 6TB of the 8TB drive is **wasted** |\n\nThis tool calculates the correct number automatically.\n\n---\n\n## Verified Hardware\n\n| Hardware | OS | Pool Layout | Status |\n|---|---|---|---|\n| Raspberry Pi 5 (16GB) | Raspberry Pi OS Bookworm | 4TB + 2TB + 1TB + 1TB RAID1 | Verified |\n\nASG uses generic Linux interfaces (`/proc`, `btrfs-progs`). It should work on any Linux system with BTRFS. If you test on other hardware, please open a PR to add it to this table.\n\n---\n\n## Notifications\n\nAll backends are optional and independent. Configure any combination in `config.yaml`:\n\n| Backend | Config Key | Notes |\n|---|---|---|\n| Discord | `notifications.discord.webhook_url` | Standard Discord webhook |\n| NTFY | `notifications.ntfy.url` | Full topic URL |\n| Gotify | `notifications.gotify.url` + `.token` | Base URL + app token |\n\n---\n\n## Security\n\nASG runs `btrfs` commands via `sudo`. It requires that the executing user has passwordless `sudo` for `btrfs` (standard on Raspberry Pi OS) or is run as root.\n\nThe tool:\n- Never writes to arbitrary paths (only state files in a configured directory)\n- Validates the pool UUID before every operation\n- Supports `--dry-run` for all destructive commands\n- Has zero network dependencies (notifications are opt-in)\n\n---\n\n## Related Projects\n\n- **[ADRG](https://github.com/jaldertech/adrg)** — Aldertech Dynamic Resource Governor: Automatic CPU/RAM management for high-density Docker stacks.\n\n## Licence\n\nMIT — see [LICENCE](LICENCE) file.\n\n---\n\n*Built by [Aldertech](https://aldertech.uk)*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaldertech%2Fasg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaldertech%2Fasg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaldertech%2Fasg/lists"}