{"id":51558501,"url":"https://github.com/git001/nfs-sync-via-rclone","last_synced_at":"2026-07-10T08:01:45.495Z","repository":{"id":357381429,"uuid":"1236607580","full_name":"git001/nfs-sync-via-rclone","owner":"git001","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-12T14:08:21.000Z","size":39,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-12T15:35:20.491Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/git001.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-12T12:04:45.000Z","updated_at":"2026-05-12T14:43:05.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/git001/nfs-sync-via-rclone","commit_stats":null,"previous_names":["git001/nfs-sync-via-rclone"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/git001/nfs-sync-via-rclone","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fnfs-sync-via-rclone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fnfs-sync-via-rclone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fnfs-sync-via-rclone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fnfs-sync-via-rclone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/git001","download_url":"https://codeload.github.com/git001/nfs-sync-via-rclone/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git001%2Fnfs-sync-via-rclone/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35325188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-10T08:01:44.830Z","updated_at":"2026-07-10T08:01:45.484Z","avatar_url":"https://github.com/git001.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nfs-sync\n\nHardened, idempotent one-shot sync from an NFS mount to any rclone-supported\nobject store (Azure Blob, S3, GCS, B2, …). Runs as a systemd timer with\ngap-based scheduling, flock-based overlap protection, quiescence filtering,\nand a mandatory pre-production benchmark gate.\n\nDesigned for 1M+ file trees where NFS walk latency is the critical variable.\n\n## How it works\n\n```\nNFS mount  ──►  nfs-sync.sh  ──►  rclone sync  ──►  Object Store\n               (preflight +         (--fast-list\n                flock +              --checksum\n                quiescence)          --min-age)\n```\n\n- **Polling, not events.** `inotify` does not fire on NFS clients for writes\n  made by other hosts. The reconcile loop against the filesystem as\n  source-of-truth is intentional.\n- **Gap-based timer.** `OnUnitInactiveSec` guarantees a fixed gap between runs\n  regardless of how long a run takes — no two syncs can overlap.\n- **Quiescence filter.** `--min-age` skips files touched within the last N\n  seconds, avoiding uploads of files still being written.\n- **Hard loss SLA.** Every file version is eventually synced, even after crash\n  or network drop, because each run is a full reconcile.\n\n## Requirements\n\n| Dependency | Min version | Notes |\n|---|---|---|\n| rclone | 1.58 | `upload_concurrency` (Azure) added in v1.58 |\n| bash | 4.4 | `mapfile`, `EPOCHSECONDS` |\n| util-linux | any | `flock`, `mountpoint` |\n| systemd | any | timer + hardened service unit |\n\n## Installation\n\n### From source (any distro)\n\n```bash\nsudo ./rclone/install.sh\n```\n\nIdempotent. Existing `/etc/rclone/rclone.conf` and `/etc/default/nfs-sync`\nare never overwritten — a `.new` file is written next to them for manual\ndiffing.\n\n### Packages\n\nPre-built packages are available for the following targets:\n\n| Format | Targets | Build |\n|---|---|---|\n| RPM | RHEL 8 / 9 / 10 | `make -C packaging/rpm` |\n| DEB | Ubuntu 22/24/26, Debian 12/13 | `make -C packaging/deb` |\n| Alpine APKBUILD | Alpine 3.21+ | `abuild -r` in `packaging/alpine/` |\n| Nix Flake | x86\\_64-linux, aarch64-linux | `nix build .#nfs-sync` in `packaging/nix/` |\n\n## Configuration\n\n### 1. rclone remote (`/etc/rclone/rclone.conf`)\n\n`rclone/rclone.conf` is an annotated template for Azure Blob. Replace or\nextend it for your backend. The file is root-owned, mode 0600.\n\n```ini\n# Azure Blob example — see rclone docs for S3, GCS, B2, etc.\n[my-remote]\ntype = azureblob\naccount = mystorageaccount\nuse_msi = true          # Managed Identity (recommended on Azure VMs)\nupload_concurrency = 16\nchunk_size = 64M\n```\n\n### 2. Environment (`/etc/default/nfs-sync`)\n\n`rclone/nfs-sync.defaults` is the annotated template. Key variables:\n\n| Variable | Default | Description |\n|---|---|---|\n| `SRC` | `/mnt/nfs/source` | NFS mount path (must be a real mountpoint) |\n| `DST` | `remote:bucket` | rclone destination — must match your `rclone.conf` |\n| `MOUNT_CHECK_FILE` | _(empty)_ | Sentinel file inside `SRC`; if missing, sync aborts |\n| `MIN_AGE` | `30s` | Quiescence window — skip files newer than this |\n| `TRANSFERS` | `32` | Parallel upload streams |\n| `CHECKERS` | `32` | Parallel checksum / stat workers |\n| `MODIFY_WINDOW` | `2s` | mtime tolerance — 2s suits NFS; set to `0` for exact backends |\n| `EXCLUDE_PATTERNS` | `*.tmp,*.swp,…` | Comma-separated rclone filter globs |\n| `BWLIMIT` | _(empty)_ | Bandwidth cap, e.g. `100M`; empty = unlimited |\n| `RCLONE_CONFIG` | `/etc/rclone/rclone.conf` | Config path |\n\n## Benchmark before going live\n\nRun `nfs-sync-bench` against the **real** NFS share and object store before\nenabling the timer. It produces three numbers:\n\n```\nWALK_TIME_S   — time to stat the entire source tree\nRECONCILE_S   — end-to-end sync of an unchanged tree (your latency floor)\nUPLOAD_MBPS   — effective upload bandwidth\n```\n\nDecision rule:\n\n| Result | Action |\n|---|---|\n| `RECONCILE_S \u003c SLA × 0.5` | rclone is fine — enable the timer |\n| `RECONCILE_S \u003c SLA` | rclone works, but monitor closely |\n| `RECONCILE_S \u003e= SLA` | escalate: shorter interval, tree sharding, or custom tool |\n\n```bash\nsudo -u nfs-sync nfs-sync-bench\n```\n\n## Enabling the timer\n\n```bash\nsystemctl enable --now nfs-sync.timer   # start on boot + immediately\nsystemctl status nfs-sync.timer         # next trigger time\njournalctl -u nfs-sync.service -f       # live log\ntail -f /var/log/nfs-sync/sync-*.log    # per-run log with timing breakdown\n```\n\nThe timer fires 2 minutes after boot, then runs with a 1-minute gap between\nruns. Adjust `OnUnitInactiveSec` in `nfs-sync.timer` for your SLA budget.\n\n## Local lab (no NFS server, no cloud account required)\n\nThe `lab/` directory contains a self-contained test environment:\n\n- **Source**: tmpfs mounted at `/tmp/nfs-lab/mount` (satisfies `mountpoint -q`)\n- **Target**: Azurite (Azure Blob emulator) via Podman\n\n```bash\nsudo ./lab/lab.sh setup        # start Azurite, mount tmpfs, generate 1000 test files\n./lab/lab.sh run-sync          # run nfs-sync.sh with lab config\n./lab/lab.sh verify            # assert source count == blob count, no .tmp files leaked\n./lab/lab.sh logs              # tail the most recent sync log\n./lab/lab.sh status            # show current state\nsudo ./lab/lab.sh teardown     # stop everything, unmount, clean up\n```\n\nRequirements: `podman`, `rclone \u003e= 1.65` (Azurite endpoint support), `sudo`.\n\n## Versioning\n\n`bump-version.sh` updates the version atomically across all six packaging\nfiles (RPM spec, RPM Makefile, Debian changelog, Debian Makefile, Nix flake,\nAlpine APKBUILD):\n\n```bash\n./bump-version.sh 0.2.0 \"Your changelog message\"\n./bump-version.sh --dry-run 0.2.0   # preview changes without writing\n```\n\n## Repository layout\n\n```\nrclone/\n  nfs-sync.sh          # sync script (installed as /usr/bin/nfs-sync)\n  nfs-sync.service     # systemd service unit\n  nfs-sync.timer       # systemd timer unit (1 min gap)\n  nfs-sync.defaults    # /etc/default/nfs-sync template\n  rclone.conf          # rclone.conf template (Azure Blob example)\n  sync-bench.sh        # benchmark helper (/usr/bin/nfs-sync-bench)\n  install.sh           # source-based installer (idempotent)\nlab/\n  lab.sh               # lab management script\n  lab.defaults         # env overrides for lab runs\n  lab.rclone.conf      # Azurite config (well-known dev credentials)\npackaging/\n  rpm/                 # RPM spec + Makefile (RHEL 8/9/10)\n  deb/                 # Debian source package (Ubuntu 22/24/26, Debian 12/13)\n  nix/                 # Nix flake + NixOS module\n  alpine/              # Alpine APKBUILD + OpenRC init + crond integration\nbump-version.sh        # atomic version bump across all packaging files\n```\n\n## License\n\nApache 2.0 — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit001%2Fnfs-sync-via-rclone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgit001%2Fnfs-sync-via-rclone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit001%2Fnfs-sync-via-rclone/lists"}