{"id":17658660,"url":"https://github.com/ruuda/btrfs-mirror-subvolumes","last_synced_at":"2026-02-14T21:31:19.356Z","repository":{"id":66191219,"uuid":"266425151","full_name":"ruuda/btrfs-mirror-subvolumes","owner":"ruuda","description":"Mirror btrfs subvolumes to another file system while preserving sharing","archived":false,"fork":false,"pushed_at":"2022-11-30T17:05:50.000Z","size":37,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-28T14:49:48.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruuda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-23T21:49:31.000Z","updated_at":"2024-12-14T23:11:12.000Z","dependencies_parsed_at":"2023-03-07T10:15:57.880Z","dependency_job_id":null,"html_url":"https://github.com/ruuda/btrfs-mirror-subvolumes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ruuda/btrfs-mirror-subvolumes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruuda%2Fbtrfs-mirror-subvolumes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruuda%2Fbtrfs-mirror-subvolumes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruuda%2Fbtrfs-mirror-subvolumes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruuda%2Fbtrfs-mirror-subvolumes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruuda","download_url":"https://codeload.github.com/ruuda/btrfs-mirror-subvolumes/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruuda%2Fbtrfs-mirror-subvolumes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29456218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T21:29:27.764Z","status":"ssl_error","status_checked_at":"2026-02-14T21:28:11.111Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-23T15:28:05.536Z","updated_at":"2026-02-14T21:31:19.342Z","avatar_url":"https://github.com/ruuda.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Btrfs-mirror-subvolumes\n\nMirror btrfs subvolumes to another file system while preserving sharing.\n\n * Btrfs snapshots protect against unintentional filesystem modifications.\n * RAID 1 protects against hardware failure.\n * Mirroring a filesystem protects against software bugs that render the\n   entire filesystem unmountable.\n\nThe tool in this repository facilitates the third point.\n\n * [Usage](#usage)\n * [Motivation](#motivation)\n * [Implementation](#implementation)\n * [Alternatives](#alternatives)\n * [License](#license)\n\n## Usage\n\nBuild the Rust part with [Cargo][install-rust]:\n\n    cargo build --release\n\nPerform an initial full sync of one snapshot, to use as a base for incremental\nmirroring:\n\n    export DATE=2020-01-01\n    cd /fs2/snapshots\n    btrfs subvolume create $DATE\n    rsync -a --preallocate --info=progress2 /fs1/snapshots/$DATE/ /fs2/snapshots/$DATE\n    btrfs property set -t subvol /fs2/snapshots/$DATE ro true\n\nDo a dry-run of the script:\n\n    ./btrfs-mirror-subvolumes.py --dry-run /fs1/snapshots /fs2/snapshots\n\nSync a single snapshot (the one closest to the base snapshot):\n\n    ./btrfs-mirror-subvolumes.py --single /fs1/snapshots /fs2/snapshots\n\nIf everything looks fine in the new snapshot, mirror all of them sequentially:\n\n    ./btrfs-mirror-subvolumes.py /fs1/snapshots /fs2/snapshots\n\n## Motivation\n\nSuppose we have a btrfs filesystem at `/fs1`, with a `/current` subvolume,\nand daily read-only snapshots of that subvolume under a `/snapshots` subvolume.\nSuppose furthermore, that we have a second filesystem `/fs2` with a `/snapshots`\nsubdirectory or subvolume.\n\n    /\n    ├── fs1\n    │   ├── current\n    │   └── snapshots\n    │       ├── 2020-01-01\n    │       ├── 2020-01-02\n    │       ├── 2020-01-03\n    │       └── etc.\n    └── fs2\n        └── snapshots\n            ├── 2020-01-01\n            ├── 2020-01-02\n            ├── 2020-01-03\n            └── etc.\n\n * We want `/fs2/snapshots` to have the same contents as `/fs1/snapshots`.\n\n * We want `/fs2` to have roughly the same size as `/fs1`. The snapshots under\n   `/fs1` have been created incrementally, so they share most of their extents.\n   Each snapshot only consumes space for the data that is uniquely in that\n   snapshot, and not in any other snapshot. We need to preserve this sharing in\n   `/fs2`, to target a similar file system size.\n\n * `/fs1` and `/fs2` should be as isolated as possible. Low-level replication\n   (the extreme case being at the block level, like RAID) has a higher risk of\n   replicating bugs. We need higher-level replication, at the file level.\n\n## Implementation\n\nThe second file system is again a btrfs file system, mirrored at the file level\nwith rsync. Some custom tooling sets up snapshots and reflinked file copies in\nsuch a way that a subsequent rsync run can take advantage of sharing.\n\nTo mirror a single subvolume:\n\n * Pick a base snapshot to start from, and `btrfs subvolume snapshot` it,\n   mutable at first.\n\n * Run a custom program to heuristically detect renames and similar files\n   between the snapshots, and reflink the originals into the new snapshot with\n   a `FICLONE` ioctl. Without this, rsync might detect the relation, but still\n   write the bytes into the target file, destroying sharing. (There is\n   [a patch][rsync-reflink] that adds reflink support to rsync, but it has seen\n   no activity since 2015.)\n\n * Run rsync with two important flags:\n\n   * `--inplace` to mutate the target file in place, instead of writing to a\n     temporary file and renaming that over the old file when the transfer is\n     complete. A new temporary file would not share any extents.\n\n   * `--no-whole-file` to enable rsync’s delta algorithm even when the two file\n     systems are both local. Writing deltas, rather than rewriting the entire\n     file, ensures that the unchanged extents can be shared.\n\n  * After transfer is complete, make the snapshot read-only.\n\nApart from that, the script drives mirroring all subvolumes until the two file\nsystems are in sync. It picks the nearest existing snapshot as a base, with a\npreference to reconstruct older snapshots from more recent snapshots, because\nthis fragments the older snapshots rather than the newer ones. (An initial\ntransfer can `fallocate` the file and then write it in one go. But a delta\nmutation of that file in a subsequently synced snapshot will necessarily create\nmore extents for the changed parts.) At least one mirrored snapshot must exist\nalready.\n\n\n## Alternatives\n\n * `btrfs send` and `btrfs receive` are ruled out for two reasons:\n\n   1. They are extent-based, so they form a lower level replication mechanism\n      than file-based replication, with the associated risk.\n\n   2. The man page says:\n\n      \u003e Additionally, receive does not currently do a very good job of\n      \u003e validating that an incremental send stream actually makes sense, and it\n      \u003e is thus possible for a specially crafted send stream to create a\n      \u003e subvolume with reflinks to arbitrary files in the same filesystem.\n      \u003e Because of this, users are advised to not use btrfs receive on send\n      \u003e streams from untrusted sources, and to protect trusted streams when\n      \u003e sending them across untrusted networks.\n\n      This warning in combination with my past experience with btrfs corruption,\n      makes me distrustful of `btrfs send` and `btrfs receive` for valuable\n      data.\n\n * Mirroring to a non-btrfs file system, or even a non-Linux operating system,\n   would obviously be the best way to guard against btrfs bugs, but maintaining\n   a second storage pool with a similar feature set, but implemented differently\n   (e.g. ZFS, or XFS on top of LVM and dm-raid) would be a considerable effort\n   with its own risks.\n\n## License\n\nBtrfs-mirror-subvolumes is free software licensed under the\n[Apache 2.0][apache2] license. Please do not open an issue if\nyou disagree with the choice of license.\n\n[rsync-reflink]: https://bugzilla.samba.org/show_bug.cgi?id=10170\n[install-rust]:  https://forge.rust-lang.org/infra/other-installation-methods.html\n[apache2]:       https://www.apache.org/licenses/LICENSE-2.0\n[except]:        https://www.gnu.org/licenses/gpl-faq.html#GPLIncompatibleLibs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruuda%2Fbtrfs-mirror-subvolumes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruuda%2Fbtrfs-mirror-subvolumes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruuda%2Fbtrfs-mirror-subvolumes/lists"}