{"id":13551847,"url":"https://github.com/trapexit/mergerfs-tools","last_synced_at":"2025-04-05T03:12:35.739Z","repository":{"id":37821062,"uuid":"49983523","full_name":"trapexit/mergerfs-tools","owner":"trapexit","description":"Optional tools to help manage data in a mergerfs pool","archived":false,"fork":false,"pushed_at":"2024-04-24T02:08:45.000Z","size":111,"stargazers_count":408,"open_issues_count":34,"forks_count":44,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-29T02:09:03.753Z","etag":null,"topics":["mergerfs"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trapexit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-19T21:13:15.000Z","updated_at":"2025-03-14T14:16:39.000Z","dependencies_parsed_at":"2023-01-20T23:04:07.823Z","dependency_job_id":"7ecd1db4-f658-49db-ba33-4c00d4a5434f","html_url":"https://github.com/trapexit/mergerfs-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fmergerfs-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fmergerfs-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fmergerfs-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trapexit%2Fmergerfs-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trapexit","download_url":"https://codeload.github.com/trapexit/mergerfs-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mergerfs"],"created_at":"2024-08-01T12:01:54.466Z","updated_at":"2025-04-05T03:12:35.720Z","avatar_url":"https://github.com/trapexit.png","language":"Python","funding_links":["https://paypal.me/trapexit","https://github.com/sponsors/trapexit","https://www.patreon.com/trapexit","https://ko-fi.com/trapexit","https://opencollective.com/trapexit"],"categories":["Python","others","Tasks"],"sub_categories":[],"readme":"# mergerfs-tools\n\nOptional tools to help manage data in a mergerfs pool.\n\n## INSTALL\n\nAll of these suplimental tools are self contained Python3 apps. Make sure you have Python 3 installed and either run `make install` or copy the file to `/usr/local/bin` or wherever you keep your binarys and make it executable (chmod +x).\n\n## TOOLS\n### mergerfs.ctl\n\nA wrapper around the mergerfs xattr interface.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.ctl)\n\n```\n$ mergerfs.ctl -h\nusage: mergerfs.ctl [-h] [-m MOUNT] {add,remove,list,get,set,info} ...\n\npositional arguments:\n  {add,remove,list,get,set,info}\n\noptional arguments:\n  -h, --help            show this help message and exit\n    -m MOUNT, --mount MOUNT\n                            mergerfs mount to act on\n$ mergerfs.ctl info\n- mount: /storage\n  version: 2.14.0\n  pid: 1234\n  srcmounts:\n    - /mnt/drive0\n    - /mnt/drive1\n$ mergerfs.ctl -m /storage add path /mnt/drive2\n$ mergerfs.ctl info\n- mount: /storage\n  version: 2.14.0\n  pid: 1234\n  srcmounts:\n    - /mnt/drive0\n    - /mnt/drive1\n    - /mnt/drive2\n```\n\n### mergerfs.fsck\n\nAudits permissions and ownership of files and directories in a mergerfs mount and allows for manual and automatic fixing of them.\n\nIt's possible that files or directories can be duplicated across multiple drives and that their metadata become out of sync. Permissions, ownership, etc. This can cause some strange behavior depending on the mergerfs policies used. This tool helps find and fix those inconsistancies.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.fsck)\n\n```\n$ mergerfs.fsck -h\nusage: mergerfs.fsck [-h] [-v] [-s] [-f {manual,newest,nonroot}] dir\n\naudit a mergerfs mount for inconsistencies\n\npositional arguments:\n  dir                   starting directory\n\n  optional arguments:\n    -h, --help            show this help message and exit\n    -v, --verbose         print details of audit item\n    -s, --size            only consider if the size is the same\n    -f {manual,newest,nonroot}, --fix {manual,newest,nonroot}\n                          fix policy\n$ mergerfs.fsck -v -f manual /path/to/dir\n```\n\n### mergerfs.dup\n\nDuplicates files \u0026 directories across branches in a pool. The file selected for duplication is picked by the `dup` option. Files will be copied to drives with the most free space. Deleted from others if `prune` is enabled.\n\nSee usage for more. Run as `root`. Requires `rsync` to be installed.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.dup)\n\n```\nusage: mergerfs.dup [\u003coptions\u003e] \u003cdir\u003e\n\nDuplicate files \u0026 directories across multiple drives in a pool.\nWill print out commands for inspection and out of band use.\n\npositional arguments:\n  dir                    starting directory\n\noptional arguments:\n  -c, --count=           Number of copies to create. (default: 2)\n  -d, --dup=             Which file (if more than one exists) to choose to\n                         duplicate. Each one falls back to `mergerfs` if\n                         all files have the same value. (default: newest)\n                         * newest   : file with largest mtime\n                         * oldest   : file with smallest mtime\n                         * smallest : file with smallest size\n                         * largest  : file with largest size\n                         * mergerfs : file chosen by mergerfs' getattr\n  -p, --prune            Remove files above `count`. Without this enabled\n                         it will update all existing files.\n  -e, --execute          Execute `rsync` and `rm` commands. Not just\n                         print them.\n  -I, --include=         fnmatch compatible filter to include files.\n                         Can be used multiple times.\n  -E, --exclude=         fnmatch compatible filter to exclude files.\n                         Can be used multiple times.\n```\n\n\n### mergerfs.dedup\n\nFinds and removes duplicate files across mergerfs pool's branches. Use the\n`ignore`, `dedup`, and `strict` options to target specific use cases.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.dedup)\n\n```\nusage: mergerfs.dedup [\u003coptions\u003e] \u003cdir\u003e\n\nRemove duplicate files across branches of a mergerfs pool. Provides\nmultiple algos for determining which file to keep and what to skip.\n\npositional arguments:\n  dir                    Starting directory\n\noptional arguments:\n  -v, --verbose          Once to print `rm` commands\n                         Twice for status info\n                         Three for file info\n  -i, --ignore=          Ignore files if... (default: none)\n                         * same-size      : have the same size\n                         * different-size : have different sizes\n                         * same-time      : have the same mtime\n                         * different-time : have different mtimes\n                         * same-hash      : have the same md5sum\n                         * different-hash : have different md5sums\n  -d, --dedup=           What file to *keep* (default: newest)\n                         * manual        : ask user\n                         * oldest        : file with smallest mtime\n                         * newest        : file with largest mtime\n                         * largest       : file with largest size\n                         * smallest      : file with smallest size\n                         * mostfreespace : file on drive with most free space\n  -s, --strict           Skip dedup if all files have same value.\n                         Only applies to oldest, newest, largest, smallest.\n  -e, --execute          Will not perform file removal without this.\n  -I, --include=         fnmatch compatible filter to include files.\n                         Can be used multiple times.\n  -E, --exclude=         fnmatch compatible filter to exclude files.\n                         Can be used multiple times.\n\n# mergerfs.dedup /path/to/dir\n# Total savings: 10.0GB\n\n# mergerfs.dedup -e -d newest /path/to/dir\nmergerfs.dedup -v -d newest /media/tmp/test\nrm -vf /mnt/drive0/test/foo\nrm -vf /mnt/drive1/test/foo\nrm -vf /mnt/drive2/test/foo\nrm -vf /mnt/drive3/test/foo\n# Total savings: 10.0B\n```\n\n\n### mergerfs.balance\n\nWill move files from the most filled drive (percentage wise) to the least filled drive. Will do so till the most and least filled drives come within a user defined percentage range (defaults to 2%).\n\nRun as `root`. Requires `rsync` to be installed.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.balance)\n\n```\nusage: mergerfs.balance [-h] [-p PERCENTAGE] [-i INCLUDE] [-e EXCLUDE]\n                        [-I INCLUDEPATH] [-E EXCLUDEPATH] [-s EXCLUDELT]\n                        [-S EXCLUDEGT]\n                        dir\n\nbalance files on a mergerfs mount based on percentage drive filled\n\npositional arguments:\n  dir                   starting directory\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -p PERCENTAGE         percentage range of freespace (default 2.0)\n  -i INCLUDE, --include INCLUDE\n                        fnmatch compatible file filter (can use multiple\n                        times)\n  -e EXCLUDE, --exclude EXCLUDE\n                        fnmatch compatible file filter (can use multiple\n                        times)\n  -I INCLUDEPATH, --include-path INCLUDEPATH\n                        fnmatch compatible path filter (can use multiple\n                        times)\n  -E EXCLUDEPATH, --exclude-path EXCLUDEPATH\n                        fnmatch compatible path filter (can use multiple\n                        times)\n  -s EXCLUDELT          exclude files smaller than \u003cint\u003e[KMGT] bytes\n  -S EXCLUDEGT          exclude files larger than \u003cint\u003e[KMGT] bytes\n\n# mergerfs.balance /media\nfrom: /mnt/drive1/foo/bar\nto:   /mnt/drive2/foo/bar\nrsync ...\n```\n\n\n### mergerfs.consolidate\n\nConsolidate **files** in a **single** mergerfs directory onto a **single** drive, recursively. This does **NOT** move all files at and below that directory to 1 drive. If you want to move data between drives simply use normal rsync or similar. This tool is only useful in niche usecases where the person wants to colocate files of their TV, music, etc. files onto a single drive *after the fact.* If you really wanted that you should probably use path preservation. For most people there is only downsides to using path preservation or colocating files.\n\nRun as `root`. Requires `rsync` to be installed.\n\n[Download latest](https://raw.githubusercontent.com/trapexit/mergerfs-tools/master/src/mergerfs.consolidate)\n\n```\nusage: mergerfs.consolidate [\u003coptions\u003e] \u003cdir\u003e\n\npositional arguments:\n  dir                    starting directory\n\noptional arguments:\n  -m, --max-files=       Skip directories with more than N files.\n                         (default: 256)\n  -M, --max-size=        Skip directories with files adding up to more\n                         than N. (default: 16G)\n  -I, --include-path=    fnmatch compatible path include filter.\n                         Can be used multiple times.\n  -E, --exclude-path=    fnmatch compatible path exclude filter.\n                         Can be used multiple times.\n  -e, --execute          Execute `rsync` commands as well as print them.\n  -h, --help             Print this help.\n```\n\n## SUPPORT\n\n#### Contact / Issue submission\n* github.com: https://github.com/trapexit/mergerfs-tools/issues\n* email: trapexit@spawn.link\n* twitter: https://twitter.com/_trapexit\n\n#### Support development\n\nThis software is free to use and released under a very liberal license. That said if you like this software and would like to support its development donations are welcome.\n\n* PayPal: https://paypal.me/trapexit\n* GitHub Sponsors: https://github.com/sponsors/trapexit\n* Patreon: https://www.patreon.com/trapexit\n* SubscribeStar: https://www.subscribestar.com/trapexit\n* Ko-Fi: https://ko-fi.com/trapexit\n* Open Collective: https://opencollective.com/trapexit\n* Bitcoin (BTC): bc1qjwlywkqxgrxql3m7a7fvcsf3z3t98jvtekqp2j\n* Bitcoin Cash (BCH): qrvymmkvuk7703m7cx0pqxc3mz4mmsn6ngn9xw52kc\n* Bitcoin SV (BSV): 1FkFuxRtt3f8LbkpeUKRZq7gKJFzGSGgZV\n* Bitcoin Gold (BTG): Gfk8QbMJFgpMTcY7uB63axy6HU7uTPPWNj\n* Basic Attention Token (BAT): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F\n* Chainlink (LINK): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F\n* Dash (DASH): Xu2U3Nd3G4hM5TRQUBcP4DHJFzXH93xB84\n* Dogecoin (DOGE): DGFBPsRBYL8wHbgnvKbYkVn5FvAe854p1c\n* Ethereum (ETH): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F\n* Filecoin (FIL): f1wpypkjcluufzo74yha7p67nbxepzizlroockgcy\n* LBRY Credits (LBC): bFusyoZPkSuzM2Pr8mcthgvkymaosJZt5r\n* Litecoin (LTC): LfL7jLNYuVpy7v5TyRyc3yRZ2uhqc4UoR3\n* Monero (XMR): 45BBZMrJwPSaFwSoqLVNEggWR2BJJsXxz7bNz8FXnnFo3GyhVJFSCrCFSS7zYwDa9r1TmFmGMxQ2HTntuc11yZ9q1LeCE8f\n* Tezos (XTZ): tz1ZxerkbbALsuU9XGV9K9fFpuLWnKAGfc1C\n* Zcash (ZEC): t1bjbVBK7tx9EGBrnD2wDfjGV9yZrcyfMmr\n* Other crypto currencies: contact me for address\n\n## LINKS\n\n* https://spawn.link\n* https://github.com/trapexit/mergerfs\n* https://github.com/trapexit/mergerfs/wiki\n* https://github.com/trapexit/mergerfs-tools\n* https://github.com/trapexit/scorch\n* https://github.com/trapexit/bbf\n* https://github.com/trapexit/backup-and-recovery-howtos\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrapexit%2Fmergerfs-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrapexit%2Fmergerfs-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrapexit%2Fmergerfs-tools/lists"}