{"id":13587957,"url":"https://github.com/Spearfoot/scripts-for-using-rsync","last_synced_at":"2025-04-08T02:34:30.557Z","repository":{"id":134407633,"uuid":"286522049","full_name":"Spearfoot/scripts-for-using-rsync","owner":"Spearfoot","description":"Shell scripts for using rsync","archived":false,"fork":false,"pushed_at":"2020-10-10T13:29:56.000Z","size":161,"stargazers_count":18,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-02-13T22:07:49.095Z","etag":null,"topics":["freebsd","freenas","freenas-scripts","rsync","rsync-modules","ssh"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Spearfoot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-08-10T16:12:26.000Z","updated_at":"2024-02-13T22:07:49.674Z","dependencies_parsed_at":"2023-05-01T12:33:12.188Z","dependency_job_id":null,"html_url":"https://github.com/Spearfoot/scripts-for-using-rsync","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spearfoot%2Fscripts-for-using-rsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spearfoot%2Fscripts-for-using-rsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spearfoot%2Fscripts-for-using-rsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spearfoot%2Fscripts-for-using-rsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Spearfoot","download_url":"https://codeload.github.com/Spearfoot/scripts-for-using-rsync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223300637,"owners_count":17122656,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["freebsd","freenas","freenas-scripts","rsync","rsync-modules","ssh"],"created_at":"2024-08-01T15:06:26.560Z","updated_at":"2024-11-06T07:30:40.044Z","avatar_url":"https://github.com/Spearfoot.png","language":"Shell","readme":"# Scripts for using rsync\n\nThe __rsync__ command -- whether used standalone or installed as a module -- is a powerful tool for copying filesets between UNIX-style systems. It is very efficient in that it only copies new or changed data to the target system.\n\n### Purpose\n\nI wrote these wrapper scripts for the purpose of duplicating datasets from my primary to my secondary FreeNAS server. Because FreeNAS is based on FreeBSD, these scripts _might_ be somewhat FreeBSD-centric. But __rsync__ is a common tool on all UNIX-style systems, so it shouldn't take much effort to port the scripts to run successfully on Linux distributions, and in fact, I have used earlier versions of these scripts to transfer data to a Linux-based Synology Diskstation NAS system.\n\nMy goal was to copy new or changed files and also to delete files on the target that don't exist on the source. If your needs are different, particularly if you want to keep files on the target that have been deleted on the source, then you will need to remove the `--inplace` and  `--delete-during` options used in the scripts.\n\n### The Scripts\nThere are two scripts in this repository: one for use with modules (__rsync-module.sh__) and one to run __rsync__ directly (__rsync-invoke.sh__). Both scripts require 3 command-line arguments:\n- The source specification, including username and hostname for remote systems - example: `/mnt/tank/foo/`\n- The target specification, including username and hostname for remote systems - example: `root@boomer:/mnt/tank/foo`\n- A log filename\n\nTelling __rsync__ what to copy is a bit arcane: you have to be careful about placing the '/' character correctly. Basically, to copy a dataset from the source to the target you add a trailing '/' to the source specification and leave it off the target.\n\nThis is easier to explain with an example: to use __rsync-invoke.sh__ to copy local dataset __foo__ to remote server 'BOOMER', use this command line:\n\n`./rsync-invoke.sh /mnt/tank/foo/ root@boomer:/mnt/tank/foo /mnt/tank/bandit/log/rsync.log`\n\nModules work a little differently when specifying the path. You don't separate the user and server names from the target path with a colon, and instead of providing the full path of the target, you specify the module name. Again, this is easier to demonstrate with an example:\n\n`./rsync-module.sh /mnt/tank/foo/ root@boomer/tank/foo /mnt/tank/bandit/log/rsync.log`\n\nAfter either of the examples above complete, you can examine log file `/mnt/tank/bandit/log/rsync.log` for results, which will look something like this:\n```\n+---------------------------------------------------------------------------------\n+ Mon Aug 10 04:00:05 CDT 2020: Copy /mnt/tank/foo/ to root@boomer/tank/foo\n+---------------------------------------------------------------------------------\n2020/08/10 04:00:06 [10732] building file list\n2020/08/10 04:00:37 [10732] *deleting   ONYX/Macrium/onyx-system3-00-00.mrimg\n2020/08/10 04:00:37 [10732] *deleting   ONYX/Macrium/onyx-data3-00-00.mrimg\n2020/08/10 04:00:37 [10732] .d..t...... ONYX/Macrium/\n2020/08/10 04:00:37 [10732] \u003cf.st...... ONYX/Macrium/file-groom.log\n2020/08/10 04:08:02 [10732] \u003cf+++++++++ ONYX/Macrium/onyx-data1-00-00.mrimg\n2020/08/10 04:10:35 [10732] \u003cf+++++++++ ONYX/Macrium/onyx-system2-00-00.mrimg\n2020/08/10 04:10:36 [10732] sent 129.62G bytes  received 5.02K bytes  205.26M bytes/sec\n2020/08/10 04:10:36 [10732] total size is 2.93T  speedup is 22.58\n+ Mon Aug 10 04:10:36 CDT 2020 Transfer completed\n```\n\n### Push or Pull?\nMy use of these scripts is strictly _push_ oriented: the 'source' I specify is always a local dataset; the 'target' is always on a remote system. But note that you can use __rsync-invoke.sh__ to _pull_ data from a remote system as well.\n\nExample: this command will copy dataset 'foo' from remote server 'BOOMER' to the local system:\n\n`./rsync-invoke.sh root@boomer:/mnt/tank/foo/ /mnt/tank/foo /mnt/tank/bandit/log/rsync.log`\n\nThe module script (__rsync-module.sh__) is strictly _push_ oriented: it can only be used to copy data from the local system to a remote rsync module because the target specifier has the `rsync://` prefix hard-coded. But you could easily modify a copy of this script and put the `rsync://` prefix on the source specifier if you need _pull_ capability.\n\n### Prerequisites: SSH\nSince __rsync__ typically uses __ssh__, you will need to configure __ssh__ key-based authentication to allow logging on to your target servers without having to enter a password.\n\n### Prerequisites: rsync modules\nYou will need to configure __rsync__ modules if you plan to use them as targets. \n\nExample: on my FreeNAS server 'BOOMER' I have configured a single __rsync__ module named 'tank', with a path of '/mnt/tank', access mode of 'Read and Write', user 'root', and group 'wheel'.\n\n### Slow Network Performance\n\nJust about every __rsync__ user notices how slow it is at transferring data. This is usually due to using __ssh__ as the transport protocol, with its attendant encryption. A common approach to overcoming slow transfers is to use less CPU-intensive encryption algorithms or to do away with encryption altogether. I have found that using __rsync__ modules is much faster than standalone mode, and that disabling encryption speeds up standalone transfers.\n\nOn my 10Gb network, I get transfer rates of up to 2Gb/s using __rsync-module.sh__, quite a bit faster than the typical __rsync-invoke.sh__ rate of roughly 800Mb/s.\n\n### Windows ACL Data\n\nCopying Windows ACLs can be a problem on some systems, particularly FreeNAS/FreeBSD, and I have selected options to avoid problems with this issue. On FreeNAS this means avoiding these options:\n\n```\n-a, --archive Equals -rlptgoD (no -H, -A, -X)\n-p, --perms   Preserve permissions\n-A, --acls    Preserve ACLs (implies -p)\n```\nSee my post [\"Impaired rsync permissions support for Windows datasets\"](https://www.ixsystems.com/community/threads/impaired-rsync-permissions-support-for-windows-datasets.43973) on the iXsystem FreeNAS forum for further discussion on this issue.\n\nOn some Linux distributions, __rsync__ may support copying Windows ACLs directly, while on others it will not. In the latter case, users have reported success using __robocopy__ in conjunction with __rsync__ to transfer Windows ACL data.\n\nTo determine whether your environment supports copying Windows ACLs, explore the options above along with:\n- `-X --xattrs preserve extended attributes`\n\n### Options\n\nThese are the __rsync__ options used in both scripts:\n```\n-r  recurse into directories\n-l  copy symlinks as symlinks\n-t  preserve modification times\n-g  preserve group\n-o  preserve owner\n-D  preserve device and special files\n-h  human readable progress\n-v  increase verbosity\n--delete-during   receiver deletes during the transfer\n--inplace         write updated data directly to destination file\n--progress        show progress during transfer  \n--log-file        specify log file\n--exclude         exclude files\n```\nBoth scripts exclude the following files; modify or remove to suit your needs:\n```\nvmware.log      VMware virtual machine log files\nvmware-*.log\n@eaDir/         Synology extended attributes\n@eaDir\nThumbs.db       Windows folder-related system file\n```\nThe __rsync-invoke.sh__ script disables __SSH__ encryption and compression with these settings:\n```\n-e \"ssh -T -c none -o Compression=no -x\"\n```\nIf your system does not support `none` as an encryption scheme, try `arcfour` or another low-cost encryption algorithm.\n\n## Example Usage\n\nI run this `cron` script early every morning to synchronize datasets from my primary FreeNAS server 'BANDIT' to my secondary server 'BOOMER', using the __rsync__ module installed on 'BOOMER'. On both servers the datasets are stored on a pool named 'tank':\n\n```\n#!/bin/sh\n\n# Synchronize all tank datasets from BANDIT to BOOMER\n\nlogfile=/mnt/tank/bandit/log/bandit-to-boomer.log\n\ndatasets=\"archives backups devtools domains hardware media music ncs opsys photo systools web\"\n\nrm ${logfile}\nfor dataset in $datasets; do\n    # Use rsync-module.sh to target the rsync module on the remote server:\n    /mnt/tank/systems/scripts/rsync-module.sh /mnt/tank/$dataset/ root@boomer/tank/$dataset ${logfile}\ndone\n```\n\nThis script does exactly the same thing, only using __rsync-invoke.sh__ to call __rsync__ directly:\n```\n#!/bin/sh\n\n# Synchronize all tank datasets from BANDIT to BOOMER\n\nlogfile=/mnt/tank/bandit/log/bandit-to-boomer.log\n\ndatasets=\"archives backups devtools domains hardware media music ncs opsys photo systools web\"\n\nrm ${logfile}\nfor dataset in $datasets; do\n    # Use rsync-invoke.sh to run rsync directly:\n    /mnt/tank/systems/scripts/rsync-invoke.sh /mnt/tank/$dataset/ root@boomer:/mnt/tank/$dataset ${logfile}\ndone\n```\n\n","funding_links":[],"categories":["Shell"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSpearfoot%2Fscripts-for-using-rsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSpearfoot%2Fscripts-for-using-rsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSpearfoot%2Fscripts-for-using-rsync/lists"}