{"id":22286916,"url":"https://github.com/koraa/zfs-replicate","last_synced_at":"2025-03-25T20:45:29.582Z","repository":{"id":145352185,"uuid":"41606051","full_name":"koraa/zfs-replicate","owner":"koraa","description":"Clone of https://git.gitorious.org/zfs-replicate/zfs-replicate.git since gitorious is no longer acive","archived":false,"fork":false,"pushed_at":"2015-08-29T20:17:59.000Z","size":192,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-30T18:22:22.558Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koraa.png","metadata":{"files":{"readme":"README","changelog":"ChangeLog","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-29T20:15:15.000Z","updated_at":"2015-09-28T20:00:33.000Z","dependencies_parsed_at":"2023-04-06T11:50:47.291Z","dependency_job_id":null,"html_url":"https://github.com/koraa/zfs-replicate","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fzfs-replicate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fzfs-replicate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fzfs-replicate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koraa%2Fzfs-replicate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koraa","download_url":"https://codeload.github.com/koraa/zfs-replicate/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245542764,"owners_count":20632561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-03T16:58:35.596Z","updated_at":"2025-03-25T20:45:29.540Z","avatar_url":"https://github.com/koraa.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"zfs-replicate is a simple script to *robustly* replicate a ZFS data\nset.\n\nBASICS\n======\n\nInvocation\n----------\n\nIf we have two computers, a live system ('live') and a backup system\n('backup'), and we want 'live' to replicate 'tank/home' to 'backup',\nthen we can invoke zfs-replicate on 'live' as follows:\n\n  live# zfs-replicate tank/home backup:tank/live/tank/home\n\nAlternatively, the backup server can pull updates:\n\n  backup# zfs-replicate live:tank/home tank/live/tank/home\n\nIt is even possible to have a third machine act as an intermediary:\n\n  intermediary# zfs-replicate live:tank/home backup:tank/live/tank/home\n\nBut, you probably don't want to do this, because all traffic is\ntunneled via the intermediary.\n\n\nzfs-replicate has a few options, but generally you won't need them.\nRun zfs-replicate --help to see them.\n\n\nRegular Updates\n---------------\n\nYou can run zfs-replicate via cron like this:\n\n  # backup the system every day at 1 am.\n  0 1 * * * zfs-replicate tank/home backup:tank/live/tank/home\n\nIt is also possible to run it in a while loop:\n\n  while true\n  do\n    zfs-replicate tank/home backup:tank/live/tank/home\n  done\n\n\nSSH\n---\n\nzfs-replicate uses ssh to transfer the data.  Thus, you'll probably\nwant to set up ssh public key authentication to avoid having to enter\na password.  This is quite easy.  To set up public authentication so\nthat 'live' can replicate to 'backup', run the following on 'live':\n\n  # ssh-copy-id backup\n\nThis will copy your public key stored on 'live' (typically stored in\n~/.ssh/id_rsa.pub) to the authorized keys file\n(~/.ssh/authorized_keys) on 'backup'.\n\n\nMIGRATING TO ZFS-REPLICATE\n==========================\n\nIf you've been using another tool to replicate ZFS file systems, you\ndon't need to start from scratch when migrating to zfs-replicate.  If\nzfs-replicate sees at least one common snapshot that is also\nreasonable (both snapshots have the name creation time), it will use\nthe one with the most recent creation time (with an informational\nwarning).  Consider:\n\n  # dd if=/dev/zero of=/tmp/test-image bs=4k count=16k\n  # zpool create test /tmp/test-image\n  # zfs create test/a\n  # touch /test/a/foo\n  # zfs snapshot test/a@backup\n  # zfs send test/a@backup | zfs receive test/b\n  # ./zfs-replicate test/a test/b\n  forster:test/a and forster:test/b have no common zfs-replicate\n  snapshots!  Either the destination repository is not a replica of the\n  source or this is the first time that you are using zfs-replicate.  You\n  can force zfs-replicate to use a specific snapshot that is not managed by\n  zfs-replicate using the --snapshot option.\n  \n  Common snapshots:\n    backup\tSun May  4 12:37 2014\t\tcreation time matches\n  \n  Trying the one reasonable match (backup)\n  send from @backup to test/a@zfs-replicate-FROM-forster:test_a-TO-forster:test_b-2014.05.04-12.38.11 estimated size is 0\n  total estimated size is 0\n  TIME        SENT   SNAPSHOT\n  # zpool destroy test\n  # rm -f /tmp/test-image\n\n\nDESIGN\n======\n\nzfs-replicate tries very hard to be robust.  Thus, if the destination\nrepository diverged the last replication, it won't automatically roll\nit back (and discard the changes).  Instead, zfs-replicate will\ndisplay an error message and prompt you to rerun with the --rollback\noption.  (You can instead perform the rollback by hand, if you\nprefer.)\n\nNote: After an initial replication, zfs-replicate marks the\ndestination as read-only.  You can, of course, disable this, but if\nyou modify the destination, which is as easy as an `ls -l' (since this\nupdates the atimes!), replication will no longer work.\n\nzfs-replicate starts by taking a snapshot of the source datapool.  The\nname of the snapshot is based on the following template:\n\n  zfs-replicate-FROM-source:datapool-TO-dest:datapool:YYYY.MM.DD-HH.MM.SS\n\nIf the name is already in use, zfs-replicate will sleep for a second\nand then try again (at which point, the seconds field will be\ndifferent).  Using this scheme, it is safe to replicate the same\ndataset to multiple destinations.\n\nzfs-replicate then checks whether the destination exists.  If it\ndoesn't exist, a full replication is performed.  If the destination\ndoes exist, then we look for common zfs-replicate snapshots (i.e.,\nthose snapshots whose name matches the above template).  If at least\none common zfs-replicate snapshot exists, then the most recent\n(according to the creation time) is used as the base for an\nincremental send.  If there is no common zfs-replicate snapshot, then\nzfs-replicate looks for common snapshots.  If at least one exists with\na matching creation time, then we use the one with the most recent\ncreation time (assuming that the user is migrating to zfs-replicate).\nIf there are none, then something is wrong (e.g., the destination\nisn't a replica of the source) and we stop.\n\nWhen replicating the data, we specify the `-F' option to `zfs\nreceive'.  This does two things: it rolls back the destination to the\nmost recent snapshot and synchronizes the snapshots on the destination\nwith those on the source.  We want the latter behavior, but we don't\nwant `zfs receive' to automatically rollback the destination.  To\nprevent this, we first create a snapshot on the destination.  Thus,\nthe roll back will be a no-op (unless the file system is modified\nbetween the snapshot and the receive, which is rather unlikely).\n\nIf zfs-replicate notices that the destination diverged (i.e., was\nmodified), then it prints out an error message.  Note: the user\ntypically just needs to rerun zfs-replicate with the --rollback option\nto discard the changes.\n\nIndependent of whether the replication is successful, we try to remove\nstale snapshots to prevent them from tying up space.  (Typically, we\nmaintain just a single snapshot.)  This is a bit complicated, because\nwe aren't always certain whether the replication succeeded or not\n(e.g., a connection error could occur after the data is completely\ntransferred, but before the ssh connection is cleanly torn down).  If\nwe are certain that the replication was successful, then we only need\nto keep the snapshot that we just created.  If we are certain that the\nreplication failed (e.g., we couldn't connect to the host or the\ndestination pool did not exist), then we can only remove the snapshot\nthat we just created.  If we aren't certain of the outcome, then we\nneed to keep the current snapshot and the common snapshot (because we\ndon't know which one is good).\n\nTEST SUITE\n==========\n\nThis program comes with a small test suite.  You can run it from the\nsource directory as follows:\n\n  # ./test\n\nNote: you need to be root.\n\nThe test suite will create a small temp file and create a ZFS pool in\nit.  All tests will be done on this pool and at the end the pool will\nbe destroyed.\n\nIf the test fails, edit 'test' and set TRACE to 1 (around line 25) and\nthen mail me (neal@walfield.org) the output.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoraa%2Fzfs-replicate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoraa%2Fzfs-replicate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoraa%2Fzfs-replicate/lists"}