{"id":13551494,"url":"https://github.com/tasket/wyng-backup","last_synced_at":"2025-04-03T01:33:03.059Z","repository":{"id":37520043,"uuid":"144275988","full_name":"tasket/wyng-backup","owner":"tasket","description":"Fast Time Machine-like backups for logical volumes \u0026 disk images","archived":false,"fork":false,"pushed_at":"2024-05-22T21:10:17.000Z","size":1392,"stargazers_count":240,"open_issues_count":40,"forks_count":16,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-05-22T21:33:16.121Z","etag":null,"topics":["backup","btrfs","img","incremental","isolation","kvm","linux","lvm","pruning","qcow2","qubes-os","reflinks","security","vmdk","xen","xfs"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tasket.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-10T11:06:41.000Z","updated_at":"2024-05-29T04:17:18.735Z","dependencies_parsed_at":"2024-03-16T22:58:02.614Z","dependency_job_id":"076fa98e-eb5f-4b60-9519-cbc2a6a06f5e","html_url":"https://github.com/tasket/wyng-backup","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasket%2Fwyng-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasket%2Fwyng-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasket%2Fwyng-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasket%2Fwyng-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tasket","download_url":"https://codeload.github.com/tasket/wyng-backup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222905989,"owners_count":17055820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","btrfs","img","incremental","isolation","kvm","linux","lvm","pruning","qcow2","qubes-os","reflinks","security","vmdk","xen","xfs"],"created_at":"2024-08-01T12:01:49.402Z","updated_at":"2025-04-03T01:33:03.007Z","avatar_url":"https://github.com/tasket.png","language":"Python","funding_links":["https://liberapay.com/tasket/donate","https://ko-fi.com/tasket","https://www.buymeacoffee.com/tasket"],"categories":["Python"],"sub_categories":[],"readme":"_\u003ch1 align=\"center\"\u003eWyng\u003c/h1\u003e_\n\u003cp align=\"center\"\u003e\nFaster incremental backups for logical volumes and disk images.\n\u003c/p\u003e\n\n### Introduction\n\nWyng is able to deliver faster incremental backups for logical\nvolumes and disk images. It accesses *copy-on-write* metadata (instead of comparing all data\nfor each backup) to instantly find changes since the last backup.\nCombined with its efficient archive format, Wyng can also very quickly reclaim space\nfrom older backup sessions.\n\nHaving nearly instantaneous access to volume changes and a nimble archival format\nenables backing up even terabyte-sized volumes multiple times per hour with little\nimpact on system resources.\n\nWyng pushes data to archives in a stream-like fashion, which avoids temporary data\ncaches and re-processing data. And Wyng's ingenious snapshot rotation avoids common\n_aging snapshot_ space consumption pitfalls.\n\nWyng also doesn't require the source admin system to ever mount processed volumes or\nto handle them as anything other than blocks, so it safely handles\nuntrusted data in guest filesystems to bolster container-based security.\n\n\n### Status\n\nPublic beta with a range of features including:\n\n - Incremental backups of Linux logical volumes from Btrfs, XFS and Thin-provisioned LVM\n\n - Supported destinations: Local filesystem, Virtual machine or SSH host\n\n - Fast pruning of old backup sessions\n\n - Basic archive management such as add/delete volume and auto-pruning\n\n - Automatic creation \u0026 management of local snapshots\n\n - Data deduplication\n\n - Marking and selecting archived snapshots with user-defined tags\n\nVersion 0.8 major enhancements:\n\n - Btrfs and XFS reflink support\n\n - Authenticated encryption with auth caching\n\n - Full data \u0026 metadata integrity checking\n\n - Fast differential receive based on local snapshots\n\n - Overall faster operation\n\n - Change autoprune behavior with --apdays\n\n - Configure defaults in /etc/wyng/wyng.ini\n\n - Mountpoints no longer required at destination\n\n - Simple selection of archives and local paths: Choose any _local_ or _dest_ each time you run Wyng\n\n - Multiple volumes can now be specified for most Wyng commands; send and receive support multiple storage pools\n\n Wyng is released under a GPL license and comes with no warranties expressed or implied.\n\n\nWyng v0.8 Requirements \u0026 Setup\n---\n\nBefore starting:\n\n* Python 3.8 or greater is required for basic operation.\n\n* For encryption and top performance, the _python3-pycryptodome_ and _python3-zstd_ packages\nshould be installed, respectively.\n\n* Volumes to be backed-up should reside locally in one of the following snapshot-capable\nstorage types:  LVM thin-provisioned pool, Btrfs subvolume, or XFS/reflink capable filesystem. Otherwise, volumes may be imported from or saved to other filesystems at standard (slower) speeds.\n\n* For backing up from LVM, _thin-provisioning-tools \u0026 lvm2_ must be present on the source system.  For Btrfs, the `btrfs` command must be present.\n\n* The destination system where the Wyng archive is stored (if different from source) should\nalso have python3, plus a basic Unix command set and filesystem (i.e. a typical Linux or BSD\nsystem). Otherwise, _samba_, FUSE, etc. may be used to access remote storage using smb, sftp, s3 or other protocols\nwithout concern for python or Unix commands.\n\n* See the 'Testing' section below for tips and caveats about using the alpha and beta versions.\n\n\n## Getting Started\n\nWyng is distributed as a single Python executable with no complex\nsupporting modules or other program files; it can be placed in '/usr/local/bin'\nor another place of your choosing.\n\n\nArchives can be created with `wyng arch-init`:\n\n```\nwyng arch-init --dest=ssh://me@exmaple.com:/home/me/mylaptop.backup\n\n...or...\n\nwyng arch-init --dest=file:/mnt/drive2/mylaptop.backup\n```\n\nThe examples above create a 'mylaptop.backup' directory on the destination.\nThe `--dest` argument includes the destination type, remote system (where applicable)\nand directory path.\n\nNext you can start making backups with `wyng send`:\n\n```\nwyng send --dest=file:/mnt/drive1/mylaptop.backup --local=volgrp1/pool1 root-volume home-volume\n```\n\nThis command sends two volumes 'root-volume' and 'home-volume' from the LVM thin pool 'volgrp1/pool1' to the destination archive.\n\n\u003cbr/\u003e\n## Operation\n\nRun Wyng using the following commands and arguments in the form of:\n\n**wyng \\[--options] command \\[volume_names] \\[--options]**\n\n\n### Command summary\n\n| _Command_ | _Description_  |\n|---------|---|\n| **list** _[volume_name]_    | List volumes or volume sessions.\n| **send** _[volume_name]_    | Perform a backup of enabled volumes.\n| **receive** _volume_name [*]_   | Restore volume(s) from the archive.\n| **verify** _volume_name [*]_    | Verify volumes' data integrity.\n| **prune** _[volume_name] [*]_   | Remove older backup sessions to recover archive space.\n| **delete** _volume_name_    | Remove entire volume from config and archive.\n| **rename** _vol_name_ _new_name_  | Renames a volume in the archive.\n| **arch-init**               | Create a new Wyng archive.\n| **arch-deduplicate**        | Deduplicate existing data in an archive.\n| **version**                 | Print the Wyng version and exit.\n\n\n### Advanced commands\n\n| _Command_ | _Description_  |\n|---------|---|\n| **monitor**                     | Collect volume change metadata \u0026 rotate snapshots.\n| **diff** _volume_name [*]_      | Compare local volume with archived volume.\n| **add** _volume_name [*]_       | Adds a volume name without session data to the archive.\n| **arch-check** _[volume_name] [*]_    | Thorough check of archive data \u0026 metadata\n\n\u003cbr/\u003e\n### Command details\n\n#### send\n\nPerforms a backup by storing volume data to a new session in the archive.  If the volume\nalready exists in the archive, incremental mode is automatically used.\n\n```\n\nwyng send my_big_volume --local=vg/pool --dest=file:/mnt/drive1/mylaptop.backup\n\n\n```\n\n`send` supports automatic pruning of older backup sessions to recover disk space before the new data is sent; set `--autoprune` option to _on_ or _full_ to use this feature.\n\nVolume names for non-LVM storage may include subdirectories, making them relative paths in\nthe same manner as file paths in `tar`.\nFor example, `wyng --local=/mnt/pool1 send appvms/personal.img` will send the volume located\nat '/mnt/pool1/appvms/personal.img'.\n\n#### receive\n\nRetrieves volumes (using the latest session ID\nif `--session` isn't specified) from the archive and saves it to either the `--local`\nstorage or the path specified with `--save-to` (the latter allows receiving only\none volume at a time).\nIf `--session` is used, only one date-time is accepted. The volume name is required.\n\n```\n\nwyng receive vm-work-private --local=vg/pool --dest=file:/mnt/drive1/mylaptop.backup\n\n\n```\n\n...restores a volume called 'vm-work-private' to 'myfile.img' in\nthe LVM thin pool 'vg/pool'.  Note that `--dest` always refers to the archive location, so\nthe volume is being restored _from_ '/mnt/drive1/mylaptop.backup'.\n\nIts possible to receive to any valid file path or block device using the `--save-to` option,\nwhich can be used in place of `--local`.\nFor any save path, Wyng will try to discard old data before receiving unless `--sparse`,\n`--sparse-write` or `--use-snapshot` options are used.\n\n\n#### verify\n\nThe `verify` command is similar to `receive` without saving the data. For both\n`receive` and `verify` modes, an error will be reported with a non-zero exit\ncode if the received data does not pass integrity checks.\n\n\n#### prune\n\nReclaim space on a backup drive by removing prior backup session(s) you specify.\n\nTo use, supply a single exact date-time in _YYYYMMDD-HHMMSS_ format to remove a\nspecific session, or two date-times representing a range:\n\n```\nwyng prune --all --session=20180605-000000,20180701-140000 --dest=file:/mnt/drive1/mylaptop.backup\n```\n\n...removes backup sessions from midnight on June 5 through 2pm on July 1 for all\nvolumes. Alternately, `--all-before` may be used with a single `--session` date-time\nto prune all sessions prior to that time.\n\nThe `--keep` option can accept a single date-time or a tag in the form `^tagID`.\nMatching sessions will be excluded from pruning and autopruning.\n\nAlso see `--autoprune` option below which works with `prune` and `send` commands.\n\n\n#### delete\n\nRemoves a volume's Wyng-managed snapshots, config and metadata from the source system and\nall of its *data* from the destination archive (everything deleted except the source\nvolume). Use with caution!\n\nAn alternate form of `delete` will remove all Wyng archive-related metadata (incl. snapshots) from the\nlocal system without affecting the archive on the destination:\n\n```\n\nwyng delete --clean\n\n```\n\nAlternately, using `delete --clean --all` will remove all known Wyng metadata from the local system,\nincluding any snapshots from the `--local` path.\n\n#### rename\n```\n\nwyng rename oldname newname\n\n```\n\nRenames a volume _'oldname'_ in the archive to _'newname'_. Note: This will rename only the\narchive volume, _not_ your source volume.\n\n\n#### arch-deduplicate\n\nDe-duplicates the entire archive by removing repeating patterns. This can save space\non the destination's drive while keeping the archived volumes intact.\n\nDe-duplication can also be performed incrementally by using `--dedup` with `send`.\n\n\n```\n\nwyng arch-deduplicate\n\n```\n\n\n#### arch-init\n\nCreate a new archive on a mounted drive...\n```\n\nwyng arch-init --dest=file:/mnt/backups/archive1\n\n```\n\nCreate a new archive with stronger compression on a remote system...\n```\n\nwyng arch-init --dest=ssh://user@example.com --compression=zstd:7\n\n```\n\nOptional parameters for `arch-init` are _encrypt, compression, hashtype_ and _chunk-factor_.\nThese cannot be changed for an archive after it is initialized.\n\n\n#### arch-check\n\nIntensive check of archive integrity, reading each session's _deltas_ completely starting with\nthe newest and working back to the oldest. This differs from `verify` which first builds a complete index and checks a complete volume.\n\nUsing `--session=newest` provides a 'verify the last session' function (useful after an incremental\nbackup). Otherwise, supplying a date-time will make `arch-check` start the check from that point and\nthen continue working toward the oldest session. Session ranges are not yet supported.\n\nDepending on how `arch-check` is used, the verification process can be shorter _or much longer_\nthan using `verify` as the latter is always the size of a volume snapshot. The longest, most\ncomplete form of `arch-check` is to supply no parameters, which checks all sessions in all volumes.\n\n\n\n#### monitor\n\nFrees disk space that is cumulatively occupied by aging snapshots, thereby addressing a\ncommon resource usage issue with snapshot-based backups.\nAfter harvesting their change metadata, the older snapshots are replaced with\nnew ones occupying zero space.  Running `monitor` isn't necessary,\nbut it only takes a few seconds and is good to run on a frequent, regular basis\nif you have some volumes that are write-intensive. Volume names may also be\nspecified if its desired to monitor only certain volumes.\n\nThis rule in /etc/cron.d runs `monitor` every 20 minutes:\n\n```\n*/20 * * * * root su -l -c '/usr/local/bin/wyng monitor --all'\n```\n\n\n#### diff\n\nCompare a local volume snapshot with the archive and report any differences.\nThis is useful for diagnostics and can also be useful after a verification\nerror has occurred.\n\n\n#### add\n\nAdds new, empty volume name(s) to the archive.  On subsequent `send -a`, Wyng will backup\nthe volume data if present.\n\n\n\u003cbr/\u003e\n\n### Parameters / Options summary\n\n| _Option_                      | _Description_\n|-------------------------------|--------------\n--dest=_URL_           | Location of backup archive.\n--local=_vg/pool_  _...or..._    | Storage pool containing local volumes.\n--local=_/absolute/path_    | \n--authmin=_N_          | Remember authentication for N minutes (default: 5)\n--all, -a              | Select all volumes (most cmds); Or clean all snapshots (delete).\n--volex=_volname_      | Exclude volumes (send, monitor, list, prune).\n--dedup, -d            | Use deduplication for send (see notes).\n--session=_date-time[,date-time]_ | Select a session or session range by date-time or tag (receive, verify, prune).\n--all-before           | Select all sessions before the specified _--session date-time_ (prune).\n--autoprune=off        | Automatic pruning by calendar date.\n--apdays=_A:B:C:D_     | Number of days to keep or to thin-out older sessions\n--keep=_date-time_     | Specify date-time or tag of sessions to keep (prune).\n--tag=tagname[,desc]   | Use session tags (send, list).\n--sparse               | Receive volume data sparsely (implies --sparse-write)\n--sparse-write         | Overwrite local data only where it differs (receive)\n--use-snapshot         | Use snapshots when available for faster `receive`.\n--send-unchanged       | Record unchanged volumes, don't skip them (send)\n--unattended, -u       | Don't prompt for interactive input.\n--clean                | Perform garbage collection (arch-check) or metadata removal (delete).\n--verbose              | Increase details.\n--quiet                | Shhh...\n\n\n### Advanced Options\n\n| _Option_                      | _Description_\n|-------------------------------|--------------\n--save-to=_path_       | Save a volume to _path_ (receive).\n--local_from=_json file_ | Specify local:[volumes] sets instead of --local.\n--import-other-from    | Import volume data from a non-snapshot capable path during `send`\n--session-strict=_on_ | Don't retrieve volume from next-oldest session if no exact session match\n--encrypt=_cipher_     | Set encryption mode or _'off'_ (default: _'xchacha20-dgr'_)\n--compression          | (arch-init) Set compression type:level.\n--hashtype             | (arch-init) Set data hash algorithm: _hmac-sha256_ or _blake2b_.\n--chunk-factor         | (arch-init) Set archive chunk size.\n--vid                  | Select volume by ID (delete)\n--tar-bypass           | Use direct access for file:/ archives (send)\n--passcmd=_'command'_  | Read passphrase from output of a wallet/auth app\n--upgrade-format       | Upgrade older Wyng archive to current format. (arch-check)\n--change-uuid          | Change the archive UUID to a new random value. (arch-check)\n--dry-run              | Make `send` session a dry run, see estimate of changed data.\n--remap                | Remap volume to current archive during `send` or `diff`.\n--json                 | Output volume: session info in json format (list).\n--force                | Not used with most commands.\n--force-allow-rollback | Accept archive if it was reverted to an earlier state.\n--opt-ssh              | Override internal _ssh_ options.\n--opt-qubes            | Override internal _qvm-run_ options.\n--meta-reduce=_mode:N_ | Reduce or extend local metadata caching.\n--meta-dir=_path_      | Use a different metadata dir than the default.\n--debug                | Debug mode\n\n\n\n\n### Option Details\n\n#### `--dest=URL`\n\nThis option tells Wyng where to access the archive and has the same meaning for all read or write\ncommands. It accepts one of the following forms:\n\n| _URL Form_ | _Destination Type_\n|----------|-----------------\n|__file:__/path                           | Local filesystem\n|__ssh:__//user@example.com[:port][/path]      | SSH server\n|__qubes:__//vm-name[/path]                     | Qubes virtual machine\n|__qubes-ssh:__//vm-name:me@example.com[:port][/path]  | SSH server via a Qubes VM\n\n\n#### `--local`\n\nThe location of local storage where logical volumes, disk images, etc. reside.  This serves as\nthe _source_ for `send` commands, and as the place where `receive` restores/saves volumes.\n\nThis parameter takes one of two forms: Either the source volume group and pool as 'vgname/poolname'\nor a directory path on a reflink-capable filesystem such as Btrfs or XFS (for Btrfs the path should\nend at a subvolume).  Required for commands `send`, `monitor` and `diff` (and `receive` when\nnot using `--saveto`).\n\n\n#### `--session=\u003cdate-time\u003e[,\u003cdate-time\u003e]` OR\n#### `--session=^\u003ctag\u003e[,^\u003ctag\u003e]`\n\nSession allows you to specify a single date-time or tag spec for the`receive`, `verify`, `diff`,\nand `arch-check` commands. Using a tag selects the last session having that tag. When specifying\na tag, it must be prefixed by a `^` carat.\n\nFor `prune`, specifying\na tag will have different effects: a single spec using a tag will remove only each individual session\nwith that tag, whereas a tag in a dual (range) spec will define an inclusive range anchored at the first\ninstance of the tag (when the tag is the first spec) or the last instance (when the tag is the\nsecond range spec). Also, date-times and tags may be used together in a range spec.\n\n\n#### `--volex=\u003cvolume1\u003e [--volex=\u003cvolume2\u003e *]`\n\nExclude one or more volumes from processing. May be used with commands that operate on multiple\nvolumes in a single invocation, such as `send`.  volex is useful in cases where a volume is\nin the archive, but frequent automatic backups aren't needed.  Or when certain volumes should\nbe excluded from prune, monitor, etc.\n\n**Please note:** volex syntax had to be changed from the v0.3 option syntax which used a comma to\nspecify multiple volumes.\n\n\n#### `--sparse-write`\n\nUsed with `receive`, the sparse-write mode tells Wyng not to create a brand-new local volume and\nresults in the data being sparsely written into the existing volume instead. This is useful if\nthe existing\nlocal volume is a clone/snapshot of another volume and you wish to save local disk space. It is also\nbest used when the backup/archive storage is local (i.e. fast USB drive or similar) and you don't\nwant the added CPU usage of full `--sparse` mode.\n\n\n#### `--sparse`\n\nThe sparse mode can be used with the `receive` command to intelligently retrieve and overwrite\nan existing\nlocal volume so that only the differences between local and archived volumes will be fetched\nfrom the archive and written to the local volume. This results in reduced network\nusage at the expense of some extra CPU usage on the local machine, and also uses\nless local disk space when snapshots are a factor.  The best situation for sparse mode is when\nyou want to restore/revert a large volume with a containing a limited number of changes\nover a low-bandwidth connection.\n\n\n#### `--use-snapshot` _(experimental)_\n\nA faster-than-sparse option that uses a snapshot as the baseline for the\n`receive`, if one is available.  Use with `--sparse` if you want Wyng to fall back to\nsparse mode when snapshots are not already present.\n\n\n#### `--tar-bypass` _(experimental)_\n\nUse direct access for file:/ archives during `send`.  This can reduce sending times by\nup to 20%.\n\n\n#### `--dedup`, `-d`\n\nWhen used with the `send` command, data chunks from the new backup will be sent only if\nthey don't already exist somewhere in the archive. Otherwise, a link will be used saving\ndisk space and possibly time and bandwith.\n\nThe trade-off for deduplicating is longer startup time for Wyng, in addition to using more\nmemory and CPU resources during backups. Using `--dedup` works best if you are backing-up\nmultiple volumes that have a lot of the same content and/or you are backing-up over a slow\nInternet link.\n\n\n#### `--dry-run`\n\nHave `send` perform a dry run, where no data is saved to the archive. This is useful for testing and also getting an estimate of the amount of data that will be transmitted during a normal `send`. If a volume that had lost its snapshot or delta map is included in a dry run, its map will be re-created automatically saving time on the next `send`.\n\nSince it affects the amount of data transmitted, including the `--dedup` option in the dry run is recommended if you intend to make the actual backup with `--dedup`.\n\n\n#### `--autoprune=(off | on | full)`\n\nAutoprune may be used with either the `prune` or `send` commands and will cause Wyng to\nautomatically remove older backup sessions according to date criteria. When used with `send`\nspecifically, the autopruning process will be triggered in advance of sending new sessions\nwhen using _full_ mode, or in _on_ mode only or if the destination filesytem is\nlow on free space.  (See _--apdays_ to specify additional autoprune parameters.)\n\nSelectable modes are:\n\n__off__ is the current default.\n\n__on__ removes _some_ sessions, as space is needed on the destination.\n\n__full__ removes all sessions that are due to expire according to above criteria.\n\n#### `--apdays=A:B:C:D`\n\nAdjust autoprune with the following four parameters:\n\n* A: The oldest day before which _all_ sessions are removed.  Default is 0 (disabled).\n* B: Thinning days; the number of days before which _some_ sessions will be removed\naccording to the ratio _D/C_.  Default is 62 days.\n* C: Number of _days_ for the D/C ratio.  Default is 1.\n* D: Number of _sessions_ for the D/C ratio.  Default is 2.\n\nAn example:  `--apdays=365:31:1:2` will cause autoprune to remove all sessions that are older\nthan 365 days, and sessions older than 31 days will be thinned-out while preserving\n(roughly on average) two sessions per day.\n\n#### `--tag=\u003ctagname[,description]\u003e`\n\nWith `send`, attach a tag name of your choosing to the new backup session/snapshot; this may be\nrepeated on the command line to add multiple tags. Specifying an empty '' or '@' tag will cause Wyng to ask for one or more tags to be manually input; this also causes `list` to display tag\ninformation when listing sessions.\n\n#### `--authmin=\u003cminutes\u003e`\n#### `--passcmd=\u003ccommand\u003e`\n\nThese two options help automate Wyng authentication, and may be used together or separately.\n\n`--authmin` takes a numeric value from -1 to 60 for the\nnumber of minutes to remember the current authentication for subsequent Wyng invocations.\nThe default authmin time is 2 minutes.  Specifying a -1 will cancel a prior authentication\nand 0 will skip storing the authentication.\n\nThe `--passcmd` option takes a string representing a shell command that outputs a passphrase\nto _stdout_ which\nWyng then reads instead of prompting for passphrase input.  If a prior auth from\n`--authmin` is active, this option is ignored and the command will not be executed.\n\n\n#### `--import-other-from=volname:|:path`\n\nEnables `send`ing a volume from a path that is not a supported snapshot storage type.  This may\nbe any regular file or a block device which is seek-able.\n\nWhen it is specified this option causes slow delta comparisons to be used for the specified volume(s)\ninstead of the default fast snapshot-based delta comparisons.  It is not recommended for regular\nuse with large volumes.\n\nThe special delimeter used to separate the _volname_ (archive volume name) and the _path_ is ':|:'\nwhich means this option cannot be used to `send` directly to volume names in the archive which\ncontain that character sequence.\n\n\n#### `--session-strict=on|off`\n\nFor receive, verify, diff: If set to 'on' (the default) Wyng won't retrieve volumes from next-oldest session if the\nspecified volumes don't have an exact match for the specified session.  When set to 'off'\nWyng will try to retrieve the next-oldest version of the volume if one exists.\n\n\n#### `--local_from=_json file_`\n\nSpecify both local storage and volume names for `send` or `receive` as sets, instead\nof using --local and volume names on the command line.  The json file must take the form\nof `{local-a: [[volname1, alias1], [volnameN, aliasN], ...], ...]}`.  This allows multiple\nlocal storage sources to be sent/received in a single session.\n\n_Alias_ can be _'null'_ for no alias or any valid name. However, the volume names (or aliases)\nmust all be unique across different sources as they are stored in the same archive.  Aliases define which local volume name into which an archive volume will be received, or when sending\nthey indicate a request to actually _rename_ the target volume to the alias.\n\n\n#### `--meta-reduce=mode:minutes`\n\nControl the degree to which locally cached session metadata is retained or removed when\nWyng exits. This can effect a noticeable reduction in the space that Wyng uses in /var\nwhile trading off a little speed.\n\n___Mode___ is one of _off, on,_ or _extra_: _off_ results in no reduction (all metadata is\nretained); _on_ removes uncompressed metadata; _extra_ removes both compressed and uncompressed\nmetadata.\n\n___Minutes___ is an integer defining the metadata's maximum age in minutes, where '0'\nwill cause it to be removed immediately when Wyng exits.\n\nThe default setting is _'on:3000'_.\n\n\n#### `--compression`\n\nAccepts the forms `type` or `type:level`. The three types available are `zstd` (zstandard),\nplus `zlib` and `bz2` (bzip2). Note that Wyng will only default\nto `zstd` when the 'python3-zstd' package is installed; otherwise it will fall back to the less\ncapable `zlib`. (default=zstd:3)\n\n\n#### `--hashtype`\n\nAccepts a value of either _'blake2b'_ or _'hmac-sha256'_ (default).  The digest size is 256 bits.\n\n\n#### `--chunk-factor`\n\nSets the pre-compression data chunk size used within the destination archive.\nAccepted range is an integer exponent from '1' to '6', resulting in a chunk size of 64kB for\nfactor '1', 128kB for factor '2', 256kB for factor '3' and so on. To maintain a good\nspace efficiency and performance balance, a factor of '2' or greater is suggested for archives\nthat will store volumes larger than about 100GB. (default=2)\n\n\n#### `--encrypt`\n\nSelects the encryption cipher/mode.  The available modes are:\n\n- `xchacha20-dgr` — Using HMAC-SHA256(rnd||hash) function.  This is the default.\n- `xchacha20-msr` — Using HMAC-SHA256(rnd||msg) function.\n- `xchacha20-ct` — Counter based; fast like _*-dgr_ with different safety trade-offs (see issue [158](https://github.com/tasket/wyng-backup/issues/158)).\n- `off` — Turns off Wyng's authentication and encryption.\n\n\n\n\n### Configuration files\n\nWyng will look in _'/etc/wyng/wyng.ini'_ for option defaults.  For options that are flags with\nno value like `--dedup`, use a _1_ or _0_ to indicate _enable_ or _disable_ (yes or no).\nFor options allowing multiple entries per command line, in the .ini use multiple lines with the\n2nd item onward indented by at least one space.\n\nAn example _wyng.ini_ file:\n\n```\n[var-global-default]\ndedup = 1\nauthmin = 10\nautoprune = full\ndest = ssh://user@192.168.0.8/home/user/wyng.backup\nlocal = /mnt/btrfs01/vms\nvolex = misc/caches.img\n  misc/deprecated_apps.img\n  windows10_recovery.vmdk\n```\n\n\n### Verifying Code\n\nWyng code can be cryptographically verified using either `gpg` directly or via `git`:\n\n```sh\n# Import Key\n~$ cd wyng-backup\n~/wyng-backup$ gpg --import pubkey\ngpg: key 1DC4D106F07F1886: public key \"Christopher Laprise \u003ctasket@posteo.net\u003e\" imported\ngpg: Total number processed: 1\ngpg:               imported: 1\n\n# GPG Method\n~/wyng-backup$ gpg --verify src/wyng.gpg src/wyng\n\n# Git Method\n~/wyng-backup$ git verify-commit HEAD\n\n# Output:\ngpg: Signature made Sat 26 Aug 2023 04:20:46 PM EDT\ngpg:                using RSA key 0573D1F63412AF043C47B8C8448568C8B281C952\ngpg: Good signature from \"Christopher Laprise \u003ctasket@posteo.net\u003e\" [unknown]\ngpg:                 aka \"Christopher Laprise \u003ctasket@protonmail.com\u003e\" [unknown]\n```\n\n\n### Security notes\n\n#### Automated authentication:\n\nWyng supports two modes of supplying passphrase secrets:  Standard input\nand the `--passcmd` option. The former can accept a secret from a pipe or\nredirect because when auth is necessary it is always the first input prompt.\nHowever, the prompt may not always occur when `--authmin` value \u003e 0 is used since\nthe passphrase may not be needed for repeat invocations of Wyng.\n\n#### Persistence of cached archive.ini \u0026 archive.salt:\n\nAuthentication schemes in general can only verify the authenticity for an\nobject at any point in time; they aren't well suited to telling us if that object\n(i.e. a backup archive) is the most recent update, and so they are vulnerable to rollback\nattacks that replace your current archive with an older version (in Wyng this is related to\nreplay attacks, but not downgrade attacks).  Wyng guards against\nsuch attacks by checking that the time encoded in your locally cached archive.ini isn't newer\nthan the one on the destination/remote; Wyng also displays the last archive modification time\nwhenever you access it.\n\n\n#### Protecting and Verifying Archive Authenticity:\n\nWith encryption enabled, Wyng provides a kind of built-in verification of archive authenticity;\nthis is because it uses an AEAD cipher mode.  However, custom verification\n(BYOV) is also possible with Wyng and even works on non-encrypted archives.  All you need\nto do is sign the 'archive.ini' file from the top archive directory after executing any Wyng\ncommand that changes the archive (i.e. _arch-init, add, send, prune, delete, rename_).\n\nSubsequently, the steps to verify total archive authenticity would be to simply run\n`wyng arch-check --dest \u003cURL\u003e` (using Wyng's built-in authenticated encryption), or else using custom\nauthentication based on GPG, for instance:\n```\ncd /mnt/backups\ngpg --verify laptop1.sig laptop1.backup/archive.ini \u0026\u0026 wyng arch-check --dest=file:/mnt/backups/laptop1.backup\n```\nNote that custom signature files should _not_ be stored within the archive directory.\n\n(Although volumes can be verified piecemeal with the `wyng verify` command, it is not suited\nto verifying everything within an archive.)\n\n\n### Tips \u0026 Caveats\n\n* Qubes users: If you're using Wyng to backup Qubes VMs, you probably want to use the Wyng wrapper made especially for Qubes, [wyng-util-qubes](https://github.com/tasket/wyng-util-qubes), which makes saving \u0026 restoring VM settings along with data easy!\n\n* LVM users: Wyng has an internal snapshot manager which creates snapshots of volumes\nin addition to any snapshots you may already have on your local storage system.\nThis can pose a serious challenge to _lvmthin_ (aka thin-provisioned LVM) as the default space\nallocated for metadata is often too small for rigorous \u0026 repeated snapshot rotation\ncycles.  It is recommended to _at least double_ the existing or default tmeta space\non each thin pool used with `wyng send` or `wyng monitor`; see the man page\nsection _[Manually manage free metadata space of a thin pool LV](https://www.linux.org/docs/man7/lvmthin.html)_ for guidance on using\nthe `lvextend --poolmetadatasize` command.\n\n* Btrfs users: In general it is a good idea to use disk images on a Btrfs filesystem that is\nrelatively recent and well-maintained to ensure that fragmentation does not cause\nnoticeable slowdowns.  The simplest way to maintain a responsive filesystem is to\ndefragment image files monthly or weekly as it is a single command and typically takes\nonly a few minutes (there is no need to dismount the images).  For example:\n\n```\n      sudo btrfs filesystem defragment -r -t 256K /var/lib/qubes\n```\n\n\u003cul\u003e\nNote that while the 'autodefrag' mount option can be used as an alternative, the overall performance will be reduced due to the smaller fragments and constant effect of write-amplification.\u003c/ul\u003e\n\n\u003cul\u003e\nBtrfs deduplicators like `bees` or `duperemove` can quickly increase fragmentation\n(i.e. undo the effects of _defragment_) if not used carefully. The `duperemove` docs\nindicate that choosing a larger block size will limit fragmentation, making it\npreferable to _bees_. The block `-b` value should be matched to the one used \nfor `defragment -t`.\u003c/ul\u003e\n\n* Receive/restore and deduplication:  Wyng `receive` can prevent data duplication when there is an existing volume to over-write; in sparse mode it will compare existing data on disk with incoming data from the archive and avoid writing to areas that match.  The `--use-snapshot` option has a similar space-saving effect and may be combined with the _sparse_ options.  However, Wyng cannot tell if any other volumes on the system are related to the volumes being received, and it won't automatically use them as a starting point to reduce consumption of disk space; to attain the 'dedup' effect when restoring its up to you to create snapshots from related volumes at your respective receive paths/LVs before running `wyng --sparse receive`.\n\n* To reduce the size of incremental backups it may be helpful to remove or isolate cache\nfiles, if they exist in your source volumes. Typically, the greatest cache space\nconsumption comes from web browsers, so\nvolumes holding paths like /home/user/.cache can be impacted by this, depending\non the amount and type of browser use associated with the volume. Three possible\napproaches are to clear caches on browser exit, delete /home/user/.cache dirs on\nsystem/container shutdown (this reasonably assumes cached data is expendable),\nor to mount .cache on a separate volume that is not configured for backup.\n\n* If you've changed your local path without first running `wyng delete --clean` to\nremove snapshots, there may be orphaned snapshots remaining under your old volume group\nor local directory. Deleting them can prevent unnecessary consumption of disk space.  LVM snapshots can be found with the patterns `'*.tick'` and `'*.tock'` with\nthe tag \"wyng\".  Btrfs/XFS snapshots can be found with `'sn*.wyng?'`.\n\n* Keeping a [duplicate](https://github.com/tasket/wyng-backup/issues/199) archive or \"a backup of the backup\" is possible with the following:\n```\n      mv destpath destpath-incomplete\n      rsync -uaHW --delete --no-compress sourcepath/. destpath-incomplete\n      mv destpath-incomplete destpath\n```\n\n\u003cul\u003e\nThe simple rsync example above can become bogged-down with unnecessary transfers because it doesn't take into account when pruning shifts data into different paths.  A script that addresses this performance pitfall is available \u003ca href=\"https://gist.github.com/tasket/08f38279d8702c7defcb62cb4afdae7a\"\u003ehere\u003c/a\u003e.\u003c/ul\u003e\n\u003cul\u003e\nNote that since a duplicate archive is identical, including internal UUIDs, it should only be kept for an emergency such as when the original archive is no longer available or becomes unusable. Switching back and forth between the original and duplicate for regular archival operations is not supported.\u003c/ul\u003e\n\n* _Sending to multiple archives:_ If you have created separate archives (not duplicates as described in the last section) and want to backup one or more volumes to both archives, Wyng can do this seamlessly from non-LVM storage systems such as Btrfs.  With LVM, the --remap option would have to be used each time you switch archives; this slows Wyng down to the pace of a regular incremental backup program, so keeping a duplicate archive using rsync or similar may be preferable.  However, this issue doesn't affect sending different sets of volumes to different archives, only when a specific volume is sent to more than one archive.\n\n* Wyng archives should be stored on Unix-like filesystems that were formatted with default or close-to default metadata settings (i.e. _inode_ capacity has not been manually reduced). Any format providing a 16KB:1 (or lower) data-to-inode ratio should work fine regardless of the Wyng chunksize being used.\n\n\n### Troubleshooting notes\n\n* If you encounter an error during send/backup that the Btrfs path is \"not a subvolume\" its probably due to the `--local` path or pool ending in a dir not a subvolume. If you can't adjust the local path to end at an existing subvol, then a dir can be easily converted using `btfs subvol create` and `mv` commands. See the 'misc' project folder for an example dir-to-subvol conversion script.\n\n* A major change in v0.8 is that for `send` and `monitor` Wyng will no longer assume you want to\nact on all known volumes if you don't specify any volumes.  You must now use `-a` or `--all`, which\nnow work for other commands as well.  This change also enables adding new volumes while doing a\ncomplete backup, for instance: `wyng -a send my-new-volume` – updates every volume already\nin the archive plus backup 'my-new-volume' as well.\n\n* Backup sessions shown in `list` output may be seemingly (but not actually) out of\norder if the system's local time shifts\nsubstantially between backups, such as when moving between time zones (including DST).\nIf this results in undesired selections with `--session` ranges, its possible\nto nail down the precisely desired range by observing the output of\n`list volumename` and using exact date-times from the listing.\n\n* Wyng locally stores information about backups in two ways:  Snapshots alongside your local\nsource volumes, and metadata under _/var/lib/wyng_.  It is safe to _delete_\nWyng snapshots without risking the integrity of backups (although `send` will become slower).\nHowever, as with all CoW snapshot based backup tools, you should never attempt to directly mount, alter or otherwise utilize a Wyng snapshot\nas this could (very likely) result in future backup sessions being corrupt (this is why Wyng\nsnapshots are stored as read-only).  If you think you have somehow altered a Wyng snapshot, you\nshould consider it corrupt and immediately delete it before the next `send`.\nIf you're in a pinch and need to use the data in a Wyng snapshot, you should first make your own\ncopy or snapshot of the Wyng snapshot using `cp --reflink` or `lvcreate -s` and use that instead.\n\n* Error _\"Cached metadata is newer\"_ indicates that something has reverted the archive to an earlier state.  This could be due to a rollback attack, but could also be the result of your own actions such as keeping multiple copies of the same archive and alternately mounting them at the same location (in which case giving each copy a slightly different dir name can avert this error\".  Use the `--force-allow-rollback` option if you need to recover from this error and use the older archive as is.\n\n* Metadata cached under _/var/lib/wyng_ may also be manually deleted.  However, the _archive.\\*_\nroot files in each 'a_*' directory are part of Wyng's defense against rollback attacks, so if you\nfeel the need to manually reclaim space used in this dir then consider leaving the _archive.\\*_\nfiles in place.\n\n* If data corruption in the archive is suspected, use `wyng arch-check` which will scan for errors and show options for recovery.\n\n* If a volume becomes damaged and unrecoverable it may be necessary to delete it by its volume ID by using `wyng delete --vid` instead of the volume name.\n\n### Testing\n\n* Wyng v0.4alpha3 and later no longer create or require the `wyng.backup040/default`\ndirectory structure.  This means whatever you specify\nin `--dest` is all there is to the archive path.  It also means accessing an alpha1 or\nalpha2 archive will require you to either include those dirs explicitly in your --dest path\nor rename '../wyng.backup040/default' to something else you prefer to use.\n\n* Testing goals are basically stability, usability, security and efficiency. Compatibility\nis also a valued topic, where source systems are generally expected to be a fairly recent\nLinux distro or Qubes OS. Destination systems can vary a lot, they just need to have Python and\nUnix commands or support a compatible FUSE protocol such as sshfs(sftp) or s3.\n\n\n### Donations\n\n\u003ca href=\"https://liberapay.com/tasket/donate\"\u003e\u003cimg alt=\"Donate using Liberapay\" src=\"media/lp_donate.svg\" height=54\u003e\u003c/a\u003e\n\n\u003ca href=\"https://ko-fi.com/tasket\"\u003e\u003cimg src=\"media/ko-fi.png\" height=57\u003e\u003c/a\u003e \u003ca href=\"https://ko-fi.com/tasket\"\u003eKo-Fi donate\u003c/a\u003e\n\n\u003ca href=\"https://www.buymeacoffee.com/tasket\"\u003e\u003cimg src=\"media/buymeacoffee_57.png\" height=57\u003e\u003c/a\u003e \u003ca href=\"https://www.buymeacoffee.com/tasket\"\u003eBuy me a coffee!\u003c/a\u003e\n\n\nIf you like this project, monetary contributions are welcome and can\nbe made through [Liberapay](https://liberapay.com/tasket/donate) or [Ko-Fi](https://ko-fi.com/tasket) or [Buymeacoffee](https://www.buymeacoffee.com/tasket).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasket%2Fwyng-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftasket%2Fwyng-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasket%2Fwyng-backup/lists"}