{"id":18000940,"url":"https://github.com/abbbi/virtnbdbackup","last_synced_at":"2025-05-16T05:04:43.901Z","repository":{"id":37412096,"uuid":"312078818","full_name":"abbbi/virtnbdbackup","owner":"abbbi","description":"Backup utility for  Libvirt / qemu / kvm supporting incremental and differential backups + instant recovery (agentless).","archived":false,"fork":false,"pushed_at":"2025-04-10T09:09:39.000Z","size":1863,"stargazers_count":388,"open_issues_count":22,"forks_count":47,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-10T10:43:40.429Z","etag":null,"topics":["backup","backup-image","backup-manager","backup-script","backup-solution","backup-utility","cbt","differential","incremental","incremental-backups","kvm","kvm-backup","libvirt","libvirt-backup","nbd","nbd-client","qemu","qemu-kvm","virtual-machine","virtualization"],"latest_commit_sha":null,"homepage":"http://libvirtbackup.grinser.de/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abbbi.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"abbbi"}},"created_at":"2020-11-11T20:12:16.000Z","updated_at":"2025-04-10T09:09:43.000Z","dependencies_parsed_at":"2023-10-12T01:08:58.930Z","dependency_job_id":"5e45c91d-55a2-4cf6-982b-dc87d50512b3","html_url":"https://github.com/abbbi/virtnbdbackup","commit_stats":null,"previous_names":[],"tags_count":162,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbbi%2Fvirtnbdbackup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbbi%2Fvirtnbdbackup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbbi%2Fvirtnbdbackup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abbbi%2Fvirtnbdbackup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abbbi","download_url":"https://codeload.github.com/abbbi/virtnbdbackup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254471060,"owners_count":22076585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","backup-image","backup-manager","backup-script","backup-solution","backup-utility","cbt","differential","incremental","incremental-backups","kvm","kvm-backup","libvirt","libvirt-backup","nbd","nbd-client","qemu","qemu-kvm","virtual-machine","virtualization"],"created_at":"2024-10-29T23:15:29.946Z","updated_at":"2025-05-16T05:04:43.875Z","avatar_url":"https://github.com/abbbi.png","language":"Python","readme":"![ci](https://github.com/abbbi/virtnbdbackup/actions/workflows/ci-ubuntu-latest.yml/badge.svg)\n[![package-build](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml/badge.svg)](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml)\n\n# virtnbdbackup\n\nBackup utility for `libvirt`, using the latest changed block tracking features.\nCreate online, thin provisioned full and incremental or differential backups\nof your `kvm/qemu` virtual machines.\n\n![Alt text](screenshot.jpg?raw=true \"Title\")\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n- [About](#about)\n- [Prerequisites](#prerequisites)\n  - [Libvirt versions \u003c= 7.6.0 (Debian Bullseye, Ubuntu 20.x)](#libvirt-versions--760-debian-bullseye-ubuntu-20x)\n  - [RHEL/Centos Stream, Alma, Rocky Linux](#rhelcentos-stream-alma-rocky-linux)\n    - [Version \u003c= 8.5](#version--85)\n    - [Version \u003e= 8.6](#version--86)\n  - [Environment dependencies](#environment-dependencies)\n- [Installation](#installation)\n  - [Python package](#python-package)\n  - [RPM package](#rpm-package)\n  - [Debian package](#debian-package)\n  - [Virtualenv](#virtualenv)\n  - [Docker images](#docker-images)\n- [Backup modes and concept](#backup-modes-and-concept)\n- [Supported disk formats / raw disks](#supported-disk-formats--raw-disks)\n- [Backup Examples](#backup-examples)\n  - [Local full/incremental backup](#local-fullincremental-backup)\n  - [Backing up offline virtual domains](#backing-up-offline-virtual-domains)\n  - [Application consistent backups](#application-consistent-backups)\n  - [Rotating backups](#rotating-backups)\n  - [Excluding disks](#excluding-disks)\n  - [Estimating differential/incremental backup size](#estimating-differentialincremental-backup-size)\n  - [Backup threshold](#backup-threshold)\n  - [Backup concurrency](#backup-concurrency)\n  - [Compression](#compression)\n  - [Remote Backup](#remote-backup)\n    - [QEMU Sessions](#qemu-sessions)\n    - [NBD with TLS (NBDSSL)](#nbd-with-tls-nbdssl)\n    - [Using a separate network for data transfer](#using-a-separate-network-for-data-transfer)\n    - [Piping data to other hosts](#piping-data-to-other-hosts)\n  - [Kernel/initrd and additional files](#kernelinitrd-and-additional-files)\n- [Restore examples](#restore-examples)\n  - [Dumping backup information](#dumping-backup-information)\n  - [Verifying created backups](#verifying-created-backups)\n  - [Complete restore](#complete-restore)\n  - [Process only specific disks during restore](#process-only-specific-disks-during-restore)\n  - [Point in time recovery](#point-in-time-recovery)\n  - [Restoring with modified virtual machine config](#restoring-with-modified-virtual-machine-config)\n  - [Remote Restore](#remote-restore)\n- [Post restore steps and considerations](#post-restore-steps-and-considerations)\n- [Single file restore and instant recovery](#single-file-restore-and-instant-recovery)\n- [Transient virtual machines: checkpoint persistency on clusters](#transient-virtual-machines-checkpoint-persistency-on-clusters)\n- [Supported Hypervisors](#supported-hypervisors)\n  - [Ovirt, RHEV or OLVM](#ovirt-rhev-or-olvm)\n  - [OpenNebula](#opennebula)\n- [Authentication](#authentication)\n- [Internals](#internals)\n  - [Backup Format](#backup-format)\n  - [Extents](#extents)\n  - [Backup I/O and performance: scratch files](#backup-io-and-performance-scratch-files)\n  - [Debugging](#debugging)\n- [FAQ](#faq)\n  - [The thin provisioned backups are bigger than the original qcow images](#the-thin-provisioned-backups-are-bigger-than-the-original-qcow-images)\n  - [Backup fails with \"Cannot store dirty bitmaps in qcow2 v2 files\"](#backup-fails-with-cannot-store-dirty-bitmaps-in-qcow2-v2-files)\n  - [Backup fails with \"unable to execute QEMU command 'transaction': Bitmap already exists\"](#backup-fails-with-unable-to-execute-qemu-command-transaction-bitmap-already-exists)\n  - [Backup fails with \"Bitmap inconsistency detected: please cleanup checkpoints using virsh and execute new full backup\"](#backup-fails-with-bitmap-inconsistency-detected-please-cleanup-checkpoints-using-virsh-and-execute-new-full-backup)\n  - [Backup fails with \"Error during checkpoint removal: [internal error: bitmap 'XX' not found in backing chain of 'XX']\"](#backup-fails-with-error-during-checkpoint-removal-internal-error-bitmap-xx-not-found-in-backing-chain-of-xx)\n  - [Backup fails with \"Virtual machine does not support required backup features, please adjust virtual machine configuration.\"](#backup-fails-with-virtual-machine-does-not-support-required-backup-features-please-adjust-virtual-machine-configuration)\n  - [Backup fails with \"Timed out during operation: cannot acquire state change lock\"](#backup-fails-with-timed-out-during-operation-cannot-acquire-state-change-lock)\n  - [Backup fails with \"Failed to bind socket to /var/tmp/virtnbdbackup.XX: Permission denied\"](#backup-fails-with-failed-to-bind-socket-to-vartmpvirtnbdbackupxx-permission-denied)\n  - [High memory usage during backup](#high-memory-usage-during-backup)\n  - [fstrim and (incremental) backup sizes](#fstrim-and-incremental-backup-sizes)\n  - [Test your backups!](#test-your-backups)\n  - [Links](#links)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n\n# About\n\nExisting backup solutions or scripts for `libvirt/kvm` usually depend on the\nexternal snapshot feature to create backups, sometimes even require to\nshutdown or pause the virtual machine.\n\nRecent additions to both the `libvirt` and `qemu` projects have introduced new\ncapabilities that allow to create online (full and incremental) backups, by\nusing so called `dirty bitmaps` (or changed block tracking).\n\n`virtnbdbackup` uses these features to create online full and incremental\nor differential backups.\n\n`virtnbdrestore` can be used to re-construct the complete image from the\nthin provisioned backups.\n\n`virtnbdmap` can be used to map an thin provisioned backup image into a\nblock device on-the-fly, for easy single file restore or even instant\nboot from an backup image.\n\nFor backing up standalone qemu virtual machines not managed by libvirt, see\nthis project: [qmpbackup](https://github.com/abbbi/qmpbackup)\n\n# Prerequisites\n\nObviously you require a libvirt/qemu version that supports the incremental\nbackup features. Since libvirt v7.6.0 and qemu-6.1 the required features are\n[enabled by default](https://libvirt.org/news.html#v7-6-0-2021-08-02) and are\nconsidered production ready: everything will work out of the box.\n\nFollowing, you will find a short overview which older libvirt\nversions may require further adjustments to the virtual machine config.\n\n\n## Libvirt versions \u003c= 7.6.0 (Debian Bullseye, Ubuntu 20.x)\n\nIf you are using Debian Bullseye or Ubuntu 20.x, the included libvirt version\nalready has an almost complete support for incremental backup, although it\ndoesn't work properly with migration or some block jobs.\n\nIf you don't want to use migration or other blockjobs you can enable the \nincremental backup feature on these libvirt versions. Change the virtual\nmachine config using `virsh edit \u003cvm\u003e` like so: (the first line must be \nchanged, too!):\n\n ```\n  \u003cdomain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'\u003e\n  [..]\n  \u003cqemu:capabilities\u003e\n    \u003cqemu:add capability='incremental-backup'/\u003e\n  \u003c/qemu:capabilities\u003e\n  [..]\n  \u003c/domain\u003e\n ```\n\n`Note`:\n\u003e You must power cycle the virtual machine after enabling the feature!\n\u003e Upstream libvirt strongly discourages enabling the feature on production\n\u003e systems for these libvirt versions.\n\n## RHEL/Centos Stream, Alma, Rocky Linux\n\n### Version \u003c= 8.5\n\nUp to RHEL/Centos8/Almalinux 8.5, libvirt packages from the advanced\nvirtualization stream support all required features. To install libvirt from\nthe stream use:\n\n  ```\n  yum install centos-release-advanced-virtualization\n  yum makecache\n  yum module install virt\n  ```\n\nand enable the feature by adjusting the virtual machine config.\n\n### Version \u003e= 8.6\n\nAs of RHEL 8.6, the advanced virtualization stream has been deprecated,\nand all components supporting the new feature are included in the\nvirt:rhel module, the feature is enabled by default. [(Details)](https://access.redhat.com/solutions/6959344)\n\n## Environment dependencies\n \n * python libvirt module version  \u003e= 6.0.0 (yum install python3-libvirt)\n * python libnbd bindings (https://github.com/libguestfs/libnbd) version \u003e= `1.5.5` (yum install python3-libnbd)\n * The virtual machine should use qcow version 3 images to support the full feature set.\n \n# Installation\n\nThere are several ways to install the utility, below you will find an short\ndescription for each of them. For Debian and RHEL/SuSE based derivates see\n[releases](https://github.com/abbbi/virtnbdbackup/releases) for pre-built\npackages.\n\n`Note`:\n\u003e Please consider to check [past issues related to\n\u003e installation](https://github.com/abbbi/virtnbdbackup/issues?q=is%3Aissue+is%3Aclosed+label%3Ainstallation)\n\u003e if you face any troubles before opening a new issue.\n\n## Python package\n```\n git clone https://github.com/abbbi/virtnbdbackup \u0026\u0026 cd virtnbdbackup\n pip install .\n```\n\n`Note`:\n\u003e Do not install the \"nbd\" package available on PyPI, it does not provide the\n\u003e required nbd bindings (unfortunately has the same name). You have to\n\u003e additionally install the provided python3-libnbd packages by your\n\u003e distribution, or compile the libnbd bindings by yourself.\n\n## RPM package\n\nPackages for RHEL/Fedora and OpenSUSE are available via\n[releases](https://github.com/abbbi/virtnbdbackup/releases).\n\nTo create an RPM package from source by yourself you can follow the steps from\nthe github [build\nworkflow](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml).\n\n\n## Debian package\n\nOfficial packages are available:\n[https://packages.debian.org/virtnbdbackup](http://packages.debian.org/virtnbdbackup) and are maintained on\nthe [Debian salsa codespace](https://salsa.debian.org/debian/virtnbdbackup).\n\nFor the latest packages available check\n[releases](https://github.com/abbbi/virtnbdbackup/releases).\n\nTo create an Debian package from source by yourself you can follow the steps\nfrom the github [build\nworkflow](https://github.com/abbbi/virtnbdbackup/actions/workflows/build.yml).\n\n\n## Virtualenv\n\nFor setup within an virtualenv see [venv scripts](venv/).\n\n## Docker images\n\nYou can build an docker image using the existing [Dockerfile README](docker/)\n\nAll released versions and master branch are published via github container\nregistry, too. Example:\n\n```\n docker run -it ghcr.io/abbbi/virtnbdbackup:master virtnbdbackup\n```\n\nSee [packages](https://github.com/abbbi/virtnbdbackup/pkgs/container/virtnbdbackup).\n\n# Backup modes and concept\n\nFollowing backup modes can be used:\n\n* `auto`: If the target folder is empty, attempt to execute full backup,\n  otherwise switch to backup mode incremental: allows rotation of backup\n  into monthly folders.\n\n* `full`: Full, thin provisioned backup of the virtual machine, a new checkpoint\n  named `virtnbdbackup` will be created, all existent checkpoints from prior\n  backups matching this name will be removed: a new backup chain is created.\n  The Virtual machine must be online and running for this backup mode to work.\n\n* `copy`: Full, thin provisioned backup of the virtual machine disks, no\n  checkpoint is created for further incremental backups, existing checkpoints\n  will be left untouched. This is the default mode and works with qcow images\n  not supporting persistent bitmaps.\n\n* `inc`: Perform incremental backup, based on the last full or incremental\n  backup. A checkpoint for each incremental backup is created and saved.\n\n* `diff`: Perform differential backup: saves the current delta to the last\n  incremental or full backup.\n\nAll required information for restore is stored to the same directory,\nincluding the latest virtual machine configuration, checkpoint information,\ndisk data and logfiles.\n\nThe target directory must be rotated if a new backup set is created.\n\nIf the virtual domain is active and running, a backup job operation via\n`libvirt api` is started, which in turn initializes a new nbd server backend\nlistening on a local unix socket. This nbd backend provides consistent access\nto the virtual machines, disk data and dirty blocks. After the backup process\nfinishes, the job is stopped and the nbd server quits operation.\n\nIt is possible to backup multiple virtual machines on the same host system at\nthe same time, using separate calls to the application with a different target\ndirectory to store the data.\n\n# Supported disk formats / raw disks\n\n`libvirt/qemu` supports dirty bitmaps, required for incremental backups only\nwith qcow(v3) based disk images. If you are using older image versions, you can\nonly create `copy` backups, or consider converting the images to a newer\nformat using `qemu-img`:\n\n\u003e qemu-img convert -O qcow2 -o compat=1.1 disk-old.qcow2 disk.qcow2\n\nBy default `virtnbdbackup` will exclude all disks with format `raw` as well\nas direct attached (passthrough) disks such as LVM or ZVOL and ISCSI\nvolumes. These type of virtual disks do not support storing checkpoint/bitmap\nmetadata and do not support incremental/differential backup.\n[(more info)](https://patchew.org/QEMU/20210320093235.461485-1-pj@patrikjanousek.cz/)\n\nThis behavior can be changed if option `--raw` is specified, raw disks will\nthen be included during a `full` backup. This of course means that no thin\nprovisioned backup is created for these particular disks.\n\nDuring restore, these files can be copied \"as is\" from the backup folder and\nmust not be processed using `virtnbdrestore`.\n\n`Note:`\n\u003e The backup data for raw disks will only be crash consistent, be aware\n\u003e that this might result in inconsistent filesystems after restoring!\n\n# Backup Examples\n\nEach backup for a virtual machine must be saved to an individual target\ndirectory. Once the target directory includes an full backup, it can be used as\nbase for further incremental or differential backups.\n\n## Local full/incremental backup\n\nStart full backup of domain `vm1`, save data to `/tmp/backupset/vm1`:\n\n```\nvirtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1\n```\n\nStart incremental backup for domain `vm1`, backup only changed blocks to the\nlast full backup, the same directory is used as backup target:\n\n```\nvirtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1\n```\n\nThe resulting directory will contain both backups and all other files required\nto restore the virtual machine. Created logfiles can be used for analyzing\nbackup issues:\n\n```\n/tmp/backupset/vm1\n├── backup.full.05102021161752.log\n├── backup.inc.05102021161813.log\n├── checkpoints\n│   ├── virtnbdbackup.0.xml\n│   ├── virtnbdbackup.1.xml\n├── sda.full.data\n├── sda.inc.virtnbdbackup.1.data\n├── vm1.cpt\n├── vmconfig.virtnbdbackup.0.xml\n├── vmconfig.virtnbdbackup.1.xml\n```\n\n## Backing up offline virtual domains\n\nIf the virtual domain is not in running state (powered off) `virtnbdbackup`\nsupports `copy` and `inc/diff` backup modes. Incremental and differential\nbackups will then save the changed blocks since last created checkpoint.\n\nBackup mode `full` is changed to mode `copy`, because libvirt does not allow to\ncreate checkpoints for offline domains.\n\nThis behavior can be changed using the `-S` (`--start-domain`) option: prior to\nexecuting the backup, the virtual domain will then be started in `paused` state\nfor the time the backup is created: The virtual machines CPU's are halted, but\nthe running QEMU Process will allow all operations required to execute backups.\n\nUsing this option will allow for all range of backup types (full/diff/inc) and\nmakes most sense if used with the backup mode `auto`.\n\nAlso, the option won't alter the virtual domain state if it is already online,\nthus it can be used for backing up virtual machines whose state is unknown\nprior to backup.\n\n## Application consistent backups\n\nDuring backup `virtnbdbackup` attempts to freeze all file systems within the\ndomain using the qemu guest agent filesystem freeze and thaw functions.  In\ncase no qemu agent is installed or filesystem freeze fails, a warning is shown\nduring backup:\n\n```\nWARNING [..] Guest agent is not responding: QEMU guest agent is not connected\n```\n\nIn case you receive this warning, check if the qemu agent is installed and\nrunning in the domain.\n\nIt is also possible to specify one or multiple mountpoints used within\nthe virtual machine to freeze only specific filesystems, like so:\n\n`virtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 -F /mnt,/var`\n\nthis way only the underlying filesystems on */mnt* and */var* are frozen\nand thawed.\n\n`Note:`\n\u003e It is highly recommended to have an qemu agent running in the virtual\n\u003e domain to ensure file system consistency during backup!\n\n\n## Rotating backups\n\nWith backup mode `auto` it is possible to have a monthly rotation/retention.  If\nthe target folder is empty, backup mode auto will create an full backup. On the\nfollowing executions, it will automatically switch to backup mode incremental,\nif the target folder already includes an full backup. Example:\n\n```\nvirtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -\u003e creates full backup\nvirtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -\u003e creates inc backup\nvirtnbdbackup -d vm1 -l auto -o /tmp/2022-06 -\u003e creates inc backup\nvirtnbdbackup -d vm1 -l auto -o /tmp/2022-07 -\u003e creates full backup\nvirtnbdbackup -d vm1 -l auto -o /tmp/2022-07 -\u003e creates inc backup\n```\n\n## Excluding disks\n\nOption `-x` can be used to exclude certain disks from the backup. The name of\nthe disk to be excluded must match the disks target device name as configured\nin the domains xml definition, for example:\n\n```\nvirtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1 -x sda\n```\n\nSpecial devices such as `cdrom/floppy` or `direct attached luns` are excluded\nby default, as they are not supported by the changed block tracking layer.\n\nIt is also possible to only backup specific disks using the include option\n(`--include`, or `-i`):\n\n```\nvirtnbdbackup -d vm1 -l full -o /tmp/backupset/vm1 -i sdf\n```\n\n## Estimating differential/incremental backup size\n\nSometimes it can be useful to estimate the data size prior to executing the\nnext `incremental` or `differential` backup. This can be done by using the\noption `-p` which will query the virtual machine checkpoint information for the\ncurrent size:\n\n```\nvirtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 -p\n[..]\n[..] INFO virtnbdbackup - handleCheckpoints [MainThread]: Using checkpoint name: [virtnbdbackup.1].\n[..] INFO virtnbdbackup - main [MainThread]: Estimated checkpoint backup size: [24248320] Bytes\n```\n\n`Note:`\n\u003e Not all libvirt versions support the flag required to read the checkpoint\n\u003e size. If the estimated checkpoint size is always 0, your libvirt version\n\u003e might miss the required features.\n\n## Backup threshold\n\nIf an `incremental` or `differential` backup is attempted and the virtual machine\nis active, it is possible to specify an threshold for executing the backup\nusing the `--threshold` option. The backup will then only be executed if the\namount of data changed meets the specified threshold (in bytes):\n\n```\nvirtnbdbackup -d vm1 -l inc -o /tmp/backupset/vm1 --threshold 3311264\n[..]\n[..] INFO virtnbdbackup - handleCheckpoints [MainThread]: Using checkpoint name: [virtnbdbackup.1].\n[..] ]virtnbdbackup - main [MainThread]: Backup size [3211264] does not meet required threshold [3311264], skipping backup.\n```\n\n## Backup concurrency\n\nIf `virtnbdbackup` saves data to a regular target directory, it starts one\nthread for each disk it detects to speed up the backup operation.\n\nThis behavior can be changed using the `--worker` option to define an amount of\nthreads to be used for backup. Depending on how many disks your virtual machine\nhas attached, it might make sense to try a different amount of workers to see\nwhich amount your hardware can handle best.\n\nIf standard output (`-`) is defined as backup target, the amount of workers is\nalways limited to 1, to ensure a valid Zip file format.\n\n## Compression\n\nIt is possible to enable compression for the `stream` format via `lz4`\nalgorithm by using the `--compress` option. The saved data is compressed inline\nand the saveset file is appended with compression trailer including information\nabout the compressed block offsets. By default compression level `2` is set if\nno parameter is applied. Higher compression levels can be set via:\n\n `--compress=16`\n\nDuring the restore, `virtnbdrestore` will automatically detect such compressed\nbackup streams and attempts to decompress saved blocks accordingly.\n\nUsing compression will come with some CPU overhead, both lz4 checksums for\nblock and original data are enabled.\n\n## Remote Backup\n\nIt is also possible to backup remote libvirt systems. The most convenient way\nis to use ssh for initiating the libvirt connection (key authentication\nmandatory).\n\nBefore attempting an remote backup, please validate your environment meets the\nfollowing criteria:\n\n * DNS resolution (forward and reverse) must work on all involved systems.\n * SSH Login to the remote system via ssh key authentication (using ssh agent\n   or passwordless ssh key) should work without issues.\n * Unique hostnames must be set on all systems involved.\n   ([background](https://github.com/abbbi/virtnbdbackup/issues/117))\n * Firewall must allow connection on all ports involved.\n\nIf the virtual machine has additional files configured, as described in\n[Kernel/initrd and additional files](#kernelinitrd-and-additional-files), these\nfiles will be copied from the remote system via SSH(SFTP).\n\n### QEMU Sessions\n\nIn order to backup virtual machines from a remote host, you must specify an\n[libvirt URI](https://libvirt.org/uri.html) to the remote system.\n\nThe following example saves the virtual machine `vm1` from the remote libvirt\nhost `hypervisor` to the local directory `/tmp/backupset/vm1`, it uses the `root`\nuser for both the libvirt and ssh authentication:\n\n```\nvirtnbdbackup -U qemu+ssh://root@hypervisor/system --ssh-user root -d vm1 -o  /tmp/backupset/vm1\n```\n\nSee also: [Authentication](#authentication)\n\n`Note`:\n\u003e If you want to run multiple remote backups at the same time you need to pass\n\u003e an unique port for the NBD service used for data transfer via --nbd-port\n\u003e option for each backup session.\n\n### NBD with TLS (NBDSSL)\n\nBy default disk data received from a remote system will be transferred via\nregular NBD protocol. You can enable TLS for this connection, using the `--tls`\noption. Before being able to use TLS, you *must* configure the required\ncertificates on both sides. [See this\nscript](https://github.com/abbbi/virtnbdbackup/blob/master/scripts/create-cert.sh).\n\nSee the following documentation by the libvirt project for detailed\ninstructions how setup:\n\n https://wiki.libvirt.org/page/TLSCreateCACert\n\n`Note:`\n\u003e You should have installed at least version 1.12.6 of the libnbd library\n\u003e which makes the transfer via NBDS more stable [full background](https://github.com/abbbi/virtnbdbackup/issues/66#issuecomment-1196813750)\n\n### Using a separate network for data transfer\n\nIn case you want to use a dedicated network for the data transfer via NBD, you\ncan specify an specific IP address to bind the remote NBD service to via\n`--nbd-ip` option.\n\n### Piping data to other hosts\n\nIf the output target points to standard out (`-`), `virtnbdbackup` puts the\nresulting backup data into an uncompessed zip archive.\n\nA such, it is possible to transfer the backup data to different hosts, or pipe\nit to other programs.\n\nHowever, keep in mind that in case you want to perform incremental backups, you\nmust keep the checkpoint files on the host you are executing the backup utility\nfrom, until you create another full backup.\n\nIf output is set to standard out, `virtnbdbackup` will create the required\ncheckpoint files in the directory it is executed from.\n\nHere is an example:\n\n```\n # mkdir backup-weekly; cd backup-weekly\n # virtnbdbackup -d vm1 -l full -o - | ssh root@remotehost 'cat \u003e backup-full.zip'\n # [..]\n # INFO outputhelper - __init__: Writing zip file stream to stdout\n # [..]\n # INFO virtnbdbackup - main: Finished\n # INFO virtnbdbackup - main: Adding vm config to zipfile\n # [..]\n```\n\nAny subsequent incremental backup operations must be called from within this\ndirectory:\n\n```\n # cd backup-weekly\n # virtnbdbackup -d vm1 -l inc -o - | ssh root@remotehost 'cat \u003e backup-inc1.zip'\n [..]\n```\n\nYou may consider adding the created checkpoint files to some VCS system,\nlike git, to have some kind of central backup history tracking.\n\nDuring restore unzip the data from both zip files into a single directory:\n(use `virtnbdrestore` to reconstruct the virtual machine images):\n\n```\n # unzip -o -d restoredata backup-full.zip\n # unzip -o -d restoredata backup-inc1.zip\n```\n\n\n## Kernel/initrd and additional files\n\nIf an domain has configured custom kernel, initrd, loader or nvram images\n(usually the case if the domain boots from OVM UEFI BIOS), these files will be\nsaved to the backup folder as well.\n\n# Restore examples\n\nFor restoring, `virtnbdrestore` can be used. It reconstructs the streamed\nbackup format back into a usable qemu qcow image.\n\nThe restore process will create a qcow image with the original virtual size.\n\nIn a second step, the qcow image is then mapped to a ndb server instance where\nall data blocks are sent to and are applied accordingly. The resulting image\ncan be mounted (using `guestmount`) or attached to a running virtual machine in\norder to recover required files.\n\n## Dumping backup information\n\nAs a first start, the `dump` parameter can be used to dump the saveset\ninformation of an existing backup:\n\n```\nvirtnbdrestore -i /tmp/backupset/vm1 -o dump\nINFO:root:Dumping saveset meta information\n{'checkpointName': 'virtnbdbackup',\n 'dataSize': 704643072,\n 'date': '2020-11-15T20:50:36.448938',\n 'diskName': 'sda',\n 'incremental': False,\n 'parentCheckpoint': False,\n 'streamVersion': 1,\n 'virtualSize': 32212254720}\n[..]\n```\nThe output includes information about the thick and thin provisioned disk\nspace that is required for recovery, date of the backup and checkpoint chain.\n\n## Verifying created backups\n\nAs with version \u003e= 1.9.40  `virtnbdbackup` creates an check sum for each\ncreated data file. Using `virtnbdrestore` you can check the integrity for the\ncreated data files without having to restore:\n\n```\nvirtnbdrestore -i /tmp/backup/vm1 -o verify\n[..] INFO lib common - printVersion [MainThread]: Version: 1.9.39 Arguments: ./virtnbdrestore -i /tmp/backup/vm1 -o verify\n[..] INFO root virtnbdrestore - verify [MainThread]: Computing checksum for: /tmp/backup/vm1/sda.full.data\n[..] INFO root virtnbdrestore - verify [MainThread]: Checksum result: 541406837\n[..] INFO root virtnbdrestore - verify [MainThread]: Comparing checksum with stored information\n[..] INFO root virtnbdrestore - verify [MainThread]: OK\n```\n\nthis makes it easier to spot corrupted backup files due to storage issues.\n([background](https://github.com/abbbi/virtnbdbackup/issues/134))\n\n## Complete restore\n\nTo restore all disks within the backupset into a usable qcow image use\ncommand:\n\n```\nvirtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore\n```\n\nAll incremental backups found will be applied to the target images\nin the output directory `/tmp/restore`\n\n`Note`:\n\u003e The restore utility will copy the latest virtual machine config to the\n\u003e target directory, but won't alter its contents. You have to adjust the config\n\u003e file for the new paths and/or excluded disks to be able to define and run it.\n\n`Note`:\n\u003e Created disk images will be thin provisioned by default, you can change this\n\u003e behavior using option `--preallocate` to create thick provisioned images.\n\n## Process only specific disks during restore\n\nA single disk can be restored by using the option `-d`, the disk name has\nto match the virtual disks target name, for example:\n\n```\nvirtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore -d sda\n```\n\n## Point in time recovery\n\nOption `--until` allows to perform a point in time restore up to the desired\ncheckpoint. The checkpoint name has to be specified as reported by the\ndump output (field `checkpointName`), for example:\n\n```\nvirtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore --until virtnbdbackup.2\n```\n\nIt is also possible to specify the source data files specifically used for the\nrollback via `--sequence` option, but beware: you must be sure the sequence you\napply has the right order, otherwise the restored image might be errnous,\nexample:\n\n```\nvirtnbdrestore -i /tmp/backupset/vm1 -o /tmp/restore --sequence vdb.full.data,vdb.inc.virtnbdbackup.1.data\n```\n\n## Restoring with modified virtual machine config\n\nOption `-c` can be used to adjust the virtual machine configuration during\nrestore accordingly, the following changes are done:\n\n * UUID of the virtual machine is removed from the config file\n * Name of the virtual machine is prefixed with \"restore_\" (use option\n   `--name` to specify desired vm name)\n * The disk paths to the virtual machine are changed to the new target directory.\n * If virtual machine was operating on snapshots/backing store images, the\n   references to the configured backing stores will be removed.\n * Raw devices are removed from VM config if `--raw` is not specified, as well\n   as floppy or cdrom devices (which aren't part of the backup).\n\n`Note:`\n\u003e If missing, Kernel, UEFI or NVRAM files are restored to their original\n\u003e location as set in the virtual machine configuration.\n\nA restored virtual machine can then be defined and started right from the\nrestored directory (or use option `-D` to define automatically):\n\n```\nvirtnbdrestore -c -i /tmp/backupset/vm1 -o /tmp/restore\n[..]\n[..] INFO virtnbdrestore - restoreConfig [MainThread]: Adjusted config placed in: [/tmp/restore/vmconfig.virtnbdbackup.0.xml]\n[..] INFO virtnbdrestore - restoreConfig [MainThread]: Use 'virsh define /tmp/restore/vmconfig.virtnbdbackup.0.xml' to define VM\n```\n\n## Remote Restore\n\nRestoring to a remote host is possible too, same options as during backup\napply. The following example will restore the virtual machine from the local\ndirectory `/tmp/backupset` to the remote system \"hypervisor\", alter its\nconfiguration and register the virtual machine:\n\n```\nvirtnbdrestore -U qemu+ssh://root@hypervisor/system --ssh-user root -cD -i /tmp/backupset/vm1 -o /remote/target\n```\n\n# Post restore steps and considerations\n\nIf you restore the virtual machine with its original name on the same\nhypervisor, you may have to cleanup checkpoint information, otherwise backing\nup the restored virtual machine may fail, see [this\ndiscussion](https://github.com/abbbi/virtnbdbackup/discussions/48)\n\n\n# Single file restore and instant recovery\n\nThe `virtnbdmap` utility can be used to map uncompressed backup images from the\nstream format into an accessible block device on the fly. This way, you can\nrestore single files or even boot from an existing backup image without having\nto restore the complete dataset.\n\nThe utility requires `nbdkit with the python plugin` to be installed on the\nsystem along with required qemu tools (`qemu-nbd`) and an loaded nbd kernel\nmodule. It must be executed with superuser (root) rights or via sudo.\n\nThe following example maps an existing backup image to the network block\ndevice `/dev/nbd0`:\n\n```\n # modprobe nbd max_partitions=15\n # virtnbdmap -f /backupset/vm1/sda.full.data\n [..] INFO virtnbdmap - \u003cmodule\u003e [MainThread]: Done mapping backup image to [/dev/nbd0]\n [..] INFO virtnbdmap - \u003cmodule\u003e [MainThread]: Press CTRL+C to disconnect\n```\n\nWhile the process is running, you can access the backup image like a regular\nblock device:\n\n```\nfdisk -l /dev/nbd0\nDisk /dev/nbd0: 2 GiB, 2147483648 bytes, 4194304 sectors\n```\n\nYou can also create an mapped \"point in time\" recovery image by passing a\nsequence of full and incremental backups as parameter. The changes from the\nincremental backups will then be replayed to the block device on the fly and\nthe image will represent the latest state:\n\n```\nvirtnbdmap -f /backupset/vm1/sda.full.data,/backupset/vm1/sda.inc.virtnbdbackup.1.data,/backupset/vm1/sda.inc.virtnbdbackup.2.data\n[..]\n[..] INFO virtnbdmap - main [MainThread]: Need to replay incremental backups\n[..] INFO virtnbdmap - main [MainThread]: Replaying offset 420 from /backup/sda.inc.virtnbdbackup.1.data\n[..] INFO virtnbdmap - main [MainThread]: Replaying offset 131534 from /backup/sda.inc.virtnbdbackup.1.data\n[..]\n[..] INFO virtnbdmap - main [MainThread]: Replaying offset 33534 from /backup/sda.inc.virtnbdbackup.2.data\n[..] INFO virtnbdmap - \u003cmodule\u003e [MainThread]: Done mapping backup image to [/dev/nbd0]\n[..] INFO virtnbdmap - \u003cmodule\u003e [MainThread]: Press CTRL+C to disconnect\n[..]\n```\n\nThe original image will be left untouched as nbdkits copy on write filter is\nused to replay the changes.\n\nFurther you can create an overlay image via `qemu-img` and boot from it right\naway (or boot directly from the /dev/nbd0 device).\n\n```\nqemu-img create -b /dev/nbd0 -f qcow2 bootme.qcow2\nqemu-system-x86_64 -enable-kvm -m 2000 -hda bootme.qcow2\n```\n\nTo remove the mappings, stop the utility via \"CTRL-C\"\n\n`Note`:\n\u003e If the virtual machine includes volume groups, the system will attempt to\n\u003e set them online as you create the mapping, because the copy on write device \n\u003e is writable by default.\n\u003e If your host system is using the same volume group names this could lead to\n\u003e issues (check `dmesg` or `journalctl` then).\n\u003e In case the volume groups are online, it is recommended to change them to\n\u003e offline just before you remove the mapping, to free all references to the\n\u003e mapped nbd device (`vgchange -a n \u003cvg_name\u003e`)\n\n`Note`:\n\u003e If you map the image device with the `--readonly` option you may need to pass\n\u003e certain options to the mount command (-o norecovery,ro) in order to be able\n\u003e to mount the filesystems. This may also be the case if no qemu agent was\n\u003e installed within the virtual machine during backup.\n\n# Transient virtual machines: checkpoint persistency on clusters\n\nIn case virtual machines are started in transient environments, such as using\ncluster solutions like `pacemaker` situations can appear where the checkpoints\nfor the virtual machine defined by libvirt are not in sync with the bitmap\ninformation in the qcow files.\n\nIn case libvirt creates a checkpoint, the checkpoint information is stored\nin two places:\n\n * var/lib/libvirt/qemu/checkpoint/\u003cdomain_name\u003e \n * In the bitmap file of the virtual machines qcow image.\n\nDepending on the cluster solution, in case virtual machines are destroyed\non host A and are re-defined on host B, libvirt loses the information about\nthose checkpoints. Unfortunately `libvirtd` scans the checkpoint only once\nduring startup.\n\nThis can result in a situation, where the bitmap is still defined in the\nqcow image, but libvirt doesn't know about the checkpoint, backup then\nfails with:\n\n`Unable to execute QEMU command 'transaction': Bitmap already exists`\n\nBy default `virtnbdbackup` attempts to store the checkpoint information in the\ndefault backup directory, in situations where it detects a checkpoint is\nmissing, it attempts to redefine them from the prior backups.\n\nIn order to store the checkpoint information at some central place the option\n`--checkpointdir` can be used, this allows having persistent checkpoints\nstored across multiple nodes:\n\nAs example:\n\n 1) Create backup on host A, store checkpoints in a shared directory between\n hosts in `/mnt/shared/vm1`:\n\n`virtnbdbackup -d vm1 -l full -o /tmp/backup_hosta --checkpointdir /mnt/shared/vm1`\n\n 2) After backup, the virtual machine is relocated to host B and loses its\n information about checkpoints and bitmaps, thus, the next full backup\n usually fails with:\n\n```\nvirtnbdbackup -d vm1 -l full -o /tmp/backup_hostb\n[..]\nunable to execute QEMU command 'transaction': Bitmap already exists: virtnbdbackup.0\n```\n\n 3) Now pass the checkpoint dir and files written from host A, and\n virtnbdbackup will redefine missing checkpoints and execute a new full\n backup. As the new full backup removes all prior checkpoints the bitmap\n information is in sync after this operation and backup succeeds:\n\n```\nvirtnbdbackup -d vm1 -l full -o /tmp/backup_hostb --checkpointdir /mnt/shared/vm1\n[..]\nredefineCheckpoints: Redefine missing checkpoint virtnbdbackup.0\n[..]\n```\n\nSee also: https://github.com/abbbi/virtnbdbackup/pull/10\n\n# Supported Hypervisors\n\n`virtnbdbackup` uses the lowest layer on top of libvirt to allow its\nfunctionality, you can also use it with more advanced hypervisors solutions\nsuch as [ovirt](https://www.ovirt.org/), RHEV or OpenNebula, but please bear in\nmind that it was not developed to target all of those solutions specifically!\n\n## Ovirt, RHEV or OLVM\n\nIf you are using the ovirt node based hypervisor hosts you should consider\ncreating a virtualenv via the [venv scripts](venv/) and transferring it to the\nnode system.\n\nOn regular centos/alma/rhel based nodes, installation via RPM package should be\npreferred. The incremental backup functionality can be enabled via ovirt\nmanagement interface.\n\nUsually ovirt restricts access to the libvirt daemon via different\nauthentication methods. Use the `-U` parameter in order to specify an\nauthentication file, if you chose to run the utility locally on the\nhypervisor:\n\n```\nvirtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d vm1 -o /tmp/backupset/vm1\n```\n\nYou can also use remote backup functionality:\n\n * System must be reachable via ssh public key auth as described in the\n [Remote Backup](#remote-backup) section.\n * Some OVIRT based setups may deny SASL based authentication if the hostname\n   used to connect to does not match the hostname from the libvirt certificate.\n   [more info](https://github.com/abbbi/virtnbdbackup/issues/167#issuecomment-2028467071)\n * Firewall port for NBD must be open:\n\n```\n root@hv-node~# firewall-cmd --zone=public --add-port=10809/tcp\n```\n\nand then backup via:\n\n```\nvirtnbdbackup -U qemu+ssh://root@hv-node/session -d vm -o /backup --password password --user root --ssh-user root\n```\n\n`Note:`\n\u003e `virtnbdrestore` has not been adopted to cope with the ovirt specific\n\u003e domain xml format, so redefining and virtual machine on the node might not\n\u003e work.\n\n## OpenNebula\n\nSee [past issues](https://github.com/abbbi/virtnbdbackup/issues?q=label%3Aopennebula)\n\n# Authentication\n\nBoth `virtnbdbackup` and `virtnbdrestore` commands support authenticating\nagainst libvirtd with the usual URIs. Consider using the following options:\n\n `-U`: Specify an arbitrary connection URI to use against libvirt\n\n `--user`: Username to use for the specified connection URI\n\n `--password`: Password to use for the specified connection URI.\n\nIt is also possible to specify the credentials stored as authentication file\nlike it would be possible using the `virsh -c` option:\n\n```\n -U qemu:///system?authfile=/etc/virsh_auth.conf ..\n```\n\n`Note:`\n\u003e The default connection URI used is `qemu:///system` which is usually the \n\u003e case if virtual machines operate as root user. Use the `qemu:///session` URI\n\u003e to backup virtual machines as regular user.\n\n# Internals\n## Backup Format\n\nCurrently, there are two output formats implemented:\n\n * `stream`: the resulting backup image is saved in a streamlined format,\n   where the backup file consists of metadata about offsets and lengths\n   of zeroed or allocated contents of the virtual machines disk. This is\n   the default. The resulting backup image is thin provisioned.\n * `raw`: The resulting backup image will be a full provisioned raw image,\n   this should mostly be used for debugging any problems with the extent\n   handler, it won't work with incremental backups.\n\n## Extents\n\nIn order to save only used data from the images, dirty blocks are queried from\nthe NBD server. The behavior can be changed by using the option `-q` to use\ncommon qemu tools (nbdinfo). By default `virtnbdbackup` uses a custom\nimplemented extent handler.\n\n## Backup I/O and performance: scratch files\n\nIf virtual domains handle heavy I/O load during backup (such as writing or\ndeleting lots of data while the backup is active) you might consider using the\n`--scratchdir` option to change the default scratch file location.\n\nDuring the backup operation qemu will use the created scratch files for\nfleecing, thus it is recommended to store these files on storage that meets the\nsame I/O performance requirements as the backup target.\n\nThe free space on the default scratch directory (`/var/tmp`) must be enough to\nbe able to keep all fleecing data while the backup is active.\n\n## Debugging\n\nTo get more detailed debug output use `--verbose` option. To enable NBD\nspecific debugging output export LIBNBD_DEBUG environment variable prior to\nexecuting the backup or restore:\n\n```\nexport LIBNBD_DEBUG=1\nvirtnbdbackup [..] --verbose\n```\n\n# FAQ\n## The thin provisioned backups are bigger than the original qcow images\n\nVirtual machines using the qcow format do compress data. During backup, the\nimage contents are exposed as NBD device which is a RAW device. The backup data\nwill be at least as big as the used data within the virtual machine. \n\nYou can use the `--compress` option or other tools to compress the backup\nimages in order to save storage space or consider using a deduplication capable\ntarget file system.\n\n## Backup fails with \"Cannot store dirty bitmaps in qcow2 v2 files\"\n\nIf the backup fails with error:\n\n```\nERROR [..] internal error: unable to execute QEMU command dirty bitmaps in qcow2 v2 files\n```\n\nconsider migrating your qcow files to version 3 format. QEMU qcow image version\n2 does not support storing advanced bitmap information, as such only backup\nmode `copy` is supported.\n\n\n## Backup fails with \"unable to execute QEMU command 'transaction': Bitmap already exists\"\n\nDuring backup `virtnbdbackup` creates a so called \"checkpoint\" using the\nlibvirt API. This checkpoint represents an \"bitmap\" that is saved in the\nvirtual machines disk image.\n\nIf you receive this error during backup, there is an inconsistency between the\ncheckpoints that the libvirt daemon thinks exist, and the bitmaps that are\nstored in the disk image.\n\nThis inconsistency can be caused by several situations:\n\n 1) A virtual machine is operated on a cluster and is migrated between host\n    systems (See also: [Transient virtual machines: checkpoint persistency on\n    clusters](#transient-virtual-machines-checkpoint-persistency-on-clusters))\n 2) A change to the libvirt environment between backups (such as re-installing\n    the libvirt daemon) caused the system to lose track of the existing\n    checkpoints, but the bitmaps are still existent in the disk files.\n 3) Between backups, the disk image contents were reset and now the image has\n    already defined bitmaps (if disk was restored from a storage snapshot, for\n    example )\n 4) `virtnbdbackup` is started on an backup target directory with an old state\n    and starts from a wrong checkpoint count, now attempting to create an\n    checkpoint whose bitmap already exists (might happen if you rotate backup\n    directories and pick the wrong target directory with an older state for\n    some reason)\n\nTo troubleshoot this situation, use virsh to list the checkpoints that libvirt\nthinks are existent using:\n\n```\nvirsh checkpoint-list \u003cdomain\u003e\n Name              Creation Time\n ----------------------------------------------\n virtnbdbackup.0   2024-08-01 20:45:44 +0200\n```\n\nYou can also check which disks were included in the checkpoint:\n\n```\n virsh checkpoint-dumpxml vm1 virtnbdbackup.0 | grep \"\u003cdisk.*checkpoint=\"\n    \u003cdisk name='sda' checkpoint='bitmap' bitmap='virtnbdbackup.0'/\u003e\n    \u003cdisk name='sdb' checkpoint='no'/\u003e\n    \u003cdisk name='fda' checkpoint='no'/\u003e\n```\n\nThe example command shows one existing checkpoint for disk \"sda\". An bitmap\nwith the same name should be listed using the `qemu-img` tool for each\ncheckpoint.\n\nNote:\n\u003e Bitmap information is written into the qcow2 metadata only once qemu will\n\u003e close the image. As such you need to turn off the virtual machine prior to\n\u003e checking the bitmaps.\n\nTo list the bitmaps use:\n\n```\nvirsh destroy vm1       # shutdown vm\nvirsh domblklist vm1 | grep sda\n sda      /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2\nqemu-img info /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2\n[..]\n    bitmaps:\n        [0]:\n            flags:\n                [0]: auto\n            name: virtnbdbackup.1\n            granularity: 65536\n[..]\n```\n\nNow, compare which checkpoints are listed and which bitmaps exist in the qcow\nimage. In this example `virsh` only lists the checkpoint \"virtnbdbackup.0\" but\nthe bitmap is called \"virtnbdbackup.1\", indicating there is an inconsistency.\n\nRemove the dangling bitmap(s) via:\n\n```\n  qemu-img bitmap /tmp/tmp.Y2PskFFeVv/vm1-sda.qcow2 --remove virtnbdbackup.1\n```\n\nStart with an new full backup to a fresh directory.\n\n\n## Backup fails with \"Bitmap inconsistency detected: please cleanup checkpoints using virsh and execute new full backup\"\n\nIf an qemu process for a virtual machine is forcefully shutdown after a backup\n(for example due to power outage or qemu process killed/crashed) the bitmaps\nrequired for further backups may not have yet been synced to the qcow image.\n\nIn these cases, you need to delete the existent checkpoints using:\n\n```\n virsh checkpoint-delete \u003cdomain\u003e --checkpointname \u003ccheckpoint_name\u003e --metadata\n```\n\nand start with a fresh full backup.\n\n\n## Backup fails with \"Error during checkpoint removal: [internal error: bitmap 'XX' not found in backing chain of 'XX']\"\n\nIn this situation the checkpoints are still defined in the libvirt ecosystem\nbut the required bitmaps in the qcow files do not exist anymore. This is a\nsituation that is not automatically cleaned up by the backup process because it\nmay require manual intervention and should usually not happen.\n\nYou can cleanup this situation by removing the reported checkpoints via:\n\n```\n virsh checkpoint-delete \u003cdomain\u003e --checkpointname \u003ccheckpoint_name\u003e --metadata\n```\n\n## Backup fails with \"Virtual machine does not support required backup features, please adjust virtual machine configuration.\"\n\nThe libvirt version you are using does by default not expose the functionality\nrequired for creating full or incremental backups. You can either use the\nbackup mode `copy` or enable the backup features as described\n[here](#libvirt-versions--760-debian-bullseye-ubuntu-20x)\n\n## Backup fails with \"Timed out during operation: cannot acquire state change lock\"\n\nIf backups fail with error:\n\n```\nERROR [..] Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainBackupBegin)\n```\n\nthere is still some block jobs operation active on the running domain, for\nexample a live migration or another backup job. It may also happen that\n`virtnbdbackup` crashes abnormally or is forcibly killed during backup\noperation, unable to stop its own backup job.\n\nYou can use option `-k` to forcibly kill any running active block jobs for the\ndomain, but use with care. It is better to check which operation is active with\nthe `virsh domjobinfo` command first.\n\n```\nvirtnbdbackup  -d vm2 -l copy -k  -o -\n[..]\n  INFO virtnbdbackup - main: Stopping domain jobs\n```\n\n## Backup fails with \"Failed to bind socket to /var/tmp/virtnbdbackup.XX: Permission denied\"\n\nThe issue is most likely an active `apparmor` profile that prevents the qemu\ndaemon from creating its socket file for the nbd server. Try to disable\napparmor using the **aa-teardown** command for the current session you are\nexecuting a backup or restore. You can also add the following lines:\n\n```\n/var/tmp/virtnbdbackup.* rw,\n/var/tmp/backup.* rw,\n```\n\nto the configuration files (might not exist by default):\n\n```\n/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper\n/etc/apparmor.d/local/abstractions/libvirt-qemu\n/etc/apparmor.d/local/usr.sbin.libvirtd\n```\n\nor, on newer versions:\n\n```\nsudo mkdir -p /etc/apparmor.d/abstractions/libvirt-qemu.d\ncat \u003c\u003cEOF | sudo tee /etc/apparmor.d/abstractions/libvirt-qemu.d/virtnbdbackup\n/var/tmp/virtnbdbackup.* rw,\n/var/tmp/backup.* rw,\nEOF\nsudo service apparmor reload\n```\n\nSee also: https://github.com/abbbi/virtnbdbackup/issues/7\n\n## High memory usage during backup\n\nlibnbd python implementation has had various memory leaks in older versions\nwhich cause such problems.\n\nFor centos 8 based distributions these fixes have been backported to libnbd\n`1.4.0.`\n\nThe fix itself was released with libnbd 1.5.2, so be sure to use at least this\nversion if using `virtnbdbackup` on any other distribution.\n\nSee also: https://github.com/abbbi/virtnbdbackup/issues/8\n\n## fstrim and (incremental) backup sizes\n\nIf virtual machines have configured disks with discard option and fstrim is\nrunning frequently, trimmed blocks are detected during backup operation by\ndefault.\n\nIf, for example, the fstrim operation frees 30 GiB of disk space, and the\nvirtual machine has only 5 GiB of changed data blocks, 30 GiB of data\nwill be skipped during incremental backup.\n\nThis works by comparing the complete allocation bitmap of the virtual machine\ndisk images during incremental backup.\n\nBehavior can be disabled, if desired, using the `--no-sparse-detection` option.\n\n\n## Test your backups!\n\nThe utility is provided \"as is\", i take no responsibility or warranty if you\nface any issues recovering your data! The only way to ensure your backups are\nvalid and your backup plan works correctly is to repeatedly test the integrity\nby restoring them! If you discover any issues, please do not hesitate to report\nthem.\n\n## Links\n\nBackup howto for Debian Bullseye: https://abbbi.github.io/debian/\n\nShort video: https://youtu.be/dOE0iB-CEGM\n","funding_links":["https://github.com/sponsors/abbbi"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabbbi%2Fvirtnbdbackup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabbbi%2Fvirtnbdbackup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabbbi%2Fvirtnbdbackup/lists"}