{"id":13510884,"url":"https://github.com/wamdam/backy2","last_synced_at":"2026-04-06T13:03:07.717Z","repository":{"id":41390162,"uuid":"42891479","full_name":"wamdam/backy2","owner":"wamdam","description":"backy2: Deduplicating block based backup software for ceph/rbd, image files and devices","archived":false,"fork":false,"pushed_at":"2023-08-31T20:54:32.000Z","size":7566,"stargazers_count":198,"open_issues_count":14,"forks_count":39,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-03-30T17:44:52.295Z","etag":null,"topics":["backup","ceph","postgresql","python3","rbd","scrub","snapshot"],"latest_commit_sha":null,"homepage":"http://backy2.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wamdam.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-21T20:18:59.000Z","updated_at":"2025-03-22T23:08:36.000Z","dependencies_parsed_at":"2024-11-01T11:32:11.420Z","dependency_job_id":"162f31a4-013d-4205-96b6-de5f13fcbbc2","html_url":"https://github.com/wamdam/backy2","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/wamdam/backy2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wamdam%2Fbacky2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wamdam%2Fbacky2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wamdam%2Fbacky2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wamdam%2Fbacky2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wamdam","download_url":"https://codeload.github.com/wamdam/backy2/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wamdam%2Fbacky2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31473275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-06T08:36:52.050Z","status":"ssl_error","status_checked_at":"2026-04-06T08:36:51.267Z","response_time":112,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","ceph","postgresql","python3","rbd","scrub","snapshot"],"created_at":"2024-08-01T02:01:58.003Z","updated_at":"2026-04-06T13:03:07.708Z","avatar_url":"https://github.com/wamdam.png","language":"Python","funding_links":[],"categories":["Python","python3"],"sub_categories":[],"readme":"What's the state of this project?\n#################################\n\nbacky2 is an artefact of my hosting company where I recognized that there are\nno sane and reliable backup solutions out there for block-based / snapshot-based\nbackup in the many-terabyte to petabyte range.\n\nbacky2 was the second backup software I wrote for our usecases. The first one\n(you guessed rightm it was called \"backy\") was designed for .img based virtual\nmachines and had designated features for that.\n\nbacky2 was designed around our mostly ceph/rbd-based cluster, that's where the\nmost features come from.\n\nMeanwhile we have switched to local lvm thin pools and we have a very clever,\nfast and new pull-backup solution, for which I (of course) wrote the third\niteration of backup software, however it's neither called backy3, nor it's open\nsource (for now).\n\nThat's why this project receives only very minimal maintenance from me and practically\nno updates. This means you can expect to have installation issues due to\nnon-existing libraries on modern operating system versions. However they should\nbe rather easy to be fixed and in some regard I'm willing to do this. So there's\nno reason to panic if you're using backy2 in 2022 and plan to use it for the\nnext years. The python code will most likely be valid in several years and libs\nhave shown to be very compatible in never versions in the last years.\n\nThe code however is stable (no guarantees though), the software is feature\ncomplete and that's the main reason why there have been no significant commits\nin the last months. We have performend many years of stable backups and restores\nwith it.\n\n\nWhat is backy2?\n###############\n\nbacky2 is a deduplicating block based backup software which encrypts and\ncompresses by default.\n\nThe primary usecases for backy are:\n\n* fast and bandwidth-efficient backup of ceph/rbd virtual machine images to S3\n  or NFS storage\n* backup of LVM volumes (e.g. from personal computers) to external USB disks\n\n\nMain features\n-------------\n\n**Small backups**\n    backy2 deduplicates while reading from the block device and only writes\n    blocks once if they have the same checksum (sha512).\n\n**Compressed backups**\n   backy2 compresses all data blocks with respect to performance with the\n   zstandard library.\n\n**Encrypted backps**\n   All data blocks are encrypted by default. Encryption is managed in integer\n   versions, migration and re-keying procedures exist.\n\n**Fast backups**\n    With the help of ceph's ``rbd diff``, backy2 will only read the changed\n    blocks since the last backup. We have virtual machines with 600GB backed\n    up in about 30 seconds with \u003c70MB/s bandwidth.\n\n**Continuable backups and restores**\n    If the data backend storage is unreliable (as in storage, network, …)\n    and backups or restores can't finish, backy2 can continue them when the\n    outage has ended.\n\n**Small required bandwidth to the backup target**\n    As only changed blocks are written to the backup target, a small (i.e.\n    gbit) connection is sufficient even for larger backups. Even with newly\n    created block devices the traffic to the backup target is small, because\n    these block devices usually are full of \\\\0 and are deduplicated before even\n    reaching the target storage.\n\n**As simple as cp, but as clever as backup needs to be**\n    With a very small set of commands, good ``--help`` and intuitive usage,\n    backy2 feels mostly like ``cp``. And that's intentional, because we think,\n    a restore must be fool-proof and succeed even if you're woken up at 3am\n    and are drunk.\n\n    And it must be hard for you to do stupid things. For example, existing\n    files or rbd volumes will not be overwritten unless you ``--force``,\n    deletion of young backups will fail per default.\n\n**Scrubbing with or without source data against bitrod and other data loss**\n    Every backed up block keeps a checksum with it. When backy scrubs the backup,\n    it reads the block from the backup target storage, calculates it's\n    checksum and compares it to the stored checksum (and size). If the checksum\n    differs, it's most likely that there was an error when storing or reading\n    the block, or by bitrod on the backup target storage.\n\n    Then, the block and the backups it belongs to, are marked 'invalid' and the\n    block will be re-read for the next backup version even if ``rbd diff`` indicates\n    that it hasn't been changed.\n\n    Scrubbing can also take a percentage value for how many blocks of the backup\n    it should scrub. So you can statistically scrub 16% each day and have a\n    full scrub each week (16*7 \u003e 100).\n\n    .. NOTE:: Even invalid backups can be restored!\n\n**Fast restores**\n    With supporting block storage (like ceph/rbd), a sparse restore is\n    possible. This means, sparse blocks (i.e. blocks which \"don't exist\" or are\n    all \\\\0) will be skipped on restore.\n\n**Parallel: backups while scrubbing while restoring**\n    As backy2 is a long-running process, you will of course not want to wait\n    until something has finished. So there are very few places in backy where\n    a global lock will be applied (especially on a very rarely used full\n    cleanup which you can kill at any time to release the lock).\n\n    So you can scrub, backup and restore (multiple times each) on the same\n    machine.\n\n**Does not flood your caches**\n    When reading large pieces of data on linux, often buffers/caches get filled\n    with this data (which in case of backups is essentially only needed once).\n    backy2 instructs linux to immediately forget the data once it's processed.\n\n**Backs up very large volumes RAM- and CPU efficiently**\n    We backup multiple terabytes per vm (and this multiple times per night).\n    backy2 typically runs in \u003c1GB of RAM with these volume sizes. RAM usage\n    depends mostly on simultaneous reads/writes which are configured through\n    ``backy.cfg``.\n    We have seen ~16GB of RAM usage with large configured queues for 200TB\n    images and a backup performance of Ø350MB/s to an external s3 storage.\n\n**backups can be directly mounted**\n    backy2 brings it's own fuse service. So a simple linux command makes\n    backups directly mountable - even on another machine::\n\n        root@backy2:~# backy2 fuse /mnt\n\n   And on another terminal::\n\n        root@backy2:~# ls -la /mnt/by_version\n        drwx------ 0 root root 0 Mai  3 16:14 0c44841a-8d47-11ea-8b2d-3dc6919c2aca\n        drwx------ 0 root root 0 Mai  3 16:14 60ae794e-8d46-11ea-8b2d-3dc6919c2aca\n        drwx------ 0 root root 0 Mai  3 16:14 9d8cfe80-8d46-11ea-8b2d-3dc6919c2aca\n\n        root@backy2:~# ls -la /mnt/by_version_uid/9d8cfe80-8d46-11ea-8b2d-3dc6919c2aca\n        -rw------- 1 root root 280M Mai  3 14:01 data\n        -rw------- 1 root root    0 Mai  3 14:01 expire\n        -rw------- 1 root root    9 Mai  3 14:01 name\n        -rw------- 1 root root    0 Mai  3 14:01 snapshot_name\n        -rw------- 1 root root   51 Mai  3 14:01 tags\n        -rw------- 1 root root    5 Mai  3 14:01 valid\n\n        root@backy2:~# cat /mnt/by_version_uid/9d8cfe80-8d46-11ea-8b2d-3dc6919c2aca/name\n        sometest1\n\n        root@backy2:~# mount /mnt/by_version_uid/9d8cfe80-8d46-11ea-8b2d-3dc6919c2aca/data /mnt\n\n    You get the idea. The data file (and resulting partitions, mounts) read/write!\n    Writing to them will write to a temporary local file. The original backup version\n    is *not* modified!\n    This means, you may even boot a VM from this file from a remote backup.\n\n**Automatic tagging of backup versions**\n    You can tag backups with your own tags depending on your usecase. However,\n    backy2 also tags automatically with these tags::\n\n        b_daily\n        b_weekly\n        b_monthly\n\n    It has a clever algorithm to detect how long the backup for any given image\n    and this tag is ago and then tags again with the given tag. So you'll see\n    a b_weekly every 7 days (if you keep these backups).\n\n**Prevents you from doing something stupid**\n    By providing a config-value for how old backups need to be in order to be\n    able to delete them, you can't accidentially delete very young backups.\n\n    Also, with ``backy protect`` you can protect versions from being deleted.\n    This is very important when you need to restore a version which is suspect\n    to be deleted within the next hours. During restore a lock will prevent\n    deletion, however, by protecting it, it cannot be deleted until you decide\n    that it's not needed anymore.\n\n    Also, you'll need ``--force`` to overwrite existing files or volumes.\n\n**Easy installation**\n    Currently under ubuntu 18.04, you simply install the .deb. Please refer to\n    :ref:`installation` for a detailed install process.\n\n**Free and Open Source Software**\n    Anyone can review the source code and audit security and functionality.\n    backy2 is licensed under the LGPLv3 license (:ref:`license`).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwamdam%2Fbacky2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwamdam%2Fbacky2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwamdam%2Fbacky2/lists"}