{"id":17238916,"url":"https://github.com/dswd/zvault","last_synced_at":"2025-03-22T18:34:24.251Z","repository":{"id":54407346,"uuid":"94463432","full_name":"dswd/zvault","owner":"dswd","description":"Deduplicating backup solution","archived":false,"fork":false,"pushed_at":"2021-02-20T00:46:45.000Z","size":791,"stargazers_count":59,"open_issues_count":13,"forks_count":7,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-01T17:51:30.868Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dswd.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-15T17:32:55.000Z","updated_at":"2024-10-09T02:47:07.000Z","dependencies_parsed_at":"2022-08-13T14:40:53.142Z","dependency_job_id":null,"html_url":"https://github.com/dswd/zvault","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dswd%2Fzvault","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dswd%2Fzvault/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dswd%2Fzvault/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dswd%2Fzvault/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dswd","download_url":"https://codeload.github.com/dswd/zvault/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245003603,"owners_count":20545637,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T05:47:09.581Z","updated_at":"2025-03-22T18:34:23.897Z","avatar_url":"https://github.com/dswd.png","language":"Rust","funding_links":[],"categories":["Productivity tools"],"sub_categories":["Chess :chess_pawn:"],"readme":"# zVault Backup Solution\n\n[![Build Status](https://travis-ci.org/dswd/zvault.svg?branch=master)](https://travis-ci.org/dswd/zvault)\n[![Coverage Status](https://coveralls.io/repos/dswd/zvault/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/dswd/zvault?branch=master)\n\nzVault is a highly efficient deduplicating backup solution that supports\nclient-side encryption, compression and remote storage of backup data.\n\n## Main Features\n\n### Space efficient storage\nEach file is split into a number of chunks. Content-defined chunking and chunk\nfingerprints make sure that each chunk is only stored once. The chunking\nalgorithm is designed so that small changes to a file only change a few chunks\nand leave most chunks unchanged. Multiple backups of the same data set will only\ntake up the space of one copy.\n\nThe deduplication in zVault is able to reuse existing data no matter whether a\nfile is modified, stored again under a different name, renamed or moved to a\ndifferent folder.\n\nThat makes it possible to store daily backups without much overhead as backups\nwith only small changes do not take up much space.\n\nAlso multiple machines can share the same remote backup location and reuse the\ndata of each others for deduplication.\n\n### Performance\nHigh backup speed is a major design goal of zVault. Therefore is uses different\ntechniques to reach extremely fast backup speeds.\n\nAll used algorithms are hand-selected and optimized for speed.\n\nUnmodified files are detected by comparing them to the last backup which makes\nit possible to skip most of the files in regular usage.\n\nA blazingly fast memory-mapped hash table tracks the fingerprints of all known\nchunks so that chunks that are already in the repository can be skipped quickly.\n\nIn a general use case with a Linux system and a home folder of 50 GiB, backup\nruns usually take between 1 and 2 minutes.\n\n### Independent backups\nAll backups share common data in form of chunks but are independent on a higher\nlevel. Backups can be deleted and chunks that are not used by any backup can be\nremoved.\n\nOther backup solutions use differential backups organized in chains. This makes\nthose backups dependent on previous backups in the chain, so that those backups\ncan not be deleted. Also, restoring chained backups is much less efficient.\n\n### Data encryption\nThe backup data can be protected by modern and fast encryption methods on the\nclient before storing it remotely.\n\n### Compression\nThe backup data can be compressed to save even more space than by deduplication\nalone. Users can choose between zlib (medium speed and compression),\nlz4 (very fast, lower compression), brotli (medium speed, good compression), and\nlzma (quite slow but amazing compression).\n\n### Remote backup storage\nzVault supports off-site backups via mounted filesystems. Backups can be stored\non any remote storage that can be mounted as a filesystem:\n- NFS\n- SMB / Windows shares\n- SSH (via sshfs)\n- FTP (via curlftpfs)\n- Google Drive (via rclone)\n- Amazon S3 (via rclone)\n- Openstack Swift / Rackspace cloud files / Memset Memstore (via rclone)\n- Dropbox (via rclone)\n- Google Cloud Storage (via rclone)\n- Amazon Drive (via rclone)\n- Microsoft OneDrive (via rclone)\n- Hubic (via rclone)\n- Backblaze B2 (via rclone)\n- Yandex Disk (via rclone)\n- ... (potentially many more)\n\n### Backup verification\nFor long-term storage of backups it is important to check backups regularly.\nzVault offers a simple way to verify the integrity of backups.\n\n### Mount backups as filesystems\nBackups can be mounted as a user-space filesystem to investigate and restore\ntheir contents. Once mounted, graphical programs like file managers can be used\nto work on the backup data and find the needed files.\n\n\n## Example scenario\n\nI am using zVault on several of my computers. Here are some numbers from my\ndesktop PC. On this computer I am running daily backups of both the system `/`\n(excluding some folders like `/home`) with 12.9 GiB and the home folder `/home`\nwith 53.6 GiB.\n\n    $\u003e zvault config ::\n    Bundle size: 25.0 MiB\n    Chunker: fastcdc/16\n    Compression: brotli/3\n    Encryption: 8678d...\n    Hash method: blake2\n\nThe backup repository uses the default configuration with encryption enabled.\nThe repository currently contains 12 backup versions of each folder. Both\nfolders combined currently contain over 66.5 GiB not counting changes between\nthe different versions.\n\n    $\u003e zvault info ::\n    Bundles: 1675\n    Total size: 37.9 GiB\n    Uncompressed size: 58.1 GiB\n    Compression ratio: 65.3%\n    Chunk count: 5580237\n    Average chunk size: 10.9 KiB\n    Index: 192.0 MiB, 67% full\n\nThe repository info reveals that the data stored in the repository is only\n58.1 GiB, so 8.4 GiB / 12.5% has been saved by deduplication. Another 20.2 GiB /\n34.7% have been saved by compression. In total, 28.6 out of 66.5 GiB / 43% have\nbeen saved.\n\nThe data is stored in over 5 million chunks of an average size of 10.9 KiB. The\naverage chunk is smaller than configured because of files smaller than the chunk\nsize. The chunks are stored in an index file which takes up 192 MiB on disk and\nin memory during backup runs. Additionally, 337 MiB of bundle data is stored\nlocally to allow fast access to metadata. In total that is less than 1% of the\noriginal data.\n\n    $\u003e zvault info ::home/2017-06-19\n    Date: Mon, 19 Jun 2017 00:00:48 +0200\n    Source: desktop:/home\n    Duration: 0:01:57.2\n    Entries: 193624 files, 40651 dirs\n    Total backup size: 53.6 GiB\n    Modified data size: 2.4 GiB\n    Deduplicated size: 50.8 MiB, 97.9% saved\n    Compressed size: 8.9 MiB in 2 bundles, 82.4% saved\n    Chunk count: 2443, avg size: 21.3 KiB\n\nThis is the information on the last backup run for `/home`. The total data in\nthat backup is 53.6 GiB of which 2.4 GiB have been detected to have changed by\ncomparing file dates and sizes to the last backup. Of those changed files,\ndeduplication reduced the data to 50.8 MiB and compression reduced this to\n8.9 MiB. The whole backup run took less than 2 minutes.\n\n    $\u003e zvault info ::system/2017-06-19\n    Date: Mon, 19 Jun 2017 00:00:01 +0200\n    Source: desktop:/\n    Duration: 0:00:46.5\n    Entries: 435905 files, 56257 dirs\n    Total backup size: 12.9 GiB\n    Modified data size: 43.1 MiB\n    Deduplicated size: 6.8 MiB, 84.2% saved\n    Compressed size: 1.9 MiB in 2 bundles, 72.3% saved\n    Chunk count: 497, avg size: 14.0 KiB\n\nThe information of the last backup run for `/` looks similar. Out of 12.9 GiB,\ndeduplication and compression reduced the new data to 1.9 MiB and the backup\ntook less than one minute.\n\nThis data seems representative as other backup runs and other systems yield\nsimilar results.\n\n\n### Semantic Versioning\nzVault sticks to the semantic versioning scheme. In its current pre-1.0 stage\nthis has the following implications:\n- Even now the repository format is considered pretty stable. All future\n  versions will be able to read the current repository format. Maybe conversions\n  might be necessary but the backups should always be forward-compatible.\n- The CLI might see breaking changes but at least it is guaranteed that calls\n  that are currently non-destructive will not become destructive in the future.\n  Running todays commands on a future version will not cause any harm.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdswd%2Fzvault","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdswd%2Fzvault","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdswd%2Fzvault/lists"}