{"id":13495710,"url":"https://github.com/proxmox/proxmox-backup","last_synced_at":"2026-01-05T20:54:51.427Z","repository":{"id":59893146,"uuid":"447854129","full_name":"proxmox/proxmox-backup","owner":"proxmox","description":"Proxmox Backup Server and Client - read only mirror","archived":false,"fork":false,"pushed_at":"2024-04-22T11:57:46.000Z","size":13472,"stargazers_count":98,"open_issues_count":0,"forks_count":17,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-22T13:00:38.150Z","etag":null,"topics":["backup","content-addressable-storage","proxmox","proxmox-backup","proxmox-ve","rust"],"latest_commit_sha":null,"homepage":"https://pbs.proxmox.com","language":"Rust","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proxmox.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-01-14T05:34:53.000Z","updated_at":"2024-04-23T16:41:14.215Z","dependencies_parsed_at":"2023-10-15T03:27:02.830Z","dependency_job_id":"ca5a9c5f-3c61-4d1a-ab8e-4b76f9f9bbc0","html_url":"https://github.com/proxmox/proxmox-backup","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proxmox%2Fproxmox-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proxmox%2Fproxmox-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proxmox%2Fproxmox-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proxmox%2Fproxmox-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proxmox","download_url":"https://codeload.github.com/proxmox/proxmox-backup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246063189,"owners_count":20717764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","content-addressable-storage","proxmox","proxmox-backup","proxmox-ve","rust"],"created_at":"2024-07-31T19:01:37.405Z","updated_at":"2026-01-05T20:54:51.412Z","avatar_url":"https://github.com/proxmox.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\nBuild \u0026 Release Notes\n*********************\n\n``rustup`` Toolchain\n====================\n\nWe normally want to build with the ``rustc`` Debian package. To do that\nyou can set the following ``rustup`` configuration:\n\n    # rustup toolchain link system /usr\n    # rustup default system\n\n\nVersioning of proxmox helper crates\n===================================\n\nTo use current git master code of the proxmox* helper crates, add::\n\n   git = \"git://git.proxmox.com/git/proxmox\"\n\nor::\n\n   path = \"../proxmox/proxmox\"\n\nto the proxmox dependency, and update the version to reflect the current,\npre-release version number (e.g., \"0.1.1-dev.1\" instead of \"0.1.0\").\n\n\nLocal cargo config\n==================\n\nThis repository ships with a ``.cargo/config`` that replaces the crates.io\nregistry with packaged crates located in ``/usr/share/cargo/registry``.\n\nA similar config is also applied building with dh_cargo. Cargo.lock needs to be\ndeleted when switching between packaged crates and crates.io, since the\nchecksums are not compatible.\n\nTo reference new dependencies (or updated versions) that are not yet packaged,\nthe dependency needs to point directly to a path or git source (e.g., see\nexample for proxmox crate above).\n\n\nBuild\n=====\non Debian 12 Bookworm\n\nSetup:\n  1. # echo 'deb http://download.proxmox.com/debian/devel/ bookworm main' | sudo tee /etc/apt/sources.list.d/proxmox-devel.list\n  2. # sudo wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg\n  3. # sudo apt update\n  4. # sudo apt install devscripts debcargo clang\n  5. # git clone git://git.proxmox.com/git/proxmox-backup.git\n  6. # cd proxmox-backup; sudo mk-build-deps -ir\n\nNote: 2. may be skipped if you already added the PVE or PBS package repository\n\nYou are now able to build using the Makefile or cargo itself, e.g.::\n\n  # make deb\n  # # or for a non-package build\n  # cargo build --all --release\n\nDesign Notes\n************\n\nHere are some random thought about the software design (unless I find a better place).\n\n\nLarge chunk sizes\n=================\n\nIt is important to notice that large chunk sizes are crucial for performance.\nWe have a multi-user system, where different people can do different operations\non a datastore at the same time, and most operation involves reading a series\nof chunks.\n\nSo what is the maximal theoretical speed we can get when reading a series of\nchunks? Reading a chunk sequence need the following steps:\n\n- seek to the first chunk's start location\n- read the chunk data\n- seek to the next chunk's start location\n- read the chunk data\n- ...\n\nLets use the following disk performance metrics:\n\n:AST: Average Seek Time (second)\n:MRS: Maximum sequential Read Speed (bytes/second)\n:ACS: Average Chunk Size (bytes)\n\nThe maximum performance you can get is::\n\n  MAX(ACS) = ACS /(AST + ACS/MRS)\n\nPlease note that chunk data is likely to be sequential arranged on disk, but\nthis it is sort of a best case assumption.\n\nFor a typical rotational disk, we assume the following values::\n\n  AST: 10ms\n  MRS: 170MB/s\n\n  MAX(4MB)  = 115.37 MB/s\n  MAX(1MB)  =  61.85 MB/s;\n  MAX(64KB) =   6.02 MB/s;\n  MAX(4KB)  =   0.39 MB/s;\n  MAX(1KB)  =   0.10 MB/s;\n\nModern SSD are much faster, lets assume the following::\n\n  max IOPS: 20000 =\u003e AST = 0.00005\n  MRS: 500Mb/s\n\n  MAX(4MB)  = 474 MB/s\n  MAX(1MB)  = 465 MB/s;\n  MAX(64KB) = 354 MB/s;\n  MAX(4KB)  =  67 MB/s;\n  MAX(1KB)  =  18 MB/s;\n\n\nAlso, the average chunk directly relates to the number of chunks produced by\na backup::\n\n  CHUNK_COUNT = BACKUP_SIZE / ACS\n\nHere are some staticics from my developer worstation::\n\n  Disk Usage:       65 GB\n  Directories:   58971\n  Files:        726314\n  Files \u003c 64KB: 617541\n\nAs you see, there are really many small files. If we would do file\nlevel deduplication, i.e. generate one chunk per file, we end up with\nmore than 700000 chunks.\n\nInstead, our current algorithm only produce large chunks with an\naverage chunks size of 4MB. With above data, this produce about 15000\nchunks (factor 50 less chunks).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproxmox%2Fproxmox-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproxmox%2Fproxmox-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproxmox%2Fproxmox-backup/lists"}