{"id":13522289,"url":"https://github.com/dpc/rdedup","last_synced_at":"2025-05-15T08:05:01.430Z","repository":{"id":7016953,"uuid":"54696895","full_name":"dpc/rdedup","owner":"dpc","description":"Data deduplication engine, supporting optional compression and public key encryption.","archived":false,"fork":false,"pushed_at":"2022-08-25T22:14:28.000Z","size":1037,"stargazers_count":837,"open_issues_count":50,"forks_count":45,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-05-12T21:56:59.712Z","etag":null,"topics":["backup","data-deduplication","deduplication","encryption"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-03-25T05:34:05.000Z","updated_at":"2025-05-11T13:16:50.000Z","dependencies_parsed_at":"2022-08-06T20:01:05.839Z","dependency_job_id":null,"html_url":"https://github.com/dpc/rdedup","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frdedup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frdedup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frdedup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpc%2Frdedup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpc","download_url":"https://codeload.github.com/dpc/rdedup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253830913,"owners_count":21971001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","data-deduplication","deduplication","encryption"],"created_at":"2024-08-01T06:00:45.158Z","updated_at":"2025-05-15T08:05:01.376Z","avatar_url":"https://github.com/dpc.png","language":"Rust","funding_links":[],"categories":["Backup","Rust","encryption","others"],"sub_categories":[],"readme":"\u003c!-- README.md is auto-generated from README.tpl with `cargo readme` --\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://travis-ci.org/dpc/rdedup\"\u003e\n      \u003cimg src=\"https://img.shields.io/travis/dpc/rdedup/master.svg?style=flat-square\" alt=\"Travis CI Build Status\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://crates.io/crates/rdedup\"\u003e\n      \u003cimg src=\"http://meritbadge.herokuapp.com/rdedup?style=flat-square\" alt=\"crates.io\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://gitter.im/dpc/rdedup\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/GITTER-join%20chat-green.svg?style=flat-square\" alt=\"Gitter Chat\"\u003e\n  \u003c/a\u003e\n  \u003cbr\u003e\n\u003c/p\u003e\n\n# rdedup\n\n\n\n`rdedup` is a data deduplication engine and a backup software.\nSee [current project status](https://github.com/dpc/rdedup/wiki) and [original use case description](https://github.com/dpc/rdedup/wiki/My-original-use-case-%28old-README.md%29) wiki pages.\n\n`rdedup` is generally similar to existing software like\n `duplicacy`, `restic`, `attic`, `duplicity`, `zbackup`, etc., with a skew towards asymmetric\n encryption and synchronization friendly data model.\n [Thanks to Rust and solid architecture, rdedup is also exteremely performant](https://github.com/dpc/rdedup/wiki/Rust's-fearless-concurrency-in-rdedup)\n and very reliable (no data-loss bugs ever reported).\n\n`rdedup` is written in Rust and provides both command line tool\nand library API (`rdedup-lib`). The library can be used to embed the core engine into other applications,\nor building custom frontends and tools.\n\n ## Features\n\n * simple but solid cryptography:\n   * libsodium based\n   * public-key encryption mode (the only tool like that I'm aware of,\n     and primary reason `rdedup` was created)\n * flat-file synchronization friendly (Dropbox/syncthing, rsync, rclone)\n * immutable data-conflict-free data store\n * cloud backends are WIP\n * incremental, scalable garbage collection\n * variety of supported algorithms:\n   * chunking: fastcdc, gear, bup\n   * hashing: blake2b, sha256\n   * compression: zstd, deflate, xz2, bzip2, none\n   * encryption: curve25519, none\n   * very easy to add new ones\n   * check `rdedup init --help` output for up-to-date list\n * extreme performance and parallelism - see\n   [Rust fearless concurrency in `rdedup`](https://dpc.pw/rusts-fearless-concurrency-in-rdedup)\n * reliability focus (eg. `rdedup` is using `fsync` + `rename`\n   to avoid data corruption even in case of a hardware crash)\n * built-in time/performance profiler\n\n### Strong parts\n\nIt's written in Rust. It's a modern language, that is actually really nice\nto use. Rust makes it easy to have a very robust and fast software.\n\nThe author is a nice person, welcomes contributions, and helps users. Or at\nleast he's trying... :)\n\n### Shortcomings and missing features:\n\n`rdedup` currently does not implement own backup/restore functionality (own\ndirectory traversal), and because of that it's typically paired with `tar`\nor `rdup` tools. Built-in directory traversal could improve deduplication\nratio for workloads with many small, frequently changing files.\n\nCloud storage integrations are missing. The architecture to support it is\nmostly implemented, but the actual backends are not.\n\n### Installation\n\nIf you have `cargo` installed:\n\n```norust\ncargo install --locked rdedup\n```\n\nIf not, I highly recommend installing [rustup][rustup] (think `pip`, `npm`\nbut for Rust)\n\nIf you're interested in running `rdedup` with maximum possible performance,\ntry:\n\n```norust\nRUSTFLAGS=\"-C target-cpu=native\" cargo install --locked rdedup\n```\n\n[rustup]: https://www.rustup.rs/\n\nIn case of troubles, check [rdedup building issues][building-issues] or\nreport a new one (sorry)!\n\n[building-issues]: http://bit.ly/2ypLPtJ\n\n### Usage\n\nSee `rdedup -h` for help.\n\nRdedup always operates on a *repo*, that you provide as an argument\n(eg. `--dir \u003cDIR\u003e`), or via environment variable (eg. `RDEDUP_DIR`).\n\nSupported commands:\n\n* `rdedup init` - create a new *repo*.\n  * `rdedup init --help` for repository configuration options.\n* `rdedup store \u003cname\u003e` - store data from standard input under a given\n  *name*.\n* `rdedup load \u003cname\u003e` - load data stored under given *name* and write it\n  to standard output.\n* `rdedup rm \u003cname\u003e` - remove the given *name*.\n* `rdedup ls` - list all stored names.\n* `rdedup gc` - remove any no longer reachable data.\n\n\nIn combination with [rdup][rdup] this can be used to store and restore your\nbackup like this:\n\n```norust\nrdup -x /dev/null \"$HOME\" | rdedup store home\nrdedup load home | rdup-up \"$HOME.restored\"\n```\n\n`rdedup` is data agnostic, so formats like `tar`, `cpio` and other will\nwork,\nbut to get benefits of deduplication, archive format should not be\ncompressed\nor encrypted already.\n\n## `RDEDUP_PASSPHRASE` environment variable\n\nIf `RDEDUP_PASSPHRASE` is defined, it will be used\ninstead of interactively asking user for password.\n\n[bup]: https://github.com/bup/bup/\n[rdup]: https://github.com/miekg/rdup\n[syncthing]: https://syncthing.net\n[zbackup]: http://zbackup.org/\n[zbackup-issue]: https://github.com/zbackup/zbackup/issues/109\n[ddar]: https://github.com/basak/ddar/\n[ddar-issue]: https://github.com/basak/ddar/issues/10\n\n# License\n\nrdedup is licensed under: MPL-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Frdedup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpc%2Frdedup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpc%2Frdedup/lists"}