Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dpc/rdedup
Data deduplication engine, supporting optional compression and public key encryption.
https://github.com/dpc/rdedup
backup data-deduplication deduplication encryption
Last synced: about 23 hours ago
JSON representation
Data deduplication engine, supporting optional compression and public key encryption.
- Host: GitHub
- URL: https://github.com/dpc/rdedup
- Owner: dpc
- Created: 2016-03-25T05:34:05.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2022-08-25T22:14:28.000Z (over 2 years ago)
- Last Synced: 2025-01-09T23:38:03.675Z (1 day ago)
- Topics: backup, data-deduplication, deduplication, encryption
- Language: Rust
- Homepage:
- Size: 1010 KB
- Stars: 833
- Watchers: 27
- Forks: 43
- Open Issues: 49
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
- awesome-tools - dpc/rdedup - Data deduplication engine, supporting optional compression and public key encryption. (Backup)
- awesome-repositories - dpc/rdedup - Data deduplication engine, supporting optional compression and public key encryption. (Rust)
README
# rdedup
`rdedup` is a data deduplication engine and a backup software.
See [current project status](https://github.com/dpc/rdedup/wiki) and [original use case description](https://github.com/dpc/rdedup/wiki/My-original-use-case-%28old-README.md%29) wiki pages.
`rdedup` is generally similar to existing software like
`duplicacy`, `restic`, `attic`, `duplicity`, `zbackup`, etc., with a skew towards asymmetric
encryption and synchronization friendly data model.
[Thanks to Rust and solid architecture, rdedup is also exteremely performant](https://github.com/dpc/rdedup/wiki/Rust's-fearless-concurrency-in-rdedup)
and very reliable (no data-loss bugs ever reported).
`rdedup` is written in Rust and provides both command line tool
and library API (`rdedup-lib`). The library can be used to embed the core engine into other applications,
or building custom frontends and tools.
## Features
* simple but solid cryptography:
* libsodium based
* public-key encryption mode (the only tool like that I'm aware of,
and primary reason `rdedup` was created)
* flat-file synchronization friendly (Dropbox/syncthing, rsync, rclone)
* immutable data-conflict-free data store
* cloud backends are WIP
* incremental, scalable garbage collection
* variety of supported algorithms:
* chunking: fastcdc, gear, bup
* hashing: blake2b, sha256
* compression: zstd, deflate, xz2, bzip2, none
* encryption: curve25519, none
* very easy to add new ones
* check `rdedup init --help` output for up-to-date list
* extreme performance and parallelism - see
[Rust fearless concurrency in `rdedup`](https://dpc.pw/rusts-fearless-concurrency-in-rdedup)
* reliability focus (eg. `rdedup` is using `fsync` + `rename`
to avoid data corruption even in case of a hardware crash)
* built-in time/performance profiler
### Strong parts
It's written in Rust. It's a modern language, that is actually really nice
to use. Rust makes it easy to have a very robust and fast software.
The author is a nice person, welcomes contributions, and helps users. Or at
least he's trying... :)
### Shortcomings and missing features:
`rdedup` currently does not implement own backup/restore functionality (own
directory traversal), and because of that it's typically paired with `tar`
or `rdup` tools. Built-in directory traversal could improve deduplication
ratio for workloads with many small, frequently changing files.
Cloud storage integrations are missing. The architecture to support it is
mostly implemented, but the actual backends are not.
### Installation
If you have `cargo` installed:
```norust
cargo install --locked rdedup
```
If not, I highly recommend installing [rustup][rustup] (think `pip`, `npm`
but for Rust)
If you're interested in running `rdedup` with maximum possible performance,
try:
```norust
RUSTFLAGS="-C target-cpu=native" cargo install --locked rdedup
```
[rustup]: https://www.rustup.rs/
In case of troubles, check [rdedup building issues][building-issues] or
report a new one (sorry)!
[building-issues]: http://bit.ly/2ypLPtJ
### Usage
See `rdedup -h` for help.
Rdedup always operates on a *repo*, that you provide as an argument
(eg. `--dir `), or via environment variable (eg. `RDEDUP_DIR`).
Supported commands:
* `rdedup init` - create a new *repo*.
* `rdedup init --help` for repository configuration options.
* `rdedup store ` - store data from standard input under a given
*name*.
* `rdedup load ` - load data stored under given *name* and write it
to standard output.
* `rdedup rm ` - remove the given *name*.
* `rdedup ls` - list all stored names.
* `rdedup gc` - remove any no longer reachable data.
In combination with [rdup][rdup] this can be used to store and restore your
backup like this:
```norust
rdup -x /dev/null "$HOME" | rdedup store home
rdedup load home | rdup-up "$HOME.restored"
```
`rdedup` is data agnostic, so formats like `tar`, `cpio` and other will
work,
but to get benefits of deduplication, archive format should not be
compressed
or encrypted already.
## `RDEDUP_PASSPHRASE` environment variable
If `RDEDUP_PASSPHRASE` is defined, it will be used
instead of interactively asking user for password.
[bup]: https://github.com/bup/bup/
[rdup]: https://github.com/miekg/rdup
[syncthing]: https://syncthing.net
[zbackup]: http://zbackup.org/
[zbackup-issue]: https://github.com/zbackup/zbackup/issues/109
[ddar]: https://github.com/basak/ddar/
[ddar-issue]: https://github.com/basak/ddar/issues/10
# License
rdedup is licensed under: MPL-2.0