{"id":13586990,"url":"https://github.com/dropbox/fast_rsync","last_synced_at":"2025-04-07T18:35:29.187Z","repository":{"id":38281762,"uuid":"243864395","full_name":"dropbox/fast_rsync","owner":"dropbox","description":"An optimized implementation of librsync in pure Rust.","archived":false,"fork":false,"pushed_at":"2024-02-20T21:23:28.000Z","size":98,"stargazers_count":578,"open_issues_count":0,"forks_count":32,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-04-14T06:03:02.933Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dropbox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-02-28T22:18:28.000Z","updated_at":"2024-04-18T01:34:47.581Z","dependencies_parsed_at":"2024-01-07T21:04:49.988Z","dependency_job_id":"e4fe6519-3b43-42b5-bdc6-7265ec29b882","html_url":"https://github.com/dropbox/fast_rsync","commit_stats":{"total_commits":48,"total_committers":4,"mean_commits":12.0,"dds":"0.35416666666666663","last_synced_commit":"36e230d88d523f438d461d99ff1fae2584a335a6"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Ffast_rsync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Ffast_rsync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Ffast_rsync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dropbox%2Ffast_rsync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dropbox","download_url":"https://codeload.github.com/dropbox/fast_rsync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223290335,"owners_count":17120892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:05:57.250Z","updated_at":"2024-11-06T05:31:04.423Z","avatar_url":"https://github.com/dropbox.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# fast\\_rsync\n\n[![Crates.io](https://img.shields.io/crates/v/fast_rsync.svg)](https://crates.io/crates/fast_rsync)\n[![Build Status](https://github.com/dropbox/fast_rsync/workflows/Rust/badge.svg)](https://github.com/dropbox/fast_rsync/actions)\n\n[Documentation](https://docs.rs/fast_rsync)\n\nA faster implementation of [librsync](https://github.com/librsync/librsync) in\npure Rust, using SIMD operations where available. Note that only the legacy MD4\nformat is supported, not BLAKE2.\n\nSIMD is currently supported on x86, x86-64, and aarch64 targets.\n\n## The rsync algorithm\nThis crate offers three major APIs:\n\n1. `Signature::calculate`, which takes a block of data and returns a\n   \"signature\" of that data which is much smaller than the original data.\n2. `diff`, which takes a signature for some block A, and a block of data B, and\n   returns a delta between block A and block B. If A and B are \"similar\", then\n   the delta is usually much smaller than block B.\n3. `apply`, which takes a block A and a delta (as constructed by `diff`), and\n   (usually) returns the block B.\n\nThese functions can be used to implement an protocol for efficiently\ntransferring data over a network. Suppose hosts A and B have similar versions\nof some file `foo`, and host B would like to acquire A's copy.\n1. Host B calculates the `Signature` of `foo_B` and sends it to A. This is\n   cheap because the signature can be 1000X smaller than `foo_B` itself. (The\n   precise factor is configurable and creates a tradeoff between signature size\n   and usefulness. A larger signature enables the creation of smaller and more\n   precise deltas.)\n2. Host A calculates a `diff` from B's signature and `foo_A`, and sends it to\n   `B`.\n3. Host B attempts to `apply` the delta to `foo_B`. The resulting data is\n   _probably_ (\\*) equal to `foo_A`.\n\n(\\*) Note the caveat. `fast_rsync` signatures use the insecure MD4 algorithm.\nTherefore, you should not trust that `diff` will produce a correct delta. You\nmust always verify the integrity of the output of `apply` using some other\nmechanism, such as a cryptographic hash function like SHA-256.\n\n## Benchmarks\nThese were taken on a noisy laptop with a `Intel(R) Core(TM) i7-6820HQ CPU @\n2.70GHz`. The source code is available in `benches/rsync_bench.rs`.\n\n### Signature computation\n```\ncalculate_signature/fast_rsync::Signature::calculate/4194304\n                        time:   [1.0639 ms 1.0696 ms 1.0775 ms]\n                        thrpt:  [3.6253 GiB/s 3.6519 GiB/s 3.6716 GiB/s]\ncalculate_signature/librsync::whole::signature/4194304\n                        time:   [5.8013 ms 5.8521 ms 5.9235 ms]\n                        thrpt:  [675.28 MiB/s 683.51 MiB/s 689.50 MiB/s]\n```\n\n`fast_rsync` is substantially faster than `librsync` at calculating signatures,\nthanks to SIMD optimizations. The benchmark processor has AVX2 and sees a 6X\nspeedup. Processors with only SSE2 (or with less fully-featured AVX) see a\nsmaller speedup, about 3-4X.\n\nNote that `fast_rsync` will detect available vector extensions at runtime and\nuse them as appropriate; `-C target-cpu` is not required.\n\n### Computing deltas\n```\ndiff (64KB edit)/fast_rsync::diff/4194304\n                        time:   [6.8681 ms 7.0596 ms 7.1953 ms]\ndiff (64KB edit)/librsync::whole::delta/4194304\n                        time:   [7.4044 ms 7.4649 ms 7.5222 ms]\n```\n\nWhen comparing similar files, `fast_rsync` is mostly bound by the speed of\nsingle-block MD4 hashing, so it is not much faster than `librsync`.\n\n```\ndiff (random)/fast_rsync::diff/4194304\n                        time:   [37.779 ms 38.317 ms 38.607 ms]\ndiff (random)/librsync::whole::delta/4194304\n                        time:   [41.983 ms 42.758 ms 43.259 ms]\n```\n\nWhen comparing completely different files, `fast_rsync` is mostly bound by the\nspeed of hashmap lookups. Here, `fast_rsync` enjoys a slight advantage because\nof Rust's fast built-in `HashMap` implementation.\n\n```\ndiff (pathological)/fast_rsync::diff/16384\n                        time:   [6.0792 ms 6.2550 ms 6.3666 ms]\ndiff (pathological)/librsync::whole::delta/16384\n                        time:   [50.082 ms 50.185 ms 50.376 ms]\ndiff (pathological)/fast_rsync::diff/4194304\n                        time:   [32.690 ms 32.986 ms 33.171 ms]\n```\n\n`fast_rsync` is able to detect pathological cases that involve many checksum\ncollisions. Note that the 4MB version of the benchmark is prohibitively slow\nfor `librsync` and so its result is not listed.\n\n### Applying deltas\n```\napply/fast_rsync::apply/4194304\n                        time:   [276.17 us 284.20 us 293.37 us]\napply/librsync::whole::patch/4194304\n                        time:   [394.21 us 400.30 us 408.79 us]\n```\n\nApplying deltas is quite straightforward and in any case is unlikely to be a\nbottleneck, but `fast_rsync`'s implementation, which is specialized for\nin-memory buffers, enjoys a mild speedup.\n\n## Contributing\nPull requests are welcome! We ask that you agree to [Dropbox's Contributor\nLicense Agreement](https://opensource.dropbox.com/cla/) for your changes to be\nmerged.\n\n## License\nThis project is licensed under [the Apache-2.0\nlicense](http://www.apache.org/licenses/LICENSE-2.0).\n\nCopyright (c) 2019 Dropbox, Inc.  \nCopyright (c) 2016 bacher09, Artyom Pavlov (RustCrypto/hashes/MD4).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Ffast_rsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdropbox%2Ffast_rsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdropbox%2Ffast_rsync/lists"}