{"id":13875957,"url":"https://github.com/nlfiedler/fastcdc-rs","last_synced_at":"2025-04-04T19:14:35.393Z","repository":{"id":34080398,"uuid":"169003024","full_name":"nlfiedler/fastcdc-rs","owner":"nlfiedler","description":"FastCDC implementation in Rust","archived":false,"fork":false,"pushed_at":"2023-10-01T21:58:27.000Z","size":207,"stargazers_count":140,"open_issues_count":4,"forks_count":24,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-16T05:26:30.635Z","etag":null,"topics":["chunking-algorithm","deduplication","rust"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/fastcdc","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nlfiedler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-02-03T22:53:46.000Z","updated_at":"2024-10-13T23:09:06.000Z","dependencies_parsed_at":"2023-10-01T23:38:39.528Z","dependency_job_id":null,"html_url":"https://github.com/nlfiedler/fastcdc-rs","commit_stats":{"total_commits":64,"total_committers":14,"mean_commits":4.571428571428571,"dds":0.3125,"last_synced_commit":"453753f29297e86d6d62ac2bd7e65a7e1d3c07df"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlfiedler%2Ffastcdc-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlfiedler%2Ffastcdc-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlfiedler%2Ffastcdc-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlfiedler%2Ffastcdc-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nlfiedler","download_url":"https://codeload.github.com/nlfiedler/fastcdc-rs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234923,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chunking-algorithm","deduplication","rust"],"created_at":"2024-08-06T06:00:52.708Z","updated_at":"2025-04-04T19:14:35.364Z","avatar_url":"https://github.com/nlfiedler.png","language":"Rust","funding_links":[],"categories":["Rust","rust"],"sub_categories":[],"readme":"# FastCDC [![docs.rs](https://docs.rs/fastcdc/badge.svg)](https://docs.rs/fastcdc) [![Crates.io](https://img.shields.io/crates/v/fastcdc.svg)](https://crates.io/crates/fastcdc) ![Test](https://github.com/nlfiedler/fastcdc-rs/workflows/Test/badge.svg)\n\nThis crate contains multiple implementations of the \"FastCDC\" content defined chunking algorithm originally described in 2016, and later enhanced in 2020, by Wen Xia, et al. A critical aspect of its behavior is that it returns exactly the same results for the same input. To learn more about content defined chunking and its applications, see the reference material linked below.\n\n## Requirements\n\n* [Rust](https://www.rust-lang.org) stable (2018 edition)\n\n## Building and Testing\n\n```shell\n$ cargo clean\n$ cargo build\n$ cargo test\n```\n\n## Example Usage\n\nExamples can be found in the `examples` directory of the source repository, which demonstrate finding chunk boundaries in a given file. There are both streaming and non-streaming examples, where the non-streaming examples use the `memmap2` crate to read large files efficiently.\n\n```shell\n$ cargo run --example v2020 -- --size 16384 test/fixtures/SekienAkashita.jpg\n    Finished dev [unoptimized + debuginfo] target(s) in 0.03s\n     Running `target/debug/examples/v2020 --size 16384 test/fixtures/SekienAkashita.jpg`\nhash=17968276318003433923 offset=0 size=21325\nhash=4098594969649699419 offset=21325 size=17140\nhash=15733367461443853673 offset=38465 size=28084\nhash=4509236223063678303 offset=66549 size=18217\nhash=2504464741100432583 offset=84766 size=24700\n```\n\n### Non-streaming\n\nAn example using `FastCDC` to find chunk boundaries in data loaded into memory:\n\n```rust\nlet contents = std::fs::read(\"test/fixtures/SekienAkashita.jpg\").unwrap();\nlet chunker = fastcdc::v2020::FastCDC::new(\u0026contents, 16384, 32768, 65536);\nfor chunk in chunker {\n    println!(\"offset={} length={}\", chunk.offset, chunk.length);\n}\n```\n\n### Streaming\n\nBoth the `v2016` and `v2020` modules have a streaming version of FastCDC named `StreamCDC`, which takes a `Read` and uses a byte vector with capacity equal to the specified maximum chunk size.\n\n```rust\nlet source = std::fs::File::open(\"test/fixtures/SekienAkashita.jpg\").unwrap();\nlet chunker = fastcdc::v2020::StreamCDC::new(source, 4096, 16384, 65535);\nfor result in chunker {\n    let chunk = result.unwrap();\n    println!(\"offset={} length={}\", chunk.offset, chunk.length);\n}\n```\n\n### Async Streaming\n\nThe `v2020` module has an async streaming version of FastCDC named `AsyncStreamCDC`, which takes an `AsyncRead` (both `tokio` and `futures` are supported via feature flags) and uses a byte vector with capacity equal to the specified maximum chunk size.\n\n```rust\nlet source = std::fs::File::open(\"test/fixtures/SekienAkashita.jpg\").unwrap();\nlet chunker = fastcdc::v2020::AsyncStreamCDC::new(\u0026source, 4096, 16384, 65535);\nlet stream = chunker.as_stream();\nlet chunks = stream.collect::\u003cVec\u003c_\u003e\u003e().await;\n\nfor result in chunks {\n    let chunk = result.unwrap();\n    println!(\"offset={} length={}\", chunk.offset, chunk.length);\n}\n```\n\n## Migration from pre-3.0\n\nIf you were using a release of this crate from before the 3.0 release, you will need to make a small adjustment to continue using the same implementation as before.\n\nBefore the 3.0 release:\n\n```rust\nlet chunker = fastcdc::FastCDC::new(\u0026contents, 8192, 16384, 32768);\n```\n\nAfter the 3.0 release:\n\n```rust\nlet chunker = fastcdc::ronomon::FastCDC::new(\u0026contents, 8192, 16384, 32768);\n```\n\nThe cut points produced will be identical to previous releases as the `ronomon` implementation was never changed in that manner. Note, however, that the other implementations _will_ produce different results.\n\n## Reference Material\n\nThe original algorithm from 2016 is described in [FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication](https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf), while the improved \"rolling two bytes each time\" version from 2020 is detailed in [The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage Systems](https://ieeexplore.ieee.org/document/9055082).\n\n## Other Implementations\n\n* [jrobhoward/quickcdc](https://github.com/jrobhoward/quickcdc)\n    + Similar but slightly earlier algorithm by some of the same authors?\n* [rdedup_cdc at docs.rs](https://docs.rs/crate/rdedup-cdc/0.1.0/source/src/fastcdc.rs)\n    + Alternative implementation in Rust.\n* [ronomon/deduplication](https://github.com/ronomon/deduplication)\n    + C++ and JavaScript implementation of a variation of FastCDC.\n* [titusz/fastcdc-py](https://github.com/titusz/fastcdc-py)\n    + Pure Python port of FastCDC. Compatible with this implementation.\n* [wxiacode/FastCDC-c](https://github.com/wxiacode/FastCDC-c)\n    + Canonical algorithm in C with gear table generation and mask values.\n* [wxiacode/restic-FastCDC](https://github.com/wxiacode/restic-FastCDC)\n    + Alternative implementation in Go with additional mask values.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlfiedler%2Ffastcdc-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnlfiedler%2Ffastcdc-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlfiedler%2Ffastcdc-rs/lists"}