{"id":24528168,"url":"https://github.com/deric-w/vinculum","last_synced_at":"2025-03-15T16:43:56.978Z","repository":{"id":259795902,"uuid":"853552823","full_name":"Deric-W/vinculum","owner":"Deric-W","description":"Implementation of Lock-Free Deduplication in Rust","archived":false,"fork":false,"pushed_at":"2025-02-24T17:07:37.000Z","size":915,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T17:28:38.145Z","etag":null,"topics":["backup","deduplication","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deric-W.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-06T22:29:25.000Z","updated_at":"2025-02-24T17:06:53.000Z","dependencies_parsed_at":"2024-10-28T03:13:32.422Z","dependency_job_id":"41b3270d-28f3-4539-a0a8-d2ea6ef1da33","html_url":"https://github.com/Deric-W/vinculum","commit_stats":null,"previous_names":["deric-w/vinculum"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deric-W%2Fvinculum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deric-W%2Fvinculum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deric-W%2Fvinculum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deric-W%2Fvinculum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deric-W","download_url":"https://codeload.github.com/Deric-W/vinculum/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243762238,"owners_count":20343976,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","deduplication","rust"],"created_at":"2025-01-22T06:34:05.940Z","updated_at":"2025-03-15T16:43:56.970Z","avatar_url":"https://github.com/Deric-W.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vinculum\n\n![CI](https://github.com/Deric-W/vinculum/actions/workflows/ci.yml/badge.svg)\n[![codecov](https://codecov.io/gh/Deric-W/vinculum/graph/badge.svg?token=mEPwkHIezV)](https://codecov.io/gh/Deric-W/vinculum)\n\nImplementation of Lock-Free Deduplication in Rust.\n\nThis crates provides an implementation of the algorithm used by tools\nsuch as [Duplicacy](https://duplicacy.com) as described in their\n[paper](https://github.com/gilbertchen/duplicacy/blob/master/duplicacy_paper.pdf).\n\n## Overview\n\nThe most important object is the repository, which stores chunks, clients\nand manifests and is available to a number of clients.\nIt is represented by the `Repository` trait, which is designed to be\nimplemented by you.\n\nClients represent individual users which may perform operations on the\nrepository at the same time as other clients, like creating chunks or\nmanifests.\nA single client can only perform some operations like manifest creation\nsequentially, while multiple clients can perform them in parallel.\nFurthermore, clients have to be registered with the repository before\nthey perform any operations and should be removed when they won't create\nnew manifests for a long time.\n\nManifests represent data uploaded by a client which has been divided\ninto chunks and stored in the repository.\nManifests only store references to their chunks and may share them with\nother manifests, which requires periodic chunk removals after some manifests\nhave been deleted.\n\nChunks are pieces of data uploaded to the repository and may be referenced\nby manifests.\nThey allow for deduplicating manifest contents by associating them with an\nid based on their content, causing chunks with the same content to only be\nstored once in the repository.\n\nChunk creation can be done in parallel (even by a single client), but chunk\nremoval is more complicated because other clients can be in the process of\ncreating manifests referencing them, which would cause invalid references\nshould these chunks be deleted.\nTo prevent this chunks should be first turned into a special type of chunk\ncalled \"fossils\" by using `FossilCollectionBuilder`, which produces a\n`FossilCollection`.\nThis collection can be deleted when every client has created a manifest\nafter the fossil collection was created, which will either permanently\ndelete fossils or turn them back into chunks.\nIt is possible to combine the deletion of a fossil collection with the\ncreation of the next one using `PipelinedFossilCollectionBuilder`.\n\n## Concurrency\n\nWhile this crate does not depend on a particular async runtime it performs\nactions concurrently when possible, which can be controlled by the `concurrency`\nargument of some functions.\n\nThis is achieved by using `futures::stream::FuturesUnordered`, which in\nturn requires that any blocking operations (for example filesystem operations\nor hashing large amounts of data) are queued on an external thread pool should\nparallel execution be desired.\n\n## Features\n\n- `serde`: implements `serde::Serialize` and `serde::Deserialize` for `FossilCollection`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderic-w%2Fvinculum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderic-w%2Fvinculum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderic-w%2Fvinculum/lists"}