{"id":31921869,"url":"https://github.com/rmw-lib/blake3_merkle","last_synced_at":"2025-10-13T22:55:17.152Z","repository":{"id":62444599,"uuid":"498566765","full_name":"rmw-lib/blake3_merkle","owner":"rmw-lib","description":null,"archived":false,"fork":false,"pushed_at":"2022-06-07T06:08:47.000Z","size":51,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-19T10:29:47.588Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rmw-lib.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-01T02:37:11.000Z","updated_at":"2023-01-02T12:03:37.000Z","dependencies_parsed_at":"2022-11-01T23:33:36.760Z","dependency_job_id":null,"html_url":"https://github.com/rmw-lib/blake3_merkle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rmw-lib/blake3_merkle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rmw-lib%2Fblake3_merkle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rmw-lib%2Fblake3_merkle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rmw-lib%2Fblake3_merkle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rmw-lib%2Fblake3_merkle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rmw-lib","download_url":"https://codeload.github.com/rmw-lib/blake3_merkle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rmw-lib%2Fblake3_merkle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017087,"owners_count":26085984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-13T22:55:09.712Z","updated_at":"2025-10-13T22:55:17.143Z","avatar_url":"https://github.com/rmw-lib.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- 本文件由 ./make.md 自动生成，请不要直接修改此文件 --\u003e\n\n# blake3_merkle\n\n[→ 中文说明](#cn)\n\n[blake3](https://github.com/BLAKE3-team/BLAKE3) is based on merkle tree, but the exposed interface cannot export merkle tree.\n\n[bao](https://github.com/oconnor663/bao) implements blake3 streaming verification, but cannot resize the underlying [chunks](https://github.com/oconnor663/bao/issues/34) (see [support larger \"chunk groups\" for reduced space overhead](https://github.com/oconnor663/bao/issues/34) ).\n\nThat is, bao consumes 6% extra storage space to record the merkle tree, which is a significant overhead for a distributed content index.\n\nSo, I implemented [blake3_merkle](https://github.com/rmw-lib/blake3_merkle) to export 32 bytes of hash per 1MB of content with an additional storage overhead of only 0.3‱.\n\nThe merkle tree can generate hash values consistent with blake3.\n\nWhen the content is less than or equal to 1MB, the merkle tree has only one node, and the hash of this node is equal to the hash of blake3.\n\n`./examples/main.rs` As follows :\n\n```rust\nuse blake3_merkle::Merkle;\n\nuse std::{env, error::Error, fs::File, io::copy};\n\nfn main() -\u003e Result\u003c(), Box\u003cdyn Error\u003e\u003e {\n  let fpath = env::current_dir()?.join(\"test.pdf\");\n\n  let mut blake3 = blake3::Hasher::new();\n  copy(\u0026mut File::open(\u0026fpath)?, \u0026mut blake3)?;\n\n  let mut merkle = Merkle::new();\n  copy(\u0026mut File::open(\u0026fpath)?, \u0026mut merkle)?;\n  merkle.finalize();\n  dbg!(\u0026merkle.li);\n  dbg!(merkle.blake3());\n  dbg!(blake3.finalize());\n  Ok(())\n}\n```\n\nRun `./example.main.sh` and the output is as follows\n\n```\n[examples/main.rs:14] \u0026merkle.li = [\n    HashDepth {\n        hash: Hash(\n            \"eb896f431b7ff8acb4749b54981d461359a01ded0261fa0da856dd28bf29d3b3\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"4a84cc85f03f47a7c32755f8d9d81c5d3f3e04548ee8129fd480cb71c7dbc5b4\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"fbfe78e550b355cb6775e324c4fed7eb987084b115dca599aaf40056bfb031c3\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"392878c3bdc9c315d6cc8a1721d8cd0a39e49ac8716f4cb8cdf6cf83fbb666f5\",\n        ),\n        depth: 6,\n    },\n]\n[examples/main.rs:15] merkle.blake3() = Hash(\n    \"74a79d0bc37dcac64c493e872252f19e8bdb32dee306481a6827fa037b378c76\",\n)\n[examples/main.rs:16] blake3.finalize() = Hash(\n    \"74a79d0bc37dcac64c493e872252f19e8bdb32dee306481a6827fa037b378c76\",\n)\n```\n\n\u003cb id=cn\u003e\u003c/b\u003e\n\n## 基于 blake3 的 merkle tree\n\n[blake3](https://github.com/BLAKE3-team/BLAKE3) 底层是 merkle tree ，但是暴露的接口无法导出 merkle tree 。\n\n[bao](https://github.com/oconnor663/bao) 实现了 blake3 流式验证，但无法调整底层块大小 (参见 [support larger \"chunk groups\" for reduced space overhead](https://github.com/oconnor663/bao/issues/34) ) 。\n\n也就是说，bao 会消耗 6% 的额外存储空间来记录 merkle tree。对于分布式内容索引来说，这是挺大的开销。\n\n于是，我实现了 [blake3_merkle](https://github.com/rmw-lib/blake3_merkle) ，每 1MB 内容导出 32 字节的哈希，额外存储开销只有 0.3‱  。\n\n通过 merkle tree 可以生成和 blake3 一致的哈希值。\n\n当内容小于等于 1MB 时，merkle tree 只有一个节点，并且这个节点的哈希值等于 blake3 的哈希值。\n\n`./examples/main.rs` 如下 :\n\n```rust\nuse blake3_merkle::Merkle;\n\nuse std::{env, error::Error, fs::File, io::copy};\n\nfn main() -\u003e Result\u003c(), Box\u003cdyn Error\u003e\u003e {\n  let fpath = env::current_dir()?.join(\"test.pdf\");\n\n  let mut blake3 = blake3::Hasher::new();\n  copy(\u0026mut File::open(\u0026fpath)?, \u0026mut blake3)?;\n\n  let mut merkle = Merkle::new();\n  copy(\u0026mut File::open(\u0026fpath)?, \u0026mut merkle)?;\n  merkle.finalize();\n  dbg!(\u0026merkle.li);\n  dbg!(merkle.blake3());\n  dbg!(blake3.finalize());\n  Ok(())\n}\n```\n\n运行 `./example.main.sh`，输出如下\n\n```\n[examples/main.rs:14] \u0026merkle.li = [\n    HashDepth {\n        hash: Hash(\n            \"eb896f431b7ff8acb4749b54981d461359a01ded0261fa0da856dd28bf29d3b3\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"4a84cc85f03f47a7c32755f8d9d81c5d3f3e04548ee8129fd480cb71c7dbc5b4\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"fbfe78e550b355cb6775e324c4fed7eb987084b115dca599aaf40056bfb031c3\",\n        ),\n        depth: 10,\n    },\n    HashDepth {\n        hash: Hash(\n            \"392878c3bdc9c315d6cc8a1721d8cd0a39e49ac8716f4cb8cdf6cf83fbb666f5\",\n        ),\n        depth: 6,\n    },\n]\n[examples/main.rs:15] merkle.blake3() = Hash(\n    \"74a79d0bc37dcac64c493e872252f19e8bdb32dee306481a6827fa037b378c76\",\n)\n[examples/main.rs:16] blake3.finalize() = Hash(\n    \"74a79d0bc37dcac64c493e872252f19e8bdb32dee306481a6827fa037b378c76\",\n)\n```\n\n### 关于\n\n本项目隶属于 **人民网络 ([rmw.link](//rmw.link))** 代码计划。\n\n![人民网络](https://raw.githubusercontent.com/rmw-link/logo/master/rmw.red.bg.svg)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frmw-lib%2Fblake3_merkle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frmw-lib%2Fblake3_merkle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frmw-lib%2Fblake3_merkle/lists"}