{"id":21514186,"url":"https://github.com/s3git/bt2sum","last_synced_at":"2025-04-09T19:50:46.735Z","repository":{"id":55339270,"uuid":"55218723","full_name":"s3git/bt2sum","owner":"s3git","description":"Utility to compute BLAKE2 Tree hashing in 'unlimited fanout' mode","archived":false,"fork":false,"pushed_at":"2017-01-27T00:26:36.000Z","size":8,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-23T21:45:53.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s3git.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-01T09:08:51.000Z","updated_at":"2021-11-30T16:32:41.000Z","dependencies_parsed_at":"2022-08-14T21:22:54.773Z","dependency_job_id":null,"html_url":"https://github.com/s3git/bt2sum","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3git%2Fbt2sum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3git%2Fbt2sum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3git%2Fbt2sum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3git%2Fbt2sum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s3git","download_url":"https://codeload.github.com/s3git/bt2sum/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103930,"owners_count":21048244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T23:43:25.688Z","updated_at":"2025-04-09T19:50:46.705Z","avatar_url":"https://github.com/s3git.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"BT2SUM\n======\n\nThis utility is derived from [b2sum](https://bitbucket.org/dchest/b2sum) as developed by Dmitry Chestnykh and uses the [SIMD optimized](https://github.com/minio/blake2b-simd) version for the amb64 platform. It is adapted to use the BLAKE2 Tree hashing mode in so called 'unlimited fanout' mode, as it is used in [s3git](https://github.com/s3git/s3git) and described [here](https://github.com/s3git/s3git/blob/master/BLAKE2.md#blake2-tree-modeunlimited-fanout).\n\nIt computes the hashes for the chunks at the leaf level in parallel using by default the number of processors of your computer which gives a nice speed up. Once the hashes for all leaves are available the final hash is computed at level 1.\n\nBLAKE2 Tree mode/Unlimited fanout\n---------------------------------\n\nIn addition to the 'normal' sequential mode that most hashing algorithms use, BLAKE2 has a very flexible tree-hashing mode. Although BLAKE2 supports arbitrary-depth trees, s3git uses a special mode called **unlimited fanout** as shown here:\n\n```\n                  /=====\\\n                  | 1:0 |\n                  \\=====/\n\n/-----\\  /-----\\  /-----\\  /-----\\      /=====\\\n| 0:0 |  | 0:1 |  | 0:2 |  | 0:3 |  ... | 0:N | \n\\-----/  \\-----/  \\-----/  \\-----/      \\=====/\n```\n\nIn this diagram the boxes represent leaves whereby the label `i:j` represents a node's depth `i` and offset `j`. Double-lined nodes (including leaves) are the last nodes of a layer. The leaves process chunks of data of `leaf length` bytes independently of each other, and subsequently the root node hashes the concatenation of the hashes of the leaves.\n\nFor BLAKE2's unlimited fanout mode the depth is always fixed at 2 and there can be as many leaves as are required given the size of the input. Note that the `node offset` and `node depth` parameters ensure that each invocation of BLAKE2 uses a different hash function (and hence will generate a different output for the same input).\n\nBuild from source\n-----------------\n\nTo install from source:\n\n```sh\n$ go get -d github.com/s3git/bt2sum\n$ cd $GOPATH/src/github.com/s3git/bt2sum \n$ go install\n$ bt2sum -h\n```\n\nUsage\n-----\n\n```\nUsage of bt2sum:\n  -a string\n    \tHash algorithm (blake2b, blake2s) (default \"blake2b\")\n  -cpus int\n    \tNumber of CPUs to use. Defaults to number of processors. (default 8)\n  -s int\n    \tDigest size in bytes (0 defaults to max for algorithm)\n  -t string\n    \tChunk size in bytes for tree mode (defaults to 5M). If size ends with a 'k', 'm', or 'g', it is multiplied by 1024 (1K), 1048576 (1M), or 1073741824 (1G) (default \"5m\")\n```\n\nIf no filenames are specified, it reads from standard input.\n\nExamples\n--------\n\n```sh\n$ echo \"hello s3git\" | bt2sum\nBLAKE2b-64 = 18e622875a89cede0d7019b2c8afecf8928c21eac18ec51e38a8e6b829b82c3ef306dec34227929fa77b1c7c329b3d4e50ed9e72dc4dc885be0932d3f28d7053\n$\n$ # Output 40 byte checksum\n$ echo \"hello s3git\" | bt2sum -s 40\nBLAKE2b-40 = 919f330a1b4a3a02aced735e7675905c159b99e07e0c8aa087d0327b26e4d3aa8323bc82962b8e8e\n$\n$ # Compute sum for go installer\n$ bt2sum go1.6.darwin-amd64.pkg\nBLAKE2b-64 (go1.6.darwin-amd64.pkg) = 9be020e41e6fefec6b52b1ae1623a1fdd800c2a5c98d1079c9363107d362fbd558b4e3abb9500ab5f30de9ac708e53ff6b44b1c041edb81cd5df4e29f5dc4e99\n$\n$ # Now use 10 MB chunks\n$ bt2sum -t 10M go1.6.darwin-amd64.pkg\nBLAKE2b-64 (go1.6.darwin-amd64.pkg) = bf05d62548d4aeec8eae124dddefe6572482fe1693a252d01adeb0a3b8cfc308860b7e323c1cf1d14ae67542f146667e009be45313e801a952a8da702ec545a9\n$\n$ # Use blake2s algo\n$ bt2sum -a blake2s go1.6.darwin-amd64.pkg\nBLAKE2s-32 (go1.6.darwin-amd64.pkg) = 75cbcfafa371ed2afb0f4abce06af44a1261376ee071cd35e698f3f590ace529\n```\n\nPerformance\n-----------\n\nWith warm cache of a 1 GB file, the following shows the performance difference between using parallel processing for the leaf chunks versus effectively serializing the hashing of all leaf chunks.\n\n```sh\n$ # Use max number of processors to compute leaf chunks (8 in this case)\n$ time bt2sum 1000mb.bin\nBLAKE2b-64 (1000mb.bin) = e8aa2e154b96cd8948c9f8c360298d0a4352c4c6159bc0b06ef819edcb338fd14d406823973520cded13ddb5c08e59ed7c2c8b09aafe6d78ccfb6fcf6f2ae3c1\n\nreal\t0m0.560s\nuser\t0m1.853s\nsys\t0m0.277s\n$\n$ # Use two processors \n$ time bt2sum -cpus 2 1000mb.bin\nBLAKE2b-64 (1000mb.bin) = e8aa2e154b96cd8948c9f8c360298d0a4352c4c6159bc0b06ef819edcb338fd14d406823973520cded13ddb5c08e59ed7c2c8b09aafe6d78ccfb6fcf6f2ae3c1\n  \nreal\t0m1.060s\nuser\t0m1.432s\nsys\t0m0.246s\n$\n$ # Just use a single processor \n$ time bt2sum -cpus 1 1000mb.bin\nBLAKE2b-64 (1000mb.bin) = e8aa2e154b96cd8948c9f8c360298d0a4352c4c6159bc0b06ef819edcb338fd14d406823973520cded13ddb5c08e59ed7c2c8b09aafe6d78ccfb6fcf6f2ae3c1\n\nreal\t0m1.633s\nuser\t0m1.431s\nsys\t0m0.201s\n```\n\nOverall result is a factor 3x performance improvement.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3git%2Fbt2sum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs3git%2Fbt2sum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3git%2Fbt2sum/lists"}