{"id":16630951,"url":"https://github.com/sclevine/xsum","last_synced_at":"2025-09-03T03:37:14.204Z","repository":{"id":79587254,"uuid":"287191251","full_name":"sclevine/xsum","owner":"sclevine","description":"Checksums with Merkle trees and concurrency","archived":false,"fork":false,"pushed_at":"2022-05-02T01:10:48.000Z","size":163,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-12T07:48:11.518Z","etag":null,"topics":["archiving","asn1","asn1-der","audio","checksum","checksums","data-archiving","go","golang","hash","hashing","md5","merkle","merkle-tree","merkletree","pcm","sha256"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sclevine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-13T05:39:50.000Z","updated_at":"2024-11-09T16:24:31.000Z","dependencies_parsed_at":"2023-02-27T19:16:43.561Z","dependency_job_id":null,"html_url":"https://github.com/sclevine/xsum","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sclevine/xsum","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sclevine%2Fxsum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sclevine%2Fxsum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sclevine%2Fxsum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sclevine%2Fxsum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sclevine","download_url":"https://codeload.github.com/sclevine/xsum/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sclevine%2Fxsum/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273386340,"owners_count":25096244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiving","asn1","asn1-der","audio","checksum","checksums","data-archiving","go","golang","hash","hashing","md5","merkle","merkle-tree","merkletree","pcm","sha256"],"created_at":"2024-10-12T04:50:10.267Z","updated_at":"2025-09-03T03:37:14.184Z","avatar_url":"https://github.com/sclevine.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xsum\n\n[![GoDoc](https://pkg.go.dev/badge/github.com/sclevine/ztgrep?status.svg)](https://pkg.go.dev/github.com/sclevine/xsum)\n[![Tests](https://github.com/sclevine/xsum/actions/workflows/go.yml/badge.svg)](https://github.com/sclevine/xsum/actions/workflows/go.yml)\n\n**xsum** is a utility for calculating checksums that supports:\n- [18 cryptographic hash functions](#cryptographic)\n- [12 non-cryptographic hash functions](#non-cryptographic)\n\nThe `xsum` CLI can be used in place of `shasum`, `md5sum`, or similar utilities.\n\n**xsum** differs from existing tools that calculate checksums in that it can:\n- **Calculate a single checksum for an entire directory structure** using [Merkle trees](https://en.wikipedia.org/wiki/Merkle_tree).\n  - Merkle trees allow for concurrency when calculating checksums of directories. (See [Performance](#performance).)\n  - Merkle trees are the same data structure used to reference layers in Docker images.\n- **Calculate checksums that include file attributes** such as type, UID, GID, permissions, etc.\n  - Attributes are serialized deterministically using [DER-encoded ASN.1](https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der). (See [Format](#format).)\n  - Attributes include: file mode, UID, GID, mtime, ctime, xattrs, device ID\n- Execute plugins, including:\n  - [**xsum-pcm**](./cmd/xsum-pcm): calculate checksums of raw PCM inside audio files (e.g., AAC, MP3, FLAC, ALAC)\n    - Checksums remain constant when audio file metadata/tags change, but still protect audio stream.\n    - Install `xsum-pcm` to `$PATH` and use `xsum -a pcm` to invoke.\n    - Requires `ffmpeg`.\n\n## Performance\n\nxsum aims to:\n- Minimize execution time using concurrency\n- Avoid opening more files than available CPUs\n- Provide entirely deterministic output\n- Avoid buffering or delaying output\n\nThis makes xsum ideal for calculating checksums of large directory structures (e.g., for archival purposes).\n\nWith `shasum -a 256`, ~21 seconds:\n```\nlaptop:Library stephen$ time find \"The Beatles/\" -type f -print0|xargs -0 shasum -a 256\n...\n\nreal    0m24.775s\nuser    0m21.250s\nsys     0m2.209s\n```\n\nWith `xsum`, defaulting to sha256, ~3 seconds:\n```\nlaptop:Library stephen$ time find \"The Beatles/\" -type f -print0|xargs -0 xsum\n...\n\nreal    0m2.882s\nuser    0m19.297s\nsys     0m0.971s\n```\n\nChecksum of entire directory structure (including UID/GID/perms), using ASN.1 Merkle tree, ~3 seconds:\n```\nlaptop:Library stephen$ time xsum -f \"The Beatles/\"\nsha256:c1ee0a0a43b56ad834d12aa7187fdb367c9efd5b45dbd96163a9ce27830b5651:7777+ug  The Beatles\n\nreal    0m2.832s\nuser    0m19.328s\nsys     0m0.937s\n```\n(671 files, 4.2 GB total, tested on 2.3 GHz Quad-Core Intel Core i7)\n\n## Usage\n\n```\n$ xsum -h\nUsage:\n  xsum [OPTIONS] [paths...]\n\nGeneral Options:\n  -a, --algorithm=  Use specified hash function (default: sha256)\n  -w, --write=      Write a separate, adjacent file for each checksum\n                    By default, filename will be [orig-name].[alg]\n                    Use -w=ext or -wext to override extension (no space!)\n  -c, --check       Validate checksums\n  -s, --status      With --check, suppress all output\n  -q, --quiet       With --check, suppress passing checksums\n  -v, --version     Show version\n\nMask Options:\n  -m, --mask=       Apply attribute mask as [777]7[+ugx...]:\n                    +u\tInclude UID\n                    +g\tInclude GID\n                    +s\tInclude special file modes\n                    +t\tInclude modified time\n                    +c\tInclude created time\n                    +x\tInclude extended attrs\n                    +i\tInclude top-level metadata\n                    +n\tExclude file names\n                    +e\tExclude data\n                    +l\tAlways follow symlinks\n  -d, --dirs        Directory mode (implies: -m 0000)\n  -p, --portable    Portable mode, exclude names (implies: -m 0000+p)\n  -g, --git         Git mode (implies: -m 0100)\n  -f, --full        Full mode (implies: -m 7777+ug)\n  -x, --extended    Extended mode (implies: -m 7777+ugxs)\n  -e, --everything  Everything mode (implies: -m 7777+ugxsct)\n  -i, --inclusive   Include top-level metadata (enables mask, adds +i)\n  -l, --follow      Follow symlinks (enables mask, adds +l)\n  -o, --opaque      Encode attribute mask to opaque, fixed-length hex (enables mask)\n\nHelp Options:\n  -h, --help        Show this help message\n```\n\n## Format\n\nWhen extended flags are used (e.g., `xsum -d [paths...]`), xsum checksums follow a three-part format:\n```\n[checksum type]:[checksum](:[attribute mask])  [file name]\n```\nFor example:\n```\nsha256:c1ee0a0a43b56ad834d12aa7187fdb367c9efd5b45dbd96163a9ce27830b5651:7777+ug  The Beatles\nsha256:d0ed3ba499d2f79b4b4af9b5a9301918515c35fc99b0e57d88974f1ee74f7820  The Beatles.tar\n```\nThis allows xsum to:\n1. Encode which attributes (e.g., UNIX permission bits) are included in the hash (if applicable).\n2. Specify which hashing algorithm should be used to validate each hash.\n\nThe data format used for extended checksums is specified in [FORMAT.md](FORMAT.md) and may be considered stable.\n\nExtended checksums are portable across operating systems, as long as all requested attributes are supported.\n\n### Top-level Attributes\n\nBy default, xsum only calculates checksums for file/directory **contents**, including when extended mode flags are used. \nThis means that by default, extended checksums only include attributes (e.g., UNIX permissions) for files/directories that are **inside a specified directory**.\n\nUse `-i` to include top-level attributes:\n```\n$ xsum -fi \"The Beatles.tar\" \"The Beatles/\"\nsha256:60f6435e916aae9c4b1a7d4d66011963d80c29744a42c2f0b2171e4c50e90113:7777+ugi  The Beatles.tar\nsha256:7a90cbb0973419f0d3b10a82e53281aa3f0f317ab4ecce10570f26a7404975a1:7777+ugi  The Beatles\n```\n\nWithout `-i`, `xsum` will not append an attribute mask for non-directories, for example:\n```\n$ xsum -f \"The Beatles.tar\" \"The Beatles/\"\nsha256:d0ed3ba499d2f79b4b4af9b5a9301918515c35fc99b0e57d88974f1ee74f7820  The Beatles.tar  # contents only!\nsha256:c1ee0a0a43b56ad834d12aa7187fdb367c9efd5b45dbd96163a9ce27830b5651:7777+ug  The Beatles\n```\n\nAdditionally, without any extended flags, xsum checksums and errors follow the standard output format used by other checksum tools:\n```\n$ xsum \"The Beatles.tar\" \"The Beatles/\"\nd0ed3ba499d2f79b4b4af9b5a9301918515c35fc99b0e57d88974f1ee74f7820  The Beatles.tar\nxsum: The Beatles: is a directory\n```\n\n## Installation\n\n### Homebrew\n\nThe `xsum` CLI and plugins are available via [Homebrew](https://brew.sh):\n\n```\nbrew install sclevine/tap/xsum\nbrew install sclevine/tap/xsum-pcm # optional PCM plugin\n```\n\nInvoke `xsum-pcm` with `xsum -a pcm`.\n\n### Manual\n\nBinaries for macOS, Linux, and Windows are [attached to each release](https://github.com/sclevine/xsum/releases).\n\nTo install `xsum-pcm`, copy the binary to `$PATH`. Invoke it with `xsum -a pcm`.\n\n### Docker\n\n`xsum` is also available as a [Docker image](https://hub.docker.com/r/sclevine/xsum) (includes `xsum-pcm` in `:full`).\n\n## Go Package\n\nxsum may be imported as a Go package.\nSee [godoc](https://pkg.go.dev/github.com/sclevine/xsum) for details.\n\nNOTE: The current Go API should not be considered stable.\n\n## Security Considerations\n\n- xsum only uses hashing algorithms present in Go's standard library and `golang.org/x/crypto` packages.\n- xsum uses a [subset](https://luca.ntop.org/Teaching/Appunti/asn1.html) of [DER-encoded ASN.1](https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der) for deterministic and canonical encoding of all metadata and Merkle Trees.\n- Extended checksums (which include a checksum type and attribute mask) should only be validated with xsum to avoid collision with files that contain xsum's data format directly.\n- Certain (generally non-cryptographic) hash functions supported by xsum may have high collision rates with specific patterns of data.\n  These hash functions may not be appropriate when used to generate checksums of directories.\n  Unless you know what you are doing, choose a strong cryptographic hashing function (like sha256) when calculating checksums of directories.\n\n## Built-in Hash Functions\n\n### Cryptographic\n\n- `md4`\n- `md5`\n- `sha1`\n- `sha256`\n- `sha224`\n- `sha512`\n- `sha384`\n- `sha512-224`\n- `sha512-256`\n- `sha3-224`\n- `sha3-256`\n- `sha3-384`\n- `sha3-512`\n- `blake2s256`\n- `blake2b256`\n- `blake2b384`\n- `blake2b512`\n- `rmd160`\n\n### Non-cryptographic\n\n- `crc32`\n- `crc32c`\n- `crc32k`\n- `crc64iso`\n- `crc64ecma`\n- `adler32`\n- `fnv32`\n- `fnv32a`\n- `fnv64`\n- `fnv64a`\n- `fnv128`\n- `fnv128a`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsclevine%2Fxsum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsclevine%2Fxsum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsclevine%2Fxsum/lists"}