{"id":37231731,"url":"https://github.com/vyrti/quichash","last_synced_at":"2026-06-16T15:31:54.375Z","repository":{"id":324814645,"uuid":"1098624296","full_name":"vyrti/quichash","owner":"vyrti","description":"Ultra fast hashing app for Linux, Mac, Windows, Freebsd","archived":false,"fork":false,"pushed_at":"2026-03-27T23:04:54.000Z","size":340,"stargazers_count":18,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-28T05:45:05.852Z","etag":null,"topics":["blake3","cross-platform","deduplication","freebsd","hash","linux","macos","md5","rust","sha2","sha256","sha3","sha512","simd","verification","windows","xxhash","xxhash3"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vyrti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-17T23:53:36.000Z","updated_at":"2026-03-27T23:04:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/vyrti/quichash","commit_stats":null,"previous_names":["vyrti/hash-rs","vyrti/quichash"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/vyrti/quichash","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyrti%2Fquichash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyrti%2Fquichash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyrti%2Fquichash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyrti%2Fquichash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vyrti","download_url":"https://codeload.github.com/vyrti/quichash/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyrti%2Fquichash/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34412786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blake3","cross-platform","deduplication","freebsd","hash","linux","macos","md5","rust","sha2","sha256","sha3","sha512","simd","verification","windows","xxhash","xxhash3"],"created_at":"2026-01-15T03:45:35.771Z","updated_at":"2026-06-16T15:31:54.354Z","avatar_url":"https://github.com/vyrti.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QuicHash\n\nHigh-performance cryptographic hash utility with SIMD optimization.\n\n## Features\n\n- **Algorithms**: MD5, SHA-1, SHA-2/3, BLAKE2/3, xxHash3/128\n- **Defaults**: BLAKE3 algorithm, parallel processing\n- **HDD Mode**: Sequential processing with `--hdd` flag for old mechanical drives\n- **SIMD**: Automatic hardware acceleration (SSE, AVX, AVX2, AVX-512, NEON)\n- **Optional Fast Mode**: Quick hashing for large files (samples 300MB) ONLY for edge cases\n- **Flexible Input**: Files, stdin, or text strings\n- **Wildcard Patterns**: Support for `*`, `?`, and `[...]` patterns in file/directory arguments\n- **Directory Scanning**: Recursive hashing with parallel processing by default\n- **Verification**: Compare hashes against stored database\n- **Database Comparison**: Compare two databases to identify changes, moves, and differences\n- **Database Analysis**: Analyze database statistics, duplicates, and potential space savings\n- **Deduplication**: Find and report duplicate files based on hash comparison\n- **.hashignore**: Exclude files using gitignore patterns\n- **Formats**: Standard, hashdeep, JSON\n- **Compression**: LZMA compression for databases\n- **Cross-Platform**: Linux, macOS, Windows, FreeBSD\n\n## Installation\n\n```bash\ncargo install quichash\n```\n\nOr build from source:\n\n```bash\ncargo build --release\n```\n\n## Quick Start\n\n```bash\n# Hash a file (uses blake3 by default)\nhash myfile.txt\n\n# Hash text\nhash --text \"hello world\"\n\n# Hash from stdin\ncat myfile.txt | hash\n\n# Scan directory (parallel by default)\nhash scan -d ./my_dir -b hashes.db\n\n# Scan on old HDD (sequential)\nhash scan -d ./my_dir -b hashes.db --hdd\n\n# Verify\nhash verify -b hashes.db -d ./my_dir\n\n# Analyze database\nhash analyze -d hashes.db\n\n# List algorithms\nhash list\n```\n\n## Usage\n\n### Hash Files\n\n```bash\nhash myfile.txt                              # Uses blake3 by default\nhash myfile.txt -a sha256                    # Specify algorithm\nhash myfile.txt -a sha256 -a blake3          # Multiple algorithms\nhash largefile.iso -f                        # Fast mode\nhash myfile.txt -b output.txt                # Save to file\nhash myfile.txt --json                       # JSON output\n```\n\n### Wildcard Patterns\n\nHash multiple files using wildcard patterns:\n\n```bash\nhash \"*.txt\" -a sha256                       # All .txt files\nhash \"file?.bin\" -a sha256                   # file1.bin, fileA.bin, etc.\nhash \"[abc]*.jpg\" -a sha256                  # Files starting with a, b, or c\nhash \"img202405*.jpg\" -a sha256              # All images from May 2024\n```\n\nPatterns work with all commands:\n\n```bash\nhash scan -d \"data/*/hashes\" -a sha256 -b output.db    # Multiple directories\nhash verify -b \"*.db\" -d \"data/*\" --json               # Multiple databases/dirs\n```\n\n### Hash Text or Stdin\n\n```bash\nhash --text \"hello world\" -a sha256          # Hash text\ncat myfile.txt | hash -a sha256              # Hash from stdin\n```\n\n### Scan Directory\n\n```bash\nhash scan -d /path/to/dir -b hashes.db                        # Basic (blake3, parallel)\nhash scan -d /path/to/dir -b hashes.db --hdd                  # Sequential for old HDDs\nhash scan -d /path/to/dir -a sha256 -b hashes.db              # Custom algorithm\nhash scan -d /path/to/dir -b hashes.db -f                     # Fast mode\nhash scan -d /path/to/dir -b hashes.db -f --hdd               # Fast mode, sequential\nhash scan -d /path/to/dir -b hashes.db --compress             # Compressed\nhash scan -d /path/to/dir -b hashes.db --format hashdeep      # Hashdeep format\n```\n\n### Verify Directory\n\n```bash\nhash verify -b hashes.db -d /path/to/dir                      # Parallel (default)\nhash verify -b hashes.db -d /path/to/dir --hdd                # Sequential for old HDDs\nhash verify -b hashes.db -d /path/to/dir --json               # JSON output\n```\n\n## Performance Optimizations\n\n### Parallel Verification (Default)\n\nThe verification engine uses parallel processing by default for significantly faster verification:\n\n```bash\n# Parallel verification (default, 2-4x faster)\nhash verify -b hashes.db -d /path/to/dir\n\n# Sequential verification (for old HDDs)\nhash verify -b hashes.db -d /path/to/dir --hdd\n```\n\n**Performance improvements:**\n- **Parallel by default**: Uses all CPU cores via rayon (like scan)\n- **Path canonicalization caching**: Reduces redundant filesystem calls\n- **Optimized file collection**: Efficient recursive directory traversal\n- **Reduced overhead**: Minimizes lock contention in parallel mode\n\n**Parallel mode (default):**\n- SSDs or NVMe drives (no seek penalty)\n- Large numbers of files (\u003e1000)\n- Fast network storage\n- Modern systems with multiple cores\n\n**Sequential mode (--hdd flag):**\n- Old mechanical HDDs (avoid thrashing)\n- Network drives with high latency\n- Systems with limited CPU cores\n- When minimizing system load\n\n```bash\nhash verify -b hashes.db -d /path/to/dir              # Verify\nhash verify -b hashes.db.xz -d /path/to/dir           # Compressed\nhash verify -b hashes.db -d /path/to/dir --json       # JSON\n```\n\nOutput shows: Matches, Mismatches, Missing files, New files\n\n### Compare Databases\n\nCompare two hash databases to identify changes, moves, and differences:\n\n```bash\nhash compare db1.txt db2.txt                          # Compare two databases\nhash compare db1.txt db2.txt -b report.txt            # Save report to file\nhash compare db1.txt db2.txt --format json            # JSON output\nhash compare db1.txt db2.txt --format hashdeep        # Hashdeep audit format\nhash compare db1.txt.xz db2.txt.xz                    # Compare compressed databases\nhash compare db1.txt db2.txt.xz                       # Mix compressed and plain\n```\n\nOutput shows:\n- **Unchanged**: Files with same hash in both databases\n- **Changed**: Files with different hashes\n- **Moved**: Files renamed or moved (same hash, different path)\n- **Removed**: Files in DB1 but not DB2\n- **Added**: Files in DB2 but not DB1\n\n### Analyze Database\n\nAnalyze a hash database to view statistics, duplicates, and potential space savings:\n\n```bash\nhash analyze -d hashes.db                             # Analyze database\nhash analyze -d hashes.db --json                      # JSON output\nhash analyze -d hashes.db -b report.txt               # Save report to file\nhash analyze -d hashes.db.xz                          # Analyze compressed database\n```\n\nOutput shows:\n- **Database info**: Path, format (standard/hashdeep), size\n- **Summary**: Total files, unique hashes, algorithms used\n- **File sizes**: Total size of all files (hashdeep format only)\n- **Duplicates**: Number of duplicate groups, duplicate files, potential space savings\n\n### Deduplicate Files\n\nFind and report duplicate files based on hash comparison:\n\n```bash\nhash dedup -d /path/to/dir                # Find duplicates (dry-run)\nhash dedup -d /path/to/dir -b report.txt  # Save report to file\nhash dedup -d /path/to/dir -f             # Fast mode\nhash dedup -d /path/to/dir --json         # JSON output\n```\n\nOutput shows duplicate groups with file paths and sizes.\n\n### Benchmark \u0026 List\n\n```bash\nhash benchmark                    # Benchmark all algorithms\nhash benchmark -s 500             # Custom data size\nhash list                         # List algorithms\nhash list --json                  # JSON output\n```\n\n## Command-Line Options\n\n| Command | Option | Description |\n|---------|--------|-------------|\n| | `FILE` | File or wildcard pattern to hash (omit for stdin) |\n| | `-t, --text \u003cTEXT\u003e` | Hash text string |\n| | `-a, --algorithm \u003cALG\u003e` | Algorithm (default: blake3) |\n| | `-b, --output \u003cFILE\u003e` | Write to file |\n| | `-f, --fast` | Fast mode (samples 300MB) |\n| | `--json` | JSON output |\n| scan | `-d, --directory \u003cDIR\u003e` | Directory or wildcard pattern to scan |\n| | `-a, --algorithm \u003cALG\u003e` | Algorithm (default: blake3) |\n| | `-b, --database \u003cFILE\u003e` | Output database |\n| | `--hdd` | Sequential mode for old HDDs (default: parallel) |\n| | `-f, --fast` | Fast mode |\n| | `--format \u003cFMT\u003e` | standard or hashdeep |\n| | `--compress` | LZMA compression |\n| | `--json` | JSON output |\n| verify | `-b, --database \u003cFILE\u003e` | Database file or wildcard pattern |\n| | `-d, --directory \u003cDIR\u003e` | Directory or wildcard pattern to verify |\n| | `--json` | JSON output |\n| compare | `DATABASE1` | First database file (supports .xz) |\n| | `DATABASE2` | Second database file (supports .xz) |\n| | `-b, --output \u003cFILE\u003e` | Write report to file |\n| | `--format \u003cFMT\u003e` | plain-text, json, or hashdeep |\n| analyze | `-d, --database \u003cFILE\u003e` | Database file to analyze (supports .xz) |\n| | `-b, --output \u003cFILE\u003e` | Write report to file |\n| | `--json` | JSON output |\n| dedup | `-d, --directory \u003cDIR\u003e` | Directory to scan for duplicates |\n| | `-f, --fast` | Fast mode |\n| | `-b, --output \u003cFILE\u003e` | Write report to file |\n| | `--json` | JSON output |\n| benchmark | `-s, --size \u003cMB\u003e` | Data size (default: 100) |\n| | `--json` | JSON output |\n\n## .hashignore\n\nExclude files using gitignore-style patterns:\n\n```bash\ncat \u003e /path/to/dir/.hashignore \u003c\u003c 'EOF'\n*.log\n*.tmp\nbuild/\nnode_modules/\n!important.log\nEOF\n\nhash scan -d /path/to/dir -a sha256 -b hashes.db\n```\n\nPatterns: `*.ext`, `dir/`, `!pattern`, `#comments`, `**/*.ext`\n\n## Output Formats\n\n**Standard** (default):\n```\n\u003chash\u003e  \u003calgorithm\u003e  \u003cmode\u003e  \u003cfilepath\u003e\n```\n\n**Hashdeep**: CSV format with file size, compatible with hashdeep tool\n\n**JSON**: Structured output for automation\n\n## Performance\n\n| Algorithm | Throughput | Use Case |\n|-----------|-----------|----------|\n| xxHash3 | 10-30 GB/s | Non-crypto, max speed |\n| BLAKE3 | 1-3 GB/s | Crypto, fastest |\n| SHA-512 | 600-900 MB/s | Crypto, 64-bit |\n| SHA-256 | 500-800 MB/s | Crypto, common |\n| SHA3-256 | 200-400 MB/s | Post-quantum |\n\n**Tips:**\n- Parallel processing is enabled by default (2-4x faster on multi-core)\n- Use `--hdd` for old mechanical drives (sequential processing)\n- Use `-f` for large files (10-100x faster)\n- BLAKE3 is the default algorithm (fastest cryptographic hash)\n- Compile with `RUSTFLAGS=\"-C target-cpu=native\"` for best performance\n\n**Fast Mode Speedup:**\n- 1 GB: ~7x faster\n- 10 GB: ~67x faster\n- 100 GB: ~667x faster\n\n## Fast Mode\n\nSamples 300MB (first/middle/last 100MB) instead of entire file.\n\n**Good for:** Quick checks, large files, backups\n**Not for:** Full verification, forensics, small files\n\n## Common Use Cases\n\n```bash\n# Verify downloaded file\nhash downloaded-file.iso -a sha256\n\n# Backup verification (parallel by default)\nhash scan -d /data -b backup.db\nhash verify -b backup.db -d /data\n\n# Backup on old HDD (sequential processing)\nhash scan -d /data -b backup.db --hdd\nhash verify -b backup.db -d /data\n\n# Monitor changes\nhash scan -d /etc/config -b baseline.db\nhash verify -b baseline.db -d /etc/config\n\n# Compare two snapshots\nhash scan -d /data -b snapshot1.db\n# ... time passes ...\nhash scan -d /data -b snapshot2.db\nhash compare snapshot1.db snapshot2.db -b changes.txt\n\n# Analyze database for duplicates and stats\nhash analyze -d media.db                          # View stats and duplicates\nhash analyze -d media.db --json                   # JSON output for automation\n\n# Find duplicates in directory\nhash dedup -d /media                              # Quick duplicate scan\n\n# Forensic analysis\nhash scan -d /evidence -a sha3-256 -b evidence.db\nhash scan -d /evidence -a sha256 -b evidence.txt --format hashdeep\n\n# Quick checksums (blake3 is default)\nhash large-backup.tar.gz -f\nhash scan -d /backups -b checksums.db -f\n\n# Automation\nhash verify -b hashes.db -d /data --json | jq '.report.mismatches'\nhash compare db1.db db2.db --format json | jq '.summary'\n```\n\n## Algorithm Selection\n\n**Recommended:**\n- SHA-256: Widely supported, good security\n- BLAKE3: Fastest cryptographic hash\n- SHA3-256: Post-quantum resistant\n\n**Deprecated:**\n- MD5, SHA-1: Use only for compatibility\n\n**Non-crypto (trusted environments):**\n- xxHash3/128: Maximum speed\n\n## SIMD Optimization\n\nAutomatic support for SSE, AVX, AVX2, AVX-512 (x86_64) and NEON (ARM).\n\nVerify: `cargo test --release --test simd_verification -- --nocapture`\n\n## Wildcard Patterns\n\nSupported patterns:\n- `*` - Matches any number of characters (e.g., `*.txt`, `file*`)\n- `?` - Matches exactly one character (e.g., `file?.bin`)\n- `[...]` - Matches any character in brackets (e.g., `[abc]*.jpg`)\n\n**Examples:**\n```bash\nhash \"*.txt\" -a sha256                       # All .txt files in current dir\nhash \"data/*.bin\" -a sha256                  # All .bin files in data/\nhash \"file?.txt\" -a sha256                   # file1.txt, fileA.txt, etc.\nhash \"[abc]*.jpg\" -a sha256                  # Files starting with a, b, or c\nhash scan -d \"backup/*/data\" -a sha256 -b db.txt  # Multiple directories\nhash verify -b \"*.db\" -d \"data/*\"            # All .db files against all data dirs\n```\n\n**Notes:**\n- Patterns are expanded by the shell or the application\n- If no files match, an error is displayed\n- Multiple matches are processed in sorted order\n- For scan/verify with multiple directories, results are aggregated\n\n## Troubleshooting\n\n| Issue | Solution |\n|-------|----------|\n| Unsupported algorithm | Run `hash list` to see available algorithms |\n| Permission errors | Use `sudo hash scan -d /protected/dir ...` |\n| Slow performance | Use `-p` for parallel, `-f` for fast mode, or BLAKE3 |\n| Fast mode not working | Fast mode only works with files (not stdin/text) |\n| .hashignore not working | Check file location: `/path/to/dir/.hashignore` |\n| Wildcard pattern not matching | Ensure pattern is quoted (e.g., `\"*.txt\"` not `*.txt`) |\n| No files match pattern | Check pattern syntax and file locations |\n\n## Contributing\n\nWe welcome contributions to QuicHash! To contribute, you must certify that you have the right to submit your contribution and agree to license it under the project's dual MIT/Apache-2.0 license.\n\n### Developer Certificate of Origin (DCO)\n\nQuicHash uses the [Developer Certificate of Origin (DCO)](https://developercertificate.org/) process. This is a lightweight way for contributors to certify that they wrote or otherwise have the right to submit code or documentation to an open source project.\n\n#### Inbound = Outbound License\n\nAll contributions to QuicHash are made under the same dual MIT/Apache-2.0 license as the project itself. By signing off on your commits, you agree that your contributions will be licensed under these same terms, with no additional restrictions.\n\n#### How to Sign Off Commits\n\nContributors sign-off that they adhere to these requirements by adding a Signed-off-by line to commit messages.\n\n```\nThis is my commit message\n\nSigned-off-by: Random J Developer \u003crandom@developer.example.org\u003e\n```\n\nGit even has a `-s` command line option to append this automatically to your commit message:\n\n```bash\n$ git commit -s -m 'This is my commit message'\n```\n\n## License\n\nQuicHash is dual-licensed under either:\n\n- **MIT License** ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n- **Apache License, Version 2.0** ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n\nat your option.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyrti%2Fquichash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvyrti%2Fquichash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyrti%2Fquichash/lists"}