https://github.com/mg98/fixed-size-chunker
Efficient implementation of FSC
https://github.com/mg98/fixed-size-chunker
Last synced: 2 months ago
JSON representation
Efficient implementation of FSC
- Host: GitHub
- URL: https://github.com/mg98/fixed-size-chunker
- Owner: mg98
- License: mit
- Created: 2023-02-25T09:34:18.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-02-27T10:27:26.000Z (about 2 years ago)
- Last Synced: 2025-01-20T20:33:29.397Z (4 months ago)
- Language: C
- Size: 8.79 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Fixed Size Chunker (FSC)
Efficiency-first implementation of the FSC algorithm for scientific analyses.
This program will take the target chunk size and a byte stream as arguments
and output for each next chunk an 8-byte fingerprint (using a fast non-cryptographic function) and the chunk's size in CSV format.
This program was done for performance measurements on the algorithm itself and will never actually write anything to disk.
## Usage
```
cat somefile.txt | ./chunker 4096
```
**Example Output:**
```
4a5564c8bcafa54b,4096
34ae13701af441f9,4096
91c63e920e5e525b,4096
0bd92be7339c1112,4096
1f1482633489ddf6,2863
```