https://github.com/cablehead/klump
Chunked blob storage experiment using fjall
https://github.com/cablehead/klump
Last synced: about 2 months ago
JSON representation
Chunked blob storage experiment using fjall
- Host: GitHub
- URL: https://github.com/cablehead/klump
- Owner: cablehead
- Created: 2026-01-21T18:14:06.000Z (7 months ago)
- Default Branch: master
- Last Pushed: 2026-01-22T05:32:14.000Z (7 months ago)
- Last Synced: 2026-06-18T21:31:42.412Z (about 2 months ago)
- Language: Rust
- Homepage:
- Size: 15.6 KB
- Stars: 4
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# klump
Chunked blob storage experiment using [fjall](https://github.com/fjall-rs/fjall).
## Problem
Storing large blobs requires buffering entire content before write. Can't stream, can't know ID until done.
## Approach
- Split incoming bytes into 64KB chunks
- Store chunks by BLAKE3 hash (content-addressed, deduped)
- Blob ID (scru128) assigned immediately, before content fully ingested
- Append-only: each chunk write adds one entry, EOF marker signals completion
Two keyspaces:
- `cas`: `hash (32B) → chunk bytes`
- `blobs`: `blob_id (16B) + seq (4B) → content-type (seq 0)` | `hash (32B)` | `empty (EOF)`
```
blob_id + seq:0 → "text/plain" (content-type)
blob_id + seq:1 → hash (chunk)
blob_id + seq:2 → hash (chunk)
blob_id + seq:3 → (empty) (EOF)
```
Status inferred: last entry empty = Complete, otherwise Ingesting.
Readers can prefix-scan `blob_id` to stream chunks as they arrive.
## Usage
```
echo "hello" | klump put -t text/plain # returns scru128 id
klump get # streams content to stdout
klump list # show all blobs
klump info # show chunks
```
## Why
Exploring streaming writes for [xs](https://github.com/cablehead/xs). Readers could subscribe to a blob ID and process chunks as they arrive, before ingest completes.