https://github.com/darkskygit/assetpack
content-addressed asset packing with precomp compress
https://github.com/darkskygit/assetpack
compression content-addressable-storage content-addressed-storage packing precompress sqlite sqlite-storage
Last synced: 20 days ago
JSON representation
content-addressed asset packing with precomp compress
- Host: GitHub
- URL: https://github.com/darkskygit/assetpack
- Owner: darkskygit
- Created: 2026-05-07T18:10:36.000Z (3 months ago)
- Default Branch: master
- Last Pushed: 2026-05-07T18:25:16.000Z (3 months ago)
- Last Synced: 2026-07-19T02:44:40.636Z (20 days ago)
- Topics: compression, content-addressable-storage, content-addressed-storage, packing, precompress, sqlite, sqlite-storage
- Language: Rust
- Homepage:
- Size: 52.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# assetpack
`assetpack` provides Rust crates for content-addressed asset packing. It focuses on splitting files into stable chunks, deduplicating payloads by SHA3-256, writing SQLite-backed packs with content-addressed objects, and recording recipes that can rebuild the original bytes.
## Crates
- `assetpack-core`: core hashing, chunking, codec, recipe, file-transform, pipeline, and SQLite pack primitives.
- `assetpack-transform-precomp2`: optional file transforms built on `precomp2`, with zstd and lzma wrapping variants.
## Features
- SHA3-256 object identity and recipe integrity checks.
- FastCDC content-defined chunking with stable default bounds.
- Per-chunk raw, zstd, and Brotli storage with automatic fallback when compression is not worthwhile.
- SQLite object storage for chunks and recipes, plus optional namespaced stores for embedding.
- File transform selection with decode verification before accepting transformed output.
- Merkle index rebuilds, integrity checks, membership proofs, and batched proof lookup.
- Optional `precomp2`, `precomp2-zstd`, and `precomp2-lzma` transforms.
## Usage
```rust
use assetpack_core::{ChunkCompressionPolicy, FastCdcSplitter, FileHint, Hash32, Pipeline, PipelineConfig};
fn main() -> assetpack_core::Result<()> {
let data = b"asset payload".to_vec();
let hash = Hash32::sha3_256(&data);
let hint = FileHint {
size: data.len() as u64,
extension: Some("bin".into()),
head: Some(data[..data.len().min(16)].to_vec()),
};
let pipeline = Pipeline::new(PipelineConfig {
splitter: FastCdcSplitter::v1_defaults(),
chunk_compression: ChunkCompressionPolicy::Auto,
discard_payload: false,
});
let plan = pipeline.run(data, &hint, hash, None)?;
assert_eq!(plan.original_hash, hash);
Ok(())
}
```
To enable `precomp2` transforms during selection, pass the transform specs from `assetpack-transform-precomp2`.
```rust
use assetpack_core::{
ChunkCompressionPolicy, FastCdcSplitter, FileHint, FileTransformConfig, Hash32, Pipeline, PipelineConfig, TransformSelector,
};
fn main() -> assetpack_core::Result<()> {
let data = b"asset payload".repeat(8);
let hash = Hash32::sha3_256(&data);
let hint = FileHint {
size: data.len() as u64,
extension: Some("bin".into()),
head: Some(data[..data.len().min(16)].to_vec()),
};
let selector = TransformSelector::new(FileTransformConfig::default(), None, assetpack_transform_precomp2::default_specs());
let pipeline = Pipeline::new(PipelineConfig {
splitter: FastCdcSplitter::v1_defaults(),
chunk_compression: ChunkCompressionPolicy::Auto,
discard_payload: false,
});
let plan = pipeline.run(data, &hint, hash, Some(&selector))?;
assert_eq!(plan.original_hash, hash);
Ok(())
}
```
## Status
These crates are early-stage and the public API and on-disk formats may change before a 1.0 release.
## License
Licensed under the AGPL-3.0-or-later license.