https://github.com/santhsecurity/ziftsieve
Search compressed data without full decompression
https://github.com/santhsecurity/ziftsieve
Last synced: 2 months ago
JSON representation
Search compressed data without full decompression
- Host: GitHub
- URL: https://github.com/santhsecurity/ziftsieve
- Owner: santhsecurity
- License: mit
- Created: 2026-04-08T23:42:28.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-04-25T23:04:26.000Z (3 months ago)
- Last Synced: 2026-04-26T01:12:12.902Z (3 months ago)
- Language: Rust
- Homepage: https://santh.dev
- Size: 17.3 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Audit: AUDIT.md
- Security: SECURITY_AUDIT_REPORT.md
Awesome Lists containing this project
README
# ziftsieve
Search compressed data without full decompression.
[](https://crates.io/crates/ziftsieve)
[](https://docs.rs/ziftsieve)
[](https://opensource.org/licenses/MIT)
## Overview
`ziftsieve` extracts literal bytes from compressed blocks and builds indexes over them. This allows skipping decompression for blocks that provably cannot contain a search pattern.
```
Traditional: SSD → Decompress (100GB/s) → Search (10GB/s) = 9GB/s effective
ziftsieve: SSD → Search compressed (50GB/s) → Decompress 10% = 45GB/s effective
5× faster
```
## Supported Formats
| Format | Algorithm | Literal Extraction | Speed | Status |
|--------|-----------|-------------------|-------|--------|
| LZ4 | LZ77 | ✅ Full | 5 GB/s | Ready |
| Snappy | LZ77 | ✅ Full | 3 GB/s | Ready |
| Zstd | LZ77+ANS | ⚠️ Partial | 1 GB/s | Basic |
| Gzip | LZ77+Huffman | ✅ Native | 1 GB/s | Basic |
## Installation
```toml
[dependencies]
ziftsieve = "0.1"
# Enable specific formats
ziftsieve = { version = "0.1", features = ["lz4", "gzip", "zstd"] }
```
## Usage
```rust
use ziftsieve::{CompressionFormat, CompressedIndex};
// Build index from compressed file
let data = std::fs::read("logs.lz4")?;
let index = CompressedIndex::from_bytes(&data, CompressionFormat::Lz4)?;
// Search - only decompresses blocks that might match
let pattern = b"ERROR";
for block_id in index.candidate_blocks(pattern) {
println!("Potential match in block {}", block_id);
// Now decompress just this block to verify
}
```
## How It Works
LZ-family compressors (LZ4, Snappy, Gzip, Zstd) use two techniques:
1. **Literal bytes** - Copied directly to output
2. **Back-references** - Copy from earlier in the output
`ziftsieve` parses the compressed stream and extracts only the literal bytes. For pattern matching, if your search pattern isn't in the literals, it can't be in the decompressed data (back-references only repeat earlier content).
This means:
- **No false negatives** - If pattern exists, it's found
- **Possible false positives** - Candidate blocks need verification
- **10-100× faster** - Skip decompression for non-matching blocks
## Performance
Benchmarks on AMD Ryzen 9 5950X, 1GB log file:
| Operation | Time | Throughput |
|-----------|------|------------|
| Full LZ4 decompression | 200ms | 5 GB/s |
| Literal extraction | 50ms | 20 GB/s |
| Pattern search | 5ms | - |
| **Effective search** | **55ms** | **18 GB/s** |
## Architecture
```
Compressed Block
│
├──► Literal Bytes ──► Bloom Filter ──► Index
│
└──► Match References ──► (ignored for indexing)
```
## Safety
- `#![forbid(unsafe_code)]` - Pure Rust implementation
- Fuzz tested with arbitrary inputs
- Property-based tested for correctness
## License
MIT License - See [LICENSE](LICENSE) for details.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.