{"id":49889301,"url":"https://github.com/santhsecurity/ziftsieve","last_synced_at":"2026-05-15T20:09:16.170Z","repository":{"id":350390994,"uuid":"1205399172","full_name":"santhsecurity/ziftsieve","owner":"santhsecurity","description":"Search compressed data without full decompression","archived":false,"fork":false,"pushed_at":"2026-04-25T23:04:26.000Z","size":18161,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-26T01:12:12.902Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://santh.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/santhsecurity.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":"AUDIT.md","citation":null,"codeowners":null,"security":"SECURITY_AUDIT_REPORT.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T23:42:28.000Z","updated_at":"2026-04-25T23:04:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/santhsecurity/ziftsieve","commit_stats":null,"previous_names":["santhsecurity/ziftsieve"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/santhsecurity/ziftsieve","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fziftsieve","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fziftsieve/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fziftsieve/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fziftsieve/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/santhsecurity","download_url":"https://codeload.github.com/santhsecurity/ziftsieve/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fziftsieve/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33078192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:05:40.333Z","status":"ssl_error","status_checked_at":"2026-05-15T20:05:38.672Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-15T20:09:15.285Z","updated_at":"2026-05-15T20:09:16.146Z","avatar_url":"https://github.com/santhsecurity.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ziftsieve\n\nSearch compressed data without full decompression.\n\n[![Crates.io](https://img.shields.io/crates/v/ziftsieve)](https://crates.io/crates/ziftsieve)\n[![Docs.rs](https://docs.rs/ziftsieve/badge.svg)](https://docs.rs/ziftsieve)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## Overview\n\n`ziftsieve` extracts literal bytes from compressed blocks and builds indexes over them. This allows skipping decompression for blocks that provably cannot contain a search pattern.\n\n```\nTraditional:  SSD → Decompress (100GB/s) → Search (10GB/s) = 9GB/s effective\nziftsieve:    SSD → Search compressed (50GB/s) → Decompress 10% = 45GB/s effective\n                                                         \n                                              5× faster\n```\n\n## Supported Formats\n\n| Format | Algorithm | Literal Extraction | Speed | Status |\n|--------|-----------|-------------------|-------|--------|\n| LZ4    | LZ77      | ✅ Full            | 5 GB/s | Ready |\n| Snappy | LZ77      | ✅ Full            | 3 GB/s | Ready |\n| Zstd   | LZ77+ANS  | ⚠️ Partial         | 1 GB/s | Basic |\n| Gzip   | LZ77+Huffman | ✅ Native         | 1 GB/s | Basic |\n\n## Installation\n\n```toml\n[dependencies]\nziftsieve = \"0.1\"\n\n# Enable specific formats\nziftsieve = { version = \"0.1\", features = [\"lz4\", \"gzip\", \"zstd\"] }\n```\n\n## Usage\n\n```rust\nuse ziftsieve::{CompressionFormat, CompressedIndex};\n\n// Build index from compressed file\nlet data = std::fs::read(\"logs.lz4\")?;\nlet index = CompressedIndex::from_bytes(\u0026data, CompressionFormat::Lz4)?;\n\n// Search - only decompresses blocks that might match\nlet pattern = b\"ERROR\";\nfor block_id in index.candidate_blocks(pattern) {\n    println!(\"Potential match in block {}\", block_id);\n    // Now decompress just this block to verify\n}\n```\n\n## How It Works\n\nLZ-family compressors (LZ4, Snappy, Gzip, Zstd) use two techniques:\n\n1. **Literal bytes** - Copied directly to output\n2. **Back-references** - Copy from earlier in the output\n\n`ziftsieve` parses the compressed stream and extracts only the literal bytes. For pattern matching, if your search pattern isn't in the literals, it can't be in the decompressed data (back-references only repeat earlier content).\n\nThis means:\n- **No false negatives** - If pattern exists, it's found\n- **Possible false positives** - Candidate blocks need verification\n- **10-100× faster** - Skip decompression for non-matching blocks\n\n## Performance\n\nBenchmarks on AMD Ryzen 9 5950X, 1GB log file:\n\n| Operation | Time | Throughput |\n|-----------|------|------------|\n| Full LZ4 decompression | 200ms | 5 GB/s |\n| Literal extraction | 50ms | 20 GB/s |\n| Pattern search | 5ms | - |\n| **Effective search** | **55ms** | **18 GB/s** |\n\n## Architecture\n\n```\nCompressed Block\n    │\n    ├──► Literal Bytes ──► Bloom Filter ──► Index\n    │\n    └──► Match References ──► (ignored for indexing)\n```\n\n## Safety\n\n- `#![forbid(unsafe_code)]` - Pure Rust implementation\n- Fuzz tested with arbitrary inputs\n- Property-based tested for correctness\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthsecurity%2Fziftsieve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanthsecurity%2Fziftsieve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthsecurity%2Fziftsieve/lists"}