{"id":49889265,"url":"https://github.com/santhsecurity/simdsieve","last_synced_at":"2026-05-15T20:09:12.144Z","repository":{"id":353864429,"uuid":"1205399150","full_name":"santhsecurity/simdsieve","owner":"santhsecurity","description":"SIMD-accelerated byte pattern pre-filtering","archived":false,"fork":false,"pushed_at":"2026-04-25T22:58:51.000Z","size":1273,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-26T00:36:20.081Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://santh.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/santhsecurity.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T23:42:25.000Z","updated_at":"2026-04-25T22:58:55.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/santhsecurity/simdsieve","commit_stats":null,"previous_names":["santhsecurity/simdsieve"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/santhsecurity/simdsieve","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fsimdsieve","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fsimdsieve/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fsimdsieve/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fsimdsieve/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/santhsecurity","download_url":"https://codeload.github.com/santhsecurity/simdsieve/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/santhsecurity%2Fsimdsieve/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33078189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:05:40.333Z","status":"ssl_error","status_checked_at":"2026-05-15T20:05:38.672Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-15T20:09:11.571Z","updated_at":"2026-05-15T20:09:12.132Z","avatar_url":"https://github.com/santhsecurity.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# simdsieve\n\n[![Crates.io](https://img.shields.io/crates/v/simdsieve)](https://crates.io/crates/simdsieve)\n[![Docs.rs](https://docs.rs/simdsieve/badge.svg)](https://docs.rs/simdsieve)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n\nA SIMD-accelerated multi-pattern pre-filtering engine with zero-allocation\nstreaming iteration. Scans byte haystacks for up to 16 patterns simultaneously\nusing the widest SIMD instructions available.\n\n## What It Does\n\n`simdsieve` is a **first-pass candidate filter** for substring search:\n\n1. **Prefix Extraction**: Takes the first 1-4 bytes of each pattern\n2. **SIMD Filtering**: Scans haystack in 32-128 byte blocks, finding positions\n   where prefixes match using vectorized comparison\n3. **Verification**: Confirms full pattern matches before yielding\n4. **Zero False Positives**: Every yielded offset is a verified match start\n\nUse this when you need to quickly filter large data streams for multiple\npatterns before applying more expensive analysis.\n\n## Architecture\n\n| Backend  | Instruction Set      | Block Size | Throughput* | Registers |\n|----------|----------------------|------------|-------------|-----------|\n| AVX-512  | AVX-512F + AVX-512BW | 128 bytes  | \u003e50 GB/s    | 512-bit   |\n| AVX2     | AVX2                 | 64 bytes   | \u003e25 GB/s    | 256-bit   |\n| NEON     | AArch64 NEON         | 64 bytes   | \u003e15 GB/s    | 128-bit   |\n| Scalar   | Portable Rust        | 64 bytes   | \u003e2 GB/s     | 64-bit    |\n\n\\* Single-byte pattern on modern x86_64. Multi-byte patterns scale linearly\nwith prefix length.\n\n### Backend Selection\n\nThe optimal backend is auto-selected at construction time:\n\n1. **AVX-512**: If `avx512f` and `avx512bw` CPU features are detected\n2. **AVX2**: If `avx2` is detected\n3. **NEON**: On `aarch64` targets (guaranteed by the architecture)\n4. **Scalar**: Portable fallback for all other platforms\n\n### Dual-Pump Processing\n\nAVX-512, AVX2, and NEON use \"dual-pump\" processing where each logical block is\nsplit into two halves processed together. This hides load latency by\ninterleaving independent operations across both load ports.\n\n### Design Limits\n\n- **16 patterns maximum**: Unrolled register layout saturates execution ports\n- **4-byte prefix**: First 4 bytes used for SIMD filtering; full pattern for verification\n- **ASCII case-insensitive**: Only `a`-`z` folded; non-ASCII compared verbatim\n\n## Usage\n\nAdd to `Cargo.toml`:\n\n```toml\n[dependencies]\nsimdsieve = \"0.1\"\n```\n\nBasic exact matching:\n\n```rust\nuse simdsieve::SimdSieve;\n\nlet haystack = b\"The quick brown fox jumps over the lazy dog\";\nlet patterns = \u0026[b\"fox\"[..], b\"dog\"[..]];\n\nlet sieve = SimdSieve::new(haystack, patterns).unwrap();\nlet matches: Vec\u003cusize\u003e = sieve.collect();\n\nassert_eq!(matches, vec![16, 40]); // Positions of \"fox\" and \"dog\"\n```\n\nCase-insensitive matching:\n\n```rust\nuse simdsieve::SimdSieve;\n\nlet haystack = b\"Hello World HELLO\";\nlet patterns = \u0026[b\"hello\"[..]];\n\nlet sieve = SimdSieve::new_case_insensitive(haystack, patterns).unwrap();\nlet matches: Vec\u003cusize\u003e = sieve.collect();\n\nassert_eq!(matches, vec![0, 12]); // Matches \"Hello\" and \"HELLO\"\n```\n\nDensity estimation (without full verification):\n\n```rust\nuse simdsieve::SimdSieve;\n\nlet haystack = b\"aaaaaa\";\nlet patterns = \u0026[b\"a\"[..]];\n\nlet count = SimdSieve::estimate_match_count(haystack, patterns, false);\nassert_eq!(count, 6); // One prefix hit at each position\n```\n\n### Iterator Behavior\n\n`SimdSieve` implements `Iterator\u003cItem = usize\u003e` and `FusedIterator`:\n\n- Yields match offsets in ascending order\n- After returning `None`, all subsequent calls return `None`\n- Zero heap allocation during iteration\n- Thread-safe: `Send` and `Sync` (immutable iterator)\n\n## Error Handling\n\nConstruction can fail with [`SimdSieveError`](crate::error::SimdSieveError):\n\n- `EmptyPatternSet`: No patterns provided\n- `PatternLimitExceeded(usize)`: More than 16 patterns provided\n\n```rust\nuse simdsieve::{SimdSieve, SimdSieveError};\n\nlet result = SimdSieve::new(b\"haystack\", \u0026[]);\nassert!(matches!(result, Err(SimdSieveError::EmptyPatternSet)));\n```\n\n## Platform Support\n\n| Platform | Backend | Notes |\n|----------|---------|-------|\n| x86_64   | AVX-512, AVX2, Scalar | Runtime feature detection |\n| aarch64  | NEON, Scalar | NEON guaranteed on AArch64 |\n| wasm32   | Scalar  | No SIMD backend yet |\n| other    | Scalar  | Portable fallback |\n\n## Safety\n\n`simdsieve` contains a small amount of `unsafe` code confined to\nplatform-specific backend modules. Every `unsafe` block has been audited for\nsoundness:\n\n- **SIMD loads** use unaligned load intrinsics (`loadu`), safe for any valid\n  pointer alignment.\n- **Block bounds** are verified before each SIMD call; multi-byte prefixes\n  require `block_len \u003e= block_size + max_prefix_len - 1` trailing bytes.\n- **Target features** are detected at construction; `#[target_feature]` gates\n  prevent executing unsupported instructions.\n- **No uninitialized memory** is read; SIMD struct arrays are zero-initialized.\n- **`unsafe_op_in_unsafe_fn` is forbidden** at the crate level.\n\n## Performance Tuning\n\n### When to Use This Crate\n\n✅ **Good for:**\n- Filtering large streams (\u003e1MB) for multiple patterns\n- Finding candidate positions for regex engines\n- Log analysis with multiple fixed-string patterns\n- Network packet inspection with signature matching\n\n❌ **Not ideal for:**\n- Single small haystacks (\u003c1KB)\n- Patterns longer than 4 bytes that differ only after position 4\n- Pattern sets that share a long common prefix against uniform haystacks\n  (this shifts the bottleneck from SIMD filtering to byte-by-byte verification,\n  resulting in O(n × pattern_count) work)\n- When you need capture groups or regex features\n\n### Prefetching\n\nThe engine automatically issues prefetch hints 512 bytes ahead. This value\n(8 cache lines) balances:\n- Enough lookahead to hide memory latency on modern CPUs\n- Not so much that we evict useful data from cache\n\n### Pattern Selection\n\nFor best performance:\n- Use patterns with distinct first bytes when possible\n- Place more frequent patterns earlier in the pattern list\n- For case-insensitive mode, use lowercase in patterns\n\n## Benchmarks\n\nRun benchmarks with:\n\n```bash\ncargo bench\n```\n\nExample throughput on AMD EPYC 9R14 (AVX-512):\n\n| Haystack | Patterns | Throughput |\n|----------|----------|------------|\n| 1 MB     | 1        | ~55 GB/s   |\n| 1 MB     | 4        | ~45 GB/s   |\n| 1 MB     | 16       | ~30 GB/s   |\n\n## License\n\nLicensed under the MIT License. See [LICENSE](LICENSE) for details.\n\n## Contributing\n\nContributions welcome! Areas of interest:\n\n- WebAssembly SIMD backend\n- Additional benchmarks and fuzz targets\n- New verification strategies\n\nPlease ensure:\n- `cargo test` passes\n- `cargo clippy -- -D warnings` is clean\n- `cargo fmt --check` passes\n- New code has module-level and item-level documentation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthsecurity%2Fsimdsieve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanthsecurity%2Fsimdsieve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanthsecurity%2Fsimdsieve/lists"}