{"id":50331785,"url":"https://github.com/rupurt/sift","last_synced_at":"2026-05-29T10:01:45.326Z","repository":{"id":343045091,"uuid":"1176016798","full_name":"rupurt/sift","owner":"rupurt","description":"Hybrid and agentic search for coding workflows. BM25, vector retrieval, reranking, and multi-turn search in a lightweight Rust CLI and library","archived":false,"fork":false,"pushed_at":"2026-04-12T05:15:01.000Z","size":4630,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-12T07:11:44.549Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.alexdk.com/blog/introducing-sift","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rupurt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-08T14:00:49.000Z","updated_at":"2026-04-12T05:15:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/rupurt/sift","commit_stats":null,"previous_names":["rupurt/sift"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rupurt/sift","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupurt%2Fsift","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupurt%2Fsift/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupurt%2Fsift/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupurt%2Fsift/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rupurt","download_url":"https://codeload.github.com/rupurt/sift/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupurt%2Fsift/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33646428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T10:01:44.609Z","updated_at":"2026-05-29T10:01:45.314Z","avatar_url":"https://github.com/rupurt.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sift\n\n[![CI](https://github.com/rupurt/sift/actions/workflows/ci.yml/badge.svg)](https://github.com/rupurt/sift/actions/workflows/ci.yml)\n[![Planning Board](https://img.shields.io/badge/Keel-Board-blue)](.keel/README.md)\n[![Release Process](https://img.shields.io/badge/Release-Process-green)](RELEASE.md)\n\n`sift` is a standalone Rust CLI and library for hybrid and agentic local search\nin development workflows. The shipped executable gives you direct search,\nplanner-driven `search --agent`, evaluation, dataset, and prompt-optimization\ncommands; the crate-root library facade exposes direct, controller, and\nautonomous search APIs for embedding.\n\nThe core idea is simple: point `sift` at a directory, extract text on demand,\nand run a layered pipeline of expansion, retrieval, fusion, and reranking.\nAgentic behavior in Sift is built on top of that same retrieval substrate\nrather than replacing it with a separate service or daemon.\n\nFor project background and design rationale, read the introductory post:\n[`Sift: Local Hybrid Search Without the Infrastructure Tax`](https://www.alexdk.com/blog/introducing-sift).\n\n## Current Contract\n\n- **Single Rust binary:** No external database, daemon, or long-running service.\n- **Local-first retrieval:** Search runs over local corpora with transparent caching in standard user cache directories.\n- **Shared runtime cache:** Direct search, `search --agent`, and the crate-root controller/autonomous APIs reuse the same cache root across fresh processes.\n- **Visible indexing:** Interactive `sift search` runs show live stderr progress for indexing and cache reuse while preparation is in flight.\n- **Default interactive strategy:** The default config strategy is `hybrid`.\n- **Current champion preset:** `page-index-hybrid` is the richer benchmark preset, combining BM25, phrase, path-fuzzy, segment-fuzzy, and vector retrieval before structural reranking.\n- **Layered pipeline:** Query Expansion -\u003e Retrieval -\u003e Fusion -\u003e Reranking.\n- **Executable surface:** `search`, `eval`, `dataset`, `optimize`, and `config` are the supported CLI commands.\n- **Library surface:** `search`, `search_with_progress`, `assemble_context`, `search_turn`, `search_controller`, and `search_autonomous` are supported at the crate root.\n- **Autonomous planning:** Bounded linear and graph autonomous planning ship through `sift search --agent` and `Sift::search_autonomous`, with heuristic and model-driven planner strategies.\n- **Emission modes:** Turn-aware library calls can emit `view`, `protocol`, or `latent` responses.\n- **Supported inputs:** Text, HTML, PDF, and OOXML files (`.docx`, `.xlsx`, `.pptx`).\n\n## Installation\n\n### Homebrew (macOS and Linux)\n\n```bash\nbrew tap rupurt/homebrew-tap\nbrew install sift\n```\n\n### One-liner Install (macOS and Linux)\n\n```bash\ncurl --proto '=https' --tlsv1.2 -LsSf https://github.com/rupurt/sift/releases/latest/download/sift-installer.sh | sh\n```\n\n### Manual Download\n\nDownload the latest pre-built binaries and installers for your platform from the [GitHub Releases](https://github.com/rupurt/sift/releases) page. We provide:\n- **Linux:** `.tar.gz` archives plus the cross-platform shell installer\n- **macOS:** `.tar.gz` archives plus the cross-platform shell installer\n- **Windows:** `.zip` archives, `.msi`, and the PowerShell installer\n\n## Development Shells\n\nThe default Nix shell and package outputs are CPU-first:\n\n```bash\nnix develop\nnix build .#sift\n```\n\nCUDA is available as an explicit opt-in for local GPU work:\n\n```bash\nnix develop .#cuda\nnix build .#sift-cuda\njust sift --cuda search . \"query\"\n```\n\nThis keeps CI and downstream consumers from pulling the CUDA toolchain unless\nthey intentionally choose the GPU lane.\n\n## CLI Interface\n\nThe executable currently exposes the following command groups:\n\n| Command | Purpose |\n|---------|---------|\n| `sift search [OPTIONS] [PATH] \u003cQUERY\u003e` | Direct single-turn search over a local corpus. |\n| `sift search [OPTIONS] [PATH] --agent \u003cROOT_TASK\u003e` | Planner-driven search over a local corpus using the shared autonomous runtime, with optional `--agent-mode linear|graph`, `--planner-strategy`, and `--planner-profile` selection. |\n| `sift eval all` | Compare all registered retrieval strategies. |\n| `sift eval quality` | Emit a JSON quality report for one strategy, optionally against a baseline. |\n| `sift eval latency` | Emit a JSON latency report for one strategy. |\n| `sift eval agentic` | Benchmark linear autonomous, graph autonomous, planned controller, and collapsed single-turn baselines. |\n| `sift dataset download` / `sift dataset materialize` | Manage evaluation datasets such as SciFact. |\n| `sift optimize` | Tune prompt templates used by generative expansion. |\n| `sift config` | Print the merged effective configuration. |\n\nThere is not a separate `sift agentic` interactive shell. Agent-mode search\nships through `sift search --agent`, while evaluation-oriented benchmarking\ncontinues to live under `eval agentic`.\n\n## Search Examples\n\nSearch with the default configured strategy (`hybrid` unless overridden in\n`sift.toml`):\n\n```bash\nsift search tests/fixtures/rich-docs \"architecture decision\"\n```\n\nRun the current benchmark champion preset explicitly:\n\n```bash\nsift search --strategy page-index-hybrid tests/fixtures/rich-docs \"architecture decision\"\n```\n\nUse the focused path-heavy preset when filename or path intent dominates:\n\n```bash\nsift search --strategy path-hybrid src \"sift_request_factory\"\n```\n\nUse a simpler lexical-only plan:\n\n```bash\nsift search --strategy bm25 \"service catalog\"\n```\n\nUse a vector-only plan:\n\n```bash\nsift search --strategy vector \"architecture\"\n```\n\nRun the supported autonomous planner runtime from the CLI:\n\n```bash\nsift search --strategy bm25 tests/fixtures/rich-docs --agent \"find the cache invalidation path\"\n```\n\nSwitch the shared autonomous runtime into bounded graph mode:\n\n```bash\nsift search --strategy bm25 tests/fixtures/rich-docs \\\n  --agent \"find the cache invalidation path\" --agent-mode graph\n```\n\nSelect the built-in model-driven planner explicitly:\n\n```bash\nsift search --strategy hybrid --agent \"trace the cache invalidation path\" \\\n  --planner-strategy model-driven --planner-profile local-planner-v1\n```\n\nOverride individual pipeline stages:\n\n```bash\nsift search --retrievers bm25,path-fuzzy,segment-fuzzy --reranking position-aware \"query\"\n```\n\nEmit JSON instead of text:\n\n```bash\nsift search --json \"query\"\n```\n\nInteractive text-mode `sift search` writes transient progress to stderr while\nthe corpus is being prepared. That progress includes file counts plus cache and\nBM25 reuse/build metrics so the first run does not look hung. The same cache\nroot is reused by direct and agent-mode runs, and bounded dirty-sector rebuilds\npreserve reuse for unchanged sectors after fresh restarts.\n\n### Verbose Mode\n\nTrace the pipeline and timings at different levels:\n\n- `-v`: Phase timings\n- `-vv`: Detailed retriever timings and cache traces\n- `-vvv`: Granular internal scoring data\n\n## Strategy Guide\n\nUse the shipped presets intentionally:\n\n- `hybrid`: Lowest-friction default when you want BM25 plus semantic recall without structural reranking.\n- `path-hybrid`: Best when the query is shaped like a filename, path fragment, module, or symbol stem and you want deterministic path-sensitive reranking.\n- `page-index-hybrid`: Best all-around direct-search preset when you want stronger recall plus snippet-bearing structural evidence for downstream synthesis or context assembly.\n- `page-index-llm` / `page-index-jina` / `page-index-gemma`: Best when the richer shortlist is already useful and you want a heavier semantic pass on top of it.\n\nThe structural fuzzy lanes are intentionally narrow:\n\n- `path-fuzzy` recovers artifacts from approximate filename and path-component intent.\n- `segment-fuzzy` recovers typo-tolerant line and segment evidence, which is especially useful when a downstream tool needs snippet-bearing support rather than just an artifact path.\n\n## Embedded Library\n\n`sift` can also be embedded from another Rust project. The supported public\nsurface lives at the crate root and includes:\n\n- `Sift`, `SiftBuilder`\n- `SearchInput`, `SearchOptions`\n- `ContextAssemblyRequest`, `ContextAssemblyResponse`\n- `SearchTurnRequest`, `SearchTurnResponse`\n- `SearchControllerRequest`, `SearchControllerResponse`\n- `AutonomousSearchRequest`, `AutonomousSearchResponse`\n- `AutonomousSearchMode`, `AutonomousPlannerState`, `AutonomousPlannerStrategy`, `AutonomousPlannerStrategyKind`\n- `AutonomousPlannerTrace`, `AutonomousPlannerDecision`, `AutonomousPlannerStopReason`\n- `AutonomousPlanner`, `HeuristicAutonomousPlanner`, `ModelDrivenAutonomousPlanner`\n- `AutonomousGraphEpisodeState`, `AutonomousGraphNode`, `AutonomousGraphEdge`, `AutonomousGraphFrontierEntry`, `replay_graph_trace`\n- `SearchEmission`, `SearchEmissionMode`\n- `SearchPlan`, `QueryExpansionPolicy`, `RetrieverPolicy`, `FusionPolicy`, `RerankingPolicy`\n- `Retriever`, `Fusion`, `Reranking`\n- `SearchResponse`, `SearchHit`, `ContextArtifact`, `ContextArtifactKind`, `ScoreConfidence`\n- `SearchProgress`, `SearchTelemetry`\n- `ModelSource`, `ModelRuntimeContract`, `PreparedModel`, `ModelArtifactFormat`, `ModelPreparationMode`, `prepare_model`\n\nEverything under `sift::internal` exists to support the bundled executable,\nbenchmarks, and repository-internal tests. It is not part of the stable\nembedding contract.\n\nEmbedders that need stronger filename/path recall and synthesis-ready snippets\ncan opt into `SearchPlan::default_page_index_hybrid()` through the library\nrequest types. That keeps planner ownership outside `sift`, which matches how\ndownstream tools such as `paddles` should consume the direct retrieval layer.\n\nIf an embedder only needs filename/path-heavy recovery, `SearchPlan::default_path_hybrid()`\nis the lighter structural option. The intended `paddles` adoption path is still\nexplicit direct retrieval selection, not handing search planning back into\n`sift`.\n\nWhen you point `Sift::builder()` or `SearchOptions` at a cache directory, direct\nsearch, controller execution, and autonomous runtime calls all reuse that same\nsector-aware cache substrate. Fresh processes can mount clean sectors prepared\nby another surface and only rebuild sectors touched by corpus changes.\n\nSee [LIBRARY.md](LIBRARY.md) for the full embedding guide, including direct\nsearch, context assembly, deterministic controller execution, supported\nautonomous search, protocol/latent emissions, local context injection, and the\nadvanced custom-planner seam.\n\n### Runnable Example Consumer\n\n[`examples/sift-embed`](examples/sift-embed) is the canonical runnable\nembedding reference. It is a standalone Rust crate that depends on `sift`\nthrough the crate-root facade and exposes a minimal `sift-embed` CLI.\n\nFrom the repo root:\n\n```bash\njust embed-build\njust embed-sift tests/fixtures/rich-docs \"architecture decision\"\n```\n\nYou can also run the example directly:\n\n```bash\ncargo run --manifest-path examples/sift-embed/Cargo.toml -- search \"agentic search\"\ncargo run --manifest-path examples/sift-embed/Cargo.toml -- search tests/fixtures/rich-docs \"architecture decision\"\n```\n\nIf `PATH` is omitted, `sift-embed search \"\u003cterm\u003e\"` searches the current\ndirectory. See [`examples/sift-embed/README.md`](examples/sift-embed/README.md)\nfor the runnable example notes.\n\n### Local Model Preparation\n\nEmbedders that need local model artifacts for the current Candle-backed runtime\ncan use the stable crate-root preparation seam:\n\n```rust\nuse sift::{ModelRuntimeContract, ModelSource, prepare_model};\n\nlet prepared = prepare_model(\n    ModelSource::hugging_face_revision(\"prism-ml/Bonsai-8B-gguf\", \"main\"),\n    ModelRuntimeContract::CandleSafetensorsBundle,\n)?;\nprintln!(\"{}\", prepared.root.display());\n```\n\nThis surface is a compatibility/preparation seam. If a source is already\ncompatible, sift reuses it. If the source is GGUF-backed, sift can invoke\nmetamorph to translate it into a Candle-loadable safetensors bundle. That\ntranslation is lossy and does not preserve the original 1-bit runtime\nefficiency; native 1-bit execution support remains a separate future project.\n\n### Add The Dependency\n\nUse the git repository until a versioned registry release is part of your\ndelivery path:\n\n```toml\n[dependencies]\nsift = { git = \"https://github.com/rupurt/sift\" }\n```\n\nFor local development against a checked-out copy:\n\n```toml\n[dependencies]\nsift = { path = \"../sift\" }\n```\n\n### Minimal Embedding Example\n\n```rust\nuse sift::{Fusion, Retriever, Reranking, SearchInput, SearchOptions, Sift};\n\nfn main() -\u003e anyhow::Result\u003c()\u003e {\n    let sift = Sift::builder().build();\n\n    let response = sift.search(\n        SearchInput::new(\"./docs\", \"agentic search\").with_options(\n            SearchOptions::default()\n                .with_strategy(\"bm25\")\n                .with_retrievers(vec![Retriever::Bm25])\n                .with_fusion(Fusion::Rrf)\n                .with_reranking(Reranking::None)\n                .with_limit(5)\n                .with_shortlist(5),\n        ),\n    )?;\n\n    for hit in response.hits {\n        println!(\"{} {}\", hit.rank, hit.path);\n    }\n\n    Ok(())\n}\n```\n\n## How Sift Works\n\nAt runtime, `sift` orchestrates a high-performance asset pipeline and retrieval\nruntime that can be invoked directly or wrapped by a turn-aware controller:\n\n```mermaid\nflowchart TD\n  A[CLI parse] --\u003e B[Resolve PATH + QUERY]\n  B --\u003e C[Directory Walk]\n\n  subgraph Pipeline [\"Optimized Asset Pipeline\"]\n    C1{Heuristic Hit?} --\u003e|Yes| C2[Mapped I/O: Load Document Blob]\n    C1 --\u003e|No| C3[BLAKE3 Hash + Check CAS]\n    C3 --\u003e|Hit| C4[Update Manifest + Load Blob]\n    C3 --\u003e|Miss| C5[Extract + Embed + Save Blob]\n  end\n\n  C --\u003e C1\n  C2 --\u003e D[Build Transient Corpus]\n  C4 --\u003e D\n  C5 --\u003e D\n\n  subgraph Strategy [\"Hybrid Retrieval Core\"]\n    D --\u003e J{Execution Path}\n    J --\u003e|Direct| E{Expansion}\n    J --\u003e|Turn-aware| A0[Plan Turn / Manage Context]\n    A0 --\u003e E\n    E --\u003e|None| E1[Direct Query]\n    E --\u003e|HyDE| E2[LLM: Hypothetical Answer]\n    E --\u003e|SPLADE| E3[LLM: Generative Expansion]\n    E --\u003e|Classified| E4[LLM: Intent Classification]\n\n    E1 --\u003e F{Retrievers}\n    E2 --\u003e F\n    E3 --\u003e F\n    E4 --\u003e F\n\n    F --\u003e|Lexical| F1[BM25]\n    F --\u003e|Exact| F2[Phrase]\n    F --\u003e|Path Intent| F3[Path Fuzzy]\n    F --\u003e|Snippet Evidence| F4[Segment Fuzzy]\n    F --\u003e|Semantic| F5[Vector - SIMD Dot Product]\n\n    subgraph QueryCache [\"Query Embedding Cache\"]\n      F5\n    end\n\n    F1 --\u003e G[Reciprocal Rank Fusion]\n    F2 --\u003e G\n    F3 --\u003e G\n    F4 --\u003e G\n    F5 --\u003e G\n    G --\u003e H{Reranking}\n    H --\u003e|Basic| H1[Position-Aware: Path + Heading + Definition Bonuses]\n    H --\u003e|Advanced| H2[LLM - Deep Semantic Pass]\n    H1 --\u003e K{Emission}\n    H2 --\u003e K\n    K --\u003e|CLI| I[Text or JSON search output]\n    K --\u003e|Library| I2[Structured response or emission]\n    K -.-\u003e|Need More Evidence| A0\n  end\n```\n\nToday the executable ships direct search, planner-driven `search --agent`, a\nbounded graph mode, and evaluation-oriented autonomous/controller benchmarks.\nThe library exposes direct, controller, and supported autonomous surfaces over\nthe same retrieval substrate. The next formal layer is richer persisted agent\nmemory and more adaptive graph execution over the shipped bounded graph\ncontract.\n\n## Performance \u0026 Scalability\n\n`sift` is optimized for speed without sacrificing its stateless UX:\n\n- **Zero-inference repeat search:** Reuses cached document blobs and query embeddings where possible.\n- **SIMD acceleration:** Vector similarity calculations use hardware-specific SIMD instructions.\n- **Mapped I/O:** Uses `mmap` for document blob reads.\n- **Fast-path heuristics:** Filesystem metadata checks allow unchanged files to bypass hashing and extraction.\n\n## Documentation Map\n\n- **[USER_GUIDE.md](USER_GUIDE.md):** End-user guide for direct search workflows.\n- **[CONFIGURATION.md](CONFIGURATION.md):** `sift.toml`, strategies, prompts, and environment variables.\n- **[EVALUATIONS.md](EVALUATIONS.md):** Retrieval and agentic evaluation workflows.\n- **[LIBRARY.md](LIBRARY.md):** Crate-root embedding guide for all supported library modes.\n- **[ARCHITECTURE.md](ARCHITECTURE.md):** Architecture, execution seams, and implementation status.\n- **[WORLD.md](WORLD.md):** Conceptual model for retrieval, emission, and structural evidence.\n- **[RESEARCH.md](RESEARCH.md):** Forward-looking graph-IR and execution direction.\n- **[CONSTITUTION.md](CONSTITUTION.md):** Non-negotiable engineering and architecture principles.\n- **[RELEASE.md](RELEASE.md):** Release checklist and artifact verification process.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupurt%2Fsift","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frupurt%2Fsift","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupurt%2Fsift/lists"}