{"id":51439706,"url":"https://github.com/datnguyenzzz/semantic-grep","last_synced_at":"2026-07-05T10:01:53.263Z","repository":{"id":365789617,"uuid":"1272523322","full_name":"datnguyenzzz/semantic-grep","owner":"datnguyenzzz","description":"Make multiple codebases available as context for any coding agent without spending too many resources or sacrificing quality","archived":false,"fork":false,"pushed_at":"2026-06-28T10:07:24.000Z","size":63788,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-28T12:06:46.905Z","etag":null,"topics":["agent","claude-code","codebase-indexing","gemini","knowledge-graph","llm","mcp-server","mcp-tools","memory-agent","model-context-protocol","rag","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datnguyenzzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-17T17:34:07.000Z","updated_at":"2026-06-28T10:07:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/datnguyenzzz/semantic-grep","commit_stats":null,"previous_names":["datnguyenzzz/agent-context","datnguyenzzz/semantic-grep"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/datnguyenzzz/semantic-grep","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datnguyenzzz%2Fsemantic-grep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datnguyenzzz%2Fsemantic-grep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datnguyenzzz%2Fsemantic-grep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datnguyenzzz%2Fsemantic-grep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datnguyenzzz","download_url":"https://codeload.github.com/datnguyenzzz/semantic-grep/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datnguyenzzz%2Fsemantic-grep/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35150063,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","claude-code","codebase-indexing","gemini","knowledge-graph","llm","mcp-server","mcp-tools","memory-agent","model-context-protocol","rag","vector-search"],"created_at":"2026-07-05T10:01:52.489Z","updated_at":"2026-07-05T10:01:53.253Z","avatar_url":"https://github.com/datnguyenzzz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## 💡 Motivation: Semantic Grep\n\nIn modern agentic harness workflows, `grep` is powerful and often [is all you need](https://arxiv.org/pdf/2605.15184). However, raw `grep` alone is highly token-inefficient because it returns unscoped matching lines and massive boilerplate noise, forcing the agent to ingest everything into its context window, driving up API costs. With an attempt to resolve this, we are introducing `Semantic Grep`. By running the grep command on your codebase and filtering the results on-the-fly using semantic meaning, we locate, extract, and load only the exact, containing AST functions. This delivers 100% precise retrieval with ~0 token waste\n\nThe agent can also trace the function's execution paths bidirectionally using our lightweight AST Call Graph tool. Instead of forcing the agent to read and piece together dozens of separate files, this relational graph lets the agent query callers and callees on-demand, explaining complex execution flows with minimal context usage.\n\n\u003e ⚠️ **Note:** Currently, the codebase indexer and call graph builder support indexing `.go`, `.tf`, `.py` and `.yml` files.\n\n---\n\n## 🛠 Exposed MCP Tools\n\n1.  **`search_memory`**: Semantic search across indexed workspace code blocks.\n2.  **`search_call_graph`**: Explores bidirectional call chains (caller/callee)\n\n---\n\n## 🚀 Quick Start\n\n### 1. Build and Install\n\nYou can register the codebase indexer extension with both Gemini CLI and Claude Code CLI. Run the following to automatically build and register with whichever CLIs are available on your system:\n\n```bash\nmake install\n```\n\nAlternatively, install individually depending on your preferred CLI environment:\n\n*   **For Gemini CLI:**\n    ```bash\n    make install-gemini\n    ```\n*   **For Claude Code CLI:**\n    ```bash\n    make install-claude\n    ```\n\n### 2. Index a Codebase\n\nBefore querying, index your codebase directory. This recursively scans, chunks, and quantizes vectors into DuckDB and TurboQuant files:\n\n```bash\nmake index DIR=/path/to/your/codebase\n```\n\n### 3. Run Tests\n```bash\nmake test         # Run unit tests\nmake test-all     # Run all tests \u0026 database self-checks\n```\n\n---\n\n## ⚙ Configuration\n\nConfigure via environment variables:\n*   `LITELLM_BASE_URL`: API base URL (Default: `http://localhost:36253/v1`)\n*   `LITELLM_EMBEDDING_MODEL`: Embedding model (Default: `gemini-embedding-001`)\n\n---\n\n## 📐 System Architecture\n\n```mermaid\nflowchart TD\n    %% Entrypoints \u0026 Client\n    subgraph Client [\"Client / Agent Entrypoints\"]\n        cli[Agent]\n        idx_cli[Indexer CLI]\n    end\n\n    %% Core Components\n    subgraph Core [\"semantic-grep Core Engine \u0026 Server\"]\n        merkle[Merkle Sync\u003cbr/\u003einternal/merkle]\n        splitter[Splitter\u003cbr/\u003einternal/splitter]\n        llm[LLM Client\u003cbr/\u003einternal/llm]\n        callgraph[Call Graph\u003cbr/\u003einternal/callgraph]\n        mcp_srv[MCP Server\u003cbr/\u003ecmd/server]\n    end\n\n    subgraph Toolings \n        ggrep[ggrep - faster grep]\n    end\n\n    %% Shared Environment \u0026 Databases\n    subgraph Storage [\"Shared Environment \u0026 Storage\"]\n        duckdb_file[(DuckDB\u003cbr/\u003e Codebases Metadata)]\n        tqv_file[(semantic-grep.tqv\u003cbr/\u003eQuantized Vectors)]\n    end\n\n    %% User's Environment\n    subgraph Workspace [\"User's Local Workspace (On Disk)\"]\n        code_files[User's Source Code\u003cbr/\u003e.go, .tf, .py, .yaml]\n    end\n\n    %% External Provider\n    litellm[LiteLLM Embedding Provider]\n\n    %% ──────────────────────────────────────────────────────────\n    %% WRITE PATH: Indexing \u0026 Ingestion (Thin lines / Dashes)\n    %% ──────────────────────────────────────────────────────────\n    idx_cli --\u003e|1. Run index| merkle\n    mcp_srv --\u003e|Periodically refresh the indexed codebases| merkle\n    code_files -.-\u003e|Scan Directory| merkle\n    merkle --\u003e|2. Split Files| splitter\n    merkle --\u003e|3. Get Embeddings| llm\n    llm \u003c--\u003e|4. API Call| litellm\n    merkle --\u003e|5. Save Function Metadata| duckdb_file\n    merkle --\u003e|5. Save Quantized Vectors| tqv_file\n    merkle --\u003e|6. Parse Call Graph| callgraph\n    callgraph --\u003e|7. Save Nodes \u0026 Edges| duckdb_file\n\n    %% ──────────────────────────────────────────────────────────\n    %% READ PATH: MCP Search \u0026 Retrieval (Thick Double Lines)\n    %% ──────────────────────────────────────────────────────────\n    cli ===\u003e|1. Call search_memory| mcp_srv\n    cli ===\u003e|1. Call search_call_graph| mcp_srv\n    mcp_srv ===\u003e|2. Get Query Embedding| llm\n    mcp_srv ===\u003e|3. Search Vectors| tqv_file\n    mcp_srv ===\u003e|4. Run ggrep on Local Disk| ggrep\n    ggrep ===\u003e|5. Read Source Code| code_files\n    mcp_srv ===\u003e|6. Invert matched lines to function scopes| duckdb_file\n    mcp_srv ===\u003e|7. Read top-scoring code chunks on-the-fly| code_files\n    mcp_srv ===\u003e|8. Return Context| cli\n\n    %% ──────────────────────────────────────────────────────────\n    %% FLOW LEGEND (Self-Explanatory Arrow Styles)\n    %% ──────────────────────────────────────────────────────────\n    subgraph Legend [\"Legend / Flow Styles\"]\n        direction LR\n        style_w[ ] --\u003e|Thin line / Dashed| desc_w(Write / Ingestion Flow)\n        style_r[ ] ===\u003e|Thick double line| desc_r(Read / Search Query Flow)\n    end\n```\n\n### 📐 Core Technical Pillars \u0026 Decisions\n\n1. **Cryptographic Merkle Trees for Incremental Syncs:**\n   To prevent expensive, redundant re-indexing of unaltered codebases, `semantic-grep` recursively structures directory states as SHA-256 cryptographic Merkle Trees. During subsequent indexing sweeps, it diffs node hashes in milliseconds to isolate only the **filesystem delta (added, modified, or deleted files)**. Only the delta is processed and embedded, drastically reducing API token costs and sweep times.\n\n2. **DuckDB for Relational Metadata and Call Graphs:**\n   We utilize **[DuckDB](https://github.com/duckdb/duckdb)** as our metadata and relational store. DuckDB is a highly performant, serverless, in-process analytical (OLAP) database engine that excels at complex queries and joins. It provides complete transactional safety (ACID), runs entirely locally with zero daemon processes, and is optimized for querying dense AST call graph nodes, edges, and function scopes (`function_name`, `cwd`, `line_start`, `line_end`) with **zero raw source code stored in-database**\n\n3. **TurboQuant for In-Process Vector Quantization:**\n   Instead of depending on an expensive, resource-heavy external vector database that is costly to host, run, and maintain, `semantic-grep` runs **[TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/)** directly inside the Go process. TurboQuant compresses high-dimensional vectors (by up to 14x on disk) using random orthogonal rotation and Lloyd-Max scalar quantization on the Beta distribution. Most importantly, **TurboQuant requires no pre-training data or prebuilt codebooks**, providing a highly optimized, zero-maintenance, local vector quantization engine without sacrificing similarity search accuracy.\n\n4. **Zero-Storage Metadata-Guided Hybrid Search with RRF \u0026 ggrep:**\n   To guarantee absolute retrieval accuracy without code footprint replication, `semantic-grep` fuses **Dense Semantic search** (TurboQuant) and **Sparse Lexical search** using **[Reciprocal Rank Fusion](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)**. \n   Our custom, ultra-fast `ggrep` library is linked natively within the server process, running high-speed multi-threaded Regex scans directly on the local codebase. When lines match, the engine performs an **inverted query in DuckDB** to map matched lines to their containing logical AST functions on-the-fly (`line_start \u003c= matched_line \u003c= line_end`). Raw code contents of top-scoring candidates are streamed from the local filesystem on-the-fly, achieving sub-millisecond search latencies and a pure zero-copy storage footprint!\n\n---\n\n## 📊 TurboQuant Vector Compression Benchmark\n\n\u003e See [script](https://github.com/datnguyenzzz/semantic-grep/blob/main/scripts/benchmark_compression_test.go) \n\n```\n================================================================================\n        📊  TURBOQUANT VECTOR COMPRESSION BENCHMARK SUITE  📊                 \n================================================================================\n\n📁 Targets: Aggregated Index (across 11 codebases)\n   • Scanned Files: 17,839 | Total Semantic Chunks: 139,072 | Dimensions: 3072\n   • Total Lines of Code (LOC): 3,435,711 | DuckDB Metadata Size: 0.76 MiB\n  -------------------------------------------------------------------------------- \n   │ Data Footprint Type            │ Footprint Size │ Comp. Ratio │ Savings    │\n   ├────────────────────────────────┼────────────────┼─────────────┼────────────┤\n   │ [1] Standard Float32[] RAM     │    1629.75 MiB │      1.0x   │     0.0%   │\n   │ [2] TurboQuant In-Memory Map   │     206.11 MiB │      7.9x   │    87.4%   │\n   │ [3] TurboQuant On-Disk .tqv    │     108.76 MiB │     15.0x   │    93.3%   │\n   └────────────────────────────────┴────────────────┴─────────────┴────────────┘\n\n   📈 Visual Storage Footprint Comparison (Bar Scale):\n\n   Standard Float32[] RAM   : [████████████████████████████████████████] (1629.75 MiB)\n   TurboQuant In-Memory Map : [█████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] (206.11 MiB)\n   TurboQuant On-Disk .tqv  : [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] (108.76 MiB)\n\n================================================================================\n```\n\n---\n\n## 📈 FAISS vs. TurboQuant Recall Accuracy Comparison\n\nTo evaluate the mathematical accuracy of our quantized TurboQuant local vector index compared to industry-standard Product Quantization (FAISS), we measure **Recall-1-@k**—the frequency with which the absolute true nearest neighbor (ground-truth unquantized top-1) is captured within the top-$k$ quantized results. \n\n\u003e We run the comparision with the [dbpedia-entities-openai3-text-embedding-3-large-1536-1M](https://huggingface.co/datasets/Qdrant/dbpedia-entities-openai3-text-embedding-3-large-1536-1M) dataset. See [script](https://github.com/datnguyenzzz/semantic-grep/blob/main/scripts/bench_turboquant_test.go)\n\n*   **1536 dimensions**\n\n![dim_1536](results/recall_chart_d1536_4bit.png)\n\n*   **3072 dimensions**\n\n![dim_3072](results/recall_chart_d3072_4bit.png)\n\n---\n\n## 📊 Search Effectiveness Benchmark (Semantic vs. Grep vs. Semantic Grep)\n\nTo evaluate real-world retrieval effectiveness under realistic search conditions, we measure how frequently each search pipeline captures the correct document under deterministic query-vector perturbation (**20% noise factor**). We evaluate 3 search strategies:\n1. **Semantic:** Concepts-only dense semantic search using our compressed 4-bit `turbovec` index.\n2. **Grep:** Precise literal/keyword matching via our `ggrep` lexical search path.\n3. **Semantic Grep:** Our optimized hybrid path fusing semantic conceptually-guided results with exact keyword matches using Reciprocal Rank Fusion (RRF).\n\nRunning the benchmarks outputs a comparative dashboard summarizing **Recall-1-@k** and **Mean Reciprocal Rank (MRR)**.\n\n\u003e We run the comparison with the [dbpedia-entities-openai3-text-embedding-3-large-1536-1M](https://huggingface.co/datasets/Qdrant/dbpedia-entities-openai3-text-embedding-3-large-1536-1M) dataset. See [script](https://github.com/datnguyenzzz/semantic-grep/blob/main/scripts/benchmark_effectiveness_test.go) \n\n*   **1536 Dimensions (100,000 documents):**\n\n![effectiveness_1536](results/hybrid_effectiveness_chart_d1536_4bit.png)\n\n*   **3072 Dimensions (50,000 documents):**\n\n![effectiveness_3072](results/hybrid_effectiveness_chart_d3072_4bit.png)\n\nThis scientifically proves how our **Semantic Grep**—fusing the conceptual power of semantic search with the absolute precision of lexical `ggrep`—achieves near-perfect retrieval recall and rank elevation.\n\n---\n\n## ⚡ ggrep Performance Benchmark\n\nTo evaluate the scanning speed and scalability of our custom `ggrep` engine, we benchmarked it against industry-standard tools: **OS grep**, **git-grep**, **ripgrep (`rg`)**, our ggrep using the standard library `regexp` package (`ggrep-std`), and our optimized custom DFA-based regular expression engine. \n\nThe benchmarks evaluate scaling across 50 production-grade literal search queries and 50 POSIX-ERE compatible regular expression queries over large source code repositories.\n\n*   **Literal Scaling Performance Comparison:**\n\n![ggrep_bench_literal](results/ggrep_bench_literal_chart.png)\n\n*   **Regex Scaling Performance Comparison:**\n\n![ggrep_bench_regex](results/ggrep_bench_regex_chart.png)\n\nThis demonstrates that our custom `ggrep` with the native DFA regex resolver matches or outperforms standard grep utilities, delivering sub-millisecond scanning latencies by using concurrent, lock-free, zero-allocation scheduling.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatnguyenzzz%2Fsemantic-grep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatnguyenzzz%2Fsemantic-grep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatnguyenzzz%2Fsemantic-grep/lists"}