{"id":50916972,"url":"https://github.com/nlm/ragstore","last_synced_at":"2026-06-16T16:03:25.316Z","repository":{"id":354702712,"uuid":"1216868610","full_name":"nlm/ragstore","owner":"nlm","description":"Headless document search engine for LLM agents. A single Linux binary, no dependencies, no daemon — index and search in one command. -- Disclaimer: 100% Vibe Coded","archived":false,"fork":false,"pushed_at":"2026-04-29T16:52:42.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-29T18:30:49.475Z","etag":null,"topics":["ai-agents","embedded","skills"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nlm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-21T10:05:58.000Z","updated_at":"2026-04-29T16:52:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nlm/ragstore","commit_stats":null,"previous_names":["nlm/ragstore"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nlm/ragstore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlm%2Fragstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlm%2Fragstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlm%2Fragstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlm%2Fragstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nlm","download_url":"https://codeload.github.com/nlm/ragstore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlm%2Fragstore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34412795,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","embedded","skills"],"created_at":"2026-06-16T16:03:24.393Z","updated_at":"2026-06-16T16:03:25.311Z","avatar_url":"https://github.com/nlm.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ragstore\n\nHeadless document search engine for LLM agents. A single Linux binary, no dependencies, no daemon — index and search in one command.\n\n## Why ragstore\n\nLLM agents need access to a document knowledge base without managing infrastructure. `ragstore` is designed to run in the same container as the agent: one binary to call, one index file on disk, clean JSON output ready to consume.\n\nSearch is powered by **BM25** (Okapi BM25), the ranking algorithm used by Elasticsearch and Lucene. It's the best fit for an isolated environment: no network calls, no embedding model to load, sub-millisecond latency, and excellent results on technical and domain-specific vocabulary.\n\n---\n\n## Installation\n\n```bash\n# Download the binary (Linux x86_64)\ncurl -Lo ragstore https://your-host/ragstore\nchmod +x ragstore\n\n# Or build from source (Go 1.22+)\ngo build -o ragstore -ldflags=\"-s -w\" .\n```\n\nThe binary is static, self-contained, 2 MB. No system dependencies required.\n\n---\n\n## Quick start\n\n```bash\n# Index a document directory\nRAG_DB=/data/rag.db.json ./ragstore index /data/docs --chunk-size 250\n\n# Search by topic\nRAG_DB=/data/rag.db.json ./ragstore search \"machine learning neural networks\" --top 5\n```\n\nOutput:\n\n```json\n{\n  \"ok\": true,\n  \"data\": [\n    {\n      \"id\": \"211ccacbd4cf3c78\",\n      \"path\": \"/data/docs/ml_intro.md\",\n      \"title\": \"ml_intro.md\",\n      \"chunk\": 0,\n      \"score\": 5.571,\n      \"snippet\": \"…Supervised learning uses labelled data to train classification and regression models. Unsupervised learning discovers hidden patterns in unlabelled data…\"\n    }\n  ]\n}\n```\n\n---\n\n## Command reference\n\n### `index \u003cpath\u003e [--chunk-size N]`\n\nIndex a file or directory (recursive). Files already present in the index are skipped — the command is **idempotent**.\n\n```bash\nragstore index /data/docs                    # default chunk size: 300 words\nragstore index /data/docs --chunk-size 150   # finer-grained chunks\nragstore index /data/docs /data/extra        # multiple paths\n```\n\nSupported formats: `.txt`, `.md`, `.rst`, `.org`, `.json`, `.yaml`, `.csv`, `.html`, `.py`, `.go`, `.js`, `.ts`, `.java`, `.c`, `.cpp`, `.rs`, `.sh`, and any readable text file. PDF is supported if `pdftotext` is installed (`apt install poppler-utils`).\n\nChunking splits text by **paragraphs** until the target word count is reached — chunks always break on natural boundaries.\n\n| `--chunk-size` | Recommended use |\n|---|---|\n| 100–150 | Dense technical docs, maximum precision |\n| 200–300 | General Q\u0026A *(recommended default)* |\n| 400–500 | Wide context, narrative documents |\n\n#### Excluding Files with `.ragignore`\n\nPlace a `.ragignore` file in any directory to exclude files and directories from indexing. The syntax is compatible with `.gitignore`:\n\n```bash\n# .ragignore example\nnode_modules/          # ignore directory\n*.log                  # ignore log files\n**/test*               # ignore test files anywhere\nbuild/                 # ignore build directory\n!important.log         # negation: don't ignore important.log\n# comment              # comments are ignored\n```\n\n**Pattern syntax:**\n- `*.ext` — match files by extension\n- `dirname/` — match directories only (trailing slash)\n- `**/pattern` — match anywhere in the directory tree\n- `!pattern` — negation (re-include previously excluded files)\n- `# comment` — comment lines\n\nMultiple `.ragignore` files can exist in subdirectories. Patterns accumulate hierarchically, and child `.ragignore` files can override parent patterns using negation (`!`).\n\n### `search \u003cquery\u003e [--top N]`\n\nSearch the index and return the most relevant chunks, ranked by descending BM25 score.\n\n```bash\nragstore search \"docker kubernetes orchestration\" --top 5\nragstore search \"refund policy terms\" --top 3\n```\n\nThe `snippet` field contains the most relevant excerpt from the chunk, centered around the query terms. It can be used directly to formulate an answer.\n\n### `get \u003cid\u003e`\n\nReturn the full content of a chunk by its ID (obtained from `search`).\n\n```bash\nragstore get 211ccacbd4cf3c78\n```\n\nUseful when a snippet is truncated and more context is needed.\n\n### `list`\n\nList all indexed documents with their path, title, chunk number, and word count.\n\n```bash\nragstore list\n```\n\n### `delete \u003cid|path\u003e`\n\nDelete one or more chunks by ID or path prefix.\n\n```bash\nragstore delete 211ccacbd4cf3c78        # a single chunk\nragstore delete /data/docs/old.md       # all chunks from a file\nragstore delete /data/docs/archive/     # all chunks under a directory\n```\n\n### `stats`\n\nDisplay index statistics.\n\n```bash\nragstore stats\n```\n\n```json\n{\n  \"ok\": true,\n  \"data\": {\n    \"total_chunks\": 42,\n    \"unique_files\": 12,\n    \"unique_terms\": 3841,\n    \"avg_chunk_len\": 187,\n    \"chunk_size_cfg\": 250,\n    \"db\": \"/data/rag.db.json\"\n  }\n}\n```\n\n### `interactive`\n\nRead commands from stdin, one per line. Useful for sending multiple queries in batch without restarting the binary.\n\n```bash\necho \"search machine learning supervised --top 3\" | ragstore interactive\n```\n\n---\n\n## Configuration\n\n| Environment variable | Default | Description |\n|---|---|---|\n| `RAG_DB` | `./rag.db.json` | Path to the index file |\n\nThe index is a self-contained JSON file. Multiple indexes can coexist by pointing `RAG_DB` to different paths.\n\n---\n\n## LLM agent integration\n\n`ragstore` is designed to be referenced as an agent **skill**. The typical workflow:\n\n1. The agent receives a question from the user\n2. It extracts 3–6 representative keywords from the topic\n3. It calls `ragstore search \"\u003ckeywords\u003e\" --top 5`\n4. It reads the `snippet` fields from the results to formulate its answer\n5. If a snippet is insufficient, it calls `ragstore get \u003cid\u003e` for the full content\n\n```\nUser:  \"What is our vacation policy?\"\nAgent → ragstore search \"vacation policy days leave\" --top 3\nAgent → reads snippets → formulates answer\n```\n\nThe `SKILL_ragstore.md` file in this repository describes this workflow in detail and can be used directly as a system skill.\n\n### Container integration\n\n```dockerfile\nCOPY ragstore /usr/local/bin/ragstore\nRUN chmod +x /usr/local/bin/ragstore\nENV RAG_DB=/data/rag.db.json\n\n# Pre-index documents at image build time\nRUN ragstore index /data/docs --chunk-size 250\n```\n\n---\n\n## Output format\n\nAll commands return JSON to **stdout**. Errors go to **stderr** with exit code 1.\n\n```json\n// Success\n{ \"ok\": true, \"message\": \"...\", \"data\": { ... } }\n\n// Error\n{ \"ok\": false, \"message\": \"error description\" }\n```\n\n---\n\n## Building from source\n\n```bash\n# Prerequisites: Go 1.22+\ngit clone https://github.com/your-org/ragstore\ncd ragstore\n\n# Standard build\ngo build -o ragstore .\n\n# Optimized build (smaller size, no debug symbols)\ngo build -o ragstore -ldflags=\"-s -w\" .\n\n# Static build (for Alpine / minimal containers)\nCGO_ENABLED=0 GOOS=linux go build -o ragstore -ldflags=\"-s -w\" .\n```\n\nThe code depends only on the Go standard library. No external modules required.\n\n---\n\n## Algorithm\n\n**BM25 (Okapi BM25)** with standard parameters `k1=1.5`, `b=0.75`.\n\nScoring of a document `d` for a query `q`:\n\n$$\\text{score}(d, q) = \\sum_{t \\in q} \\text{IDF}(t) \\cdot \\frac{f(t,d) \\cdot (k_1 + 1)}{f(t,d) + k_1 \\left(1 - b + b \\cdot \\frac{|d|}{\\text{avgdl}}\\right)}$$\n\nThe tokenizer lowercases text, strips punctuation, and filters a multilingual stop-word list. It is language-agnostic and works equally well on English, French, or source code.\n\n---\n\n## Known limitations\n\n- **No semantic search**: BM25 does not understand synonyms or paraphrasing. Compensate by broadening the query (e.g. `\"car automobile vehicle\"`). An LLM agent can rephrase automatically if the first results are insufficient.\n- **Basic PDF support**: without `pdftotext`, PDF extraction is limited to printable ASCII characters. Install `poppler-utils` for proper extraction.\n- **No incremental updates**: modifying an already-indexed file does not update its chunks. Run `delete \u003cpath\u003e` then `index \u003cpath\u003e` to reindex.\n- **Scalability**: the index is loaded entirely into memory. Suitable for up to ~100k chunks (roughly 500 MB of raw text). Beyond that, consider Qdrant or Elasticsearch.\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlm%2Fragstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnlm%2Fragstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlm%2Fragstore/lists"}