{"id":36473767,"url":"https://github.com/metawake/ragtune","last_synced_at":"2026-02-25T15:18:03.191Z","repository":{"id":331992141,"uuid":"1121702246","full_name":"metawake/ragtune","owner":"metawake","description":"EXPLAIN ANALYZE for RAG retrieval — inspect, debug, benchmark, and tune your retrieval layer","archived":false,"fork":false,"pushed_at":"2026-01-11T18:16:19.000Z","size":3087,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-12T02:44:14.195Z","etag":null,"topics":["benchmarking","chroma","cli","developer-tools","embeddings","evaluation","llm","metrics","pgvector","pinecone","qdrant","rag","recall","retrieval-augmented-generation","vector-search","weaviate"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metawake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"custom":["https://etherscan.io/address/0x0a542565b3615e8fc934cc3cc4921a0c22e5dc5e"]}},"created_at":"2025-12-23T12:12:17.000Z","updated_at":"2026-01-11T18:16:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/metawake/ragtune","commit_stats":null,"previous_names":["metawake/ragtune"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/metawake/ragtune","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metawake%2Fragtune","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metawake%2Fragtune/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metawake%2Fragtune/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metawake%2Fragtune/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metawake","download_url":"https://codeload.github.com/metawake/ragtune/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metawake%2Fragtune/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28441331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-15T00:55:22.719Z","status":"online","status_checked_at":"2026-01-15T02:00:08.019Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","chroma","cli","developer-tools","embeddings","evaluation","llm","metrics","pgvector","pinecone","qdrant","rag","recall","retrieval-augmented-generation","vector-search","weaviate"],"created_at":"2026-01-12T00:44:40.826Z","updated_at":"2026-02-25T15:18:03.181Z","avatar_url":"https://github.com/metawake.png","language":"Go","funding_links":["https://etherscan.io/address/0x0a542565b3615e8fc934cc3cc4921a0c22e5dc5e"],"categories":["Security"],"sub_categories":["Observability"],"readme":"# RagTune\n\n[![Go Version](https://img.shields.io/badge/go-1.22+-00ADD8?logo=go)](https://go.dev/)\n[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n[![Release](https://img.shields.io/github/v/release/metawake/ragtune?include_prereleases)](https://github.com/metawake/ragtune/releases)\n\n**Debug, benchmark, and monitor your RAG retrieval layer.** EXPLAIN ANALYZE for production RAG.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo.gif\" alt=\"RagTune demo\" width=\"700\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quickstart\"\u003eQuickstart\u003c/a\u003e •\n  \u003ca href=\"#commands\"\u003eCommands\u003c/a\u003e •\n  \u003ca href=\"#why-ragtune\"\u003eWhy RagTune\u003c/a\u003e •\n  \u003ca href=\"docs/concepts.md\"\u003eConcepts\u003c/a\u003e •\n  \u003ca href=\"docs/faq.md\"\u003eFAQ\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n| I want to... | Command |\n|--------------|---------|\n| **Debug a single query** | `ragtune explain \"my query\" --collection prod` |\n| **Run batch evaluation** | `ragtune simulate --collection prod --queries queries.json` |\n| **Get confidence intervals** | `ragtune simulate --queries queries.json --bootstrap 20` |\n| **Set up CI/CD quality gates** | `ragtune simulate --ci --min-recall 0.85` |\n| **Detect regressions** | `ragtune simulate --baseline runs/latest.json --fail-on-regression` |\n| **Compare embedders** | `ragtune compare --embedders ollama,openai --docs ./docs` |\n| **Evaluate external chunkers** | `ragtune ingest ./chunks/ --collection test --pre-chunked` |\n| **Find missed answer content** | `ragtune simulate --queries needles.json` (with needle annotations) |\n| **Quick health check** | `ragtune audit --collection prod --queries queries.json` |\n\n---\n\n## Quickstart\n\n```bash\n# 1. Start vector store\ndocker run -d -p 6333:6333 -p 6334:6334 qdrant/qdrant\n\n# 2. Ingest documents\nragtune ingest ./docs --collection my-docs --embedder ollama\n\n# 3. Debug retrieval\nragtune explain \"How do I reset my password?\" --collection my-docs\n```\n\nNo API keys needed with Ollama (runs locally).\n\n### Evaluate External Chunkers (POMA, Unstructured, LlamaIndex)\n\nAlready chunked your documents with an external tool? Use `--pre-chunked` to ingest them as-is — one file per chunk, no re-splitting:\n\n```bash\n# Ingest pre-chunked data (each file = one embedding unit)\nragtune ingest ./poma-chunksets/ --collection poma-test --embedder ollama --pre-chunked\n\n# Compare against naive chunking\nragtune ingest ./raw-docs/ --collection naive-test --embedder ollama --chunk-size 512\n\n# Benchmark both\nragtune simulate --collection poma-test --queries queries.json --bootstrap 20\nragtune simulate --collection naive-test --queries queries.json --bootstrap 20\n```\n\n### Already using PostgreSQL with pgvector?\n\nSkip Docker entirely. Use your existing database:\n\n```bash\nragtune ingest ./docs --collection my-docs --embedder ollama \\\n    --store pgvector --pgvector-url postgres://user:pass@localhost/mydb\n\nragtune explain \"How do I reset my password?\" --collection my-docs \\\n    --store pgvector --pgvector-url postgres://user:pass@localhost/mydb\n```\n\n### Build Your Test Suite\n\n```bash\n# Save queries as you debug\nragtune explain \"How do I reset my password?\" --collection my-docs --save\nragtune explain \"What are the rate limits?\" --collection my-docs --save\n\n# Run evaluation once you have 20+ queries\nragtune simulate --collection my-docs --queries golden-queries.json\n```\n\nEach `--save` adds the query to `golden-queries.json`.\n\n---\n\n## What You'll See\n\n### explain — Debug a Query\n\n```\nQuery: \"How do I reset my password?\"\n\n[1] Score: 0.8934 | Source: docs/auth/password-reset.md\n    Text: To reset your password: 1. Click \"Forgot Password\"...\n\n[2] Score: 0.8521 | Source: docs/auth/account-security.md\n    Text: Account Security ## Password Management...\n\nDIAGNOSTICS\n  Score range: 0.7234 - 0.8934 (spread: 0.1700)\n  ✓ Strong top match (\u003e0.85): likely high-quality retrieval\n```\n\n### simulate — Batch Metrics\n\n```\nRunning 50 queries...\n\n  Recall@5:   0.82    MRR: 0.76    Coverage: 0.94\n  Latency:    p50=45ms  p95=120ms\n\nFAILURES: 3 queries with Recall@5 = 0\n  ✗ \"How do I configure SSO?\"\n    Expected: [sso-guide.md], Retrieved: [api-keys.md...]\n\n💡 Run `ragtune explain \"\u003cquery\u003e\"` to debug\n```\n\n### NeedleCoverage — Find What Recall Misses\n\nRecall@K tells you whether the *right document* was retrieved. But a document can be retrieved and still miss the specific paragraph that actually answers the question — especially with structured or legal text where the relevant content is scattered across sections.\n\n**NeedleCoverage@K** checks whether specific text spans (\"needles\") required to answer a query are present in the retrieved chunks. Just add `needles` to your queries file:\n\n```json\n{\n  \"queries\": [{\n    \"id\": \"gdpr_fines\",\n    \"text\": \"What fines can be imposed under the GDPR?\",\n    \"relevant_docs\": [\"gdpr.txt\"],\n    \"needles\": [\n      {\"text\": \"up to 20 000 000 EUR\", \"source\": \"Art 83(5)\"},\n      {\"text\": \"up to 10 000 000 EUR\", \"source\": \"Art 83(4)\"}\n    ]\n  }]\n}\n```\n\n```bash\nragtune simulate --collection prod --queries needles.json --embedder ollama\n```\n\n```\n  Recall@5:          1.000    # Right doc? Yes, always.\n  NeedleCoverage@5:  0.280    # Right content? Only 28% of the time.\n```\n\nNo new flags needed — if your queries have `needles`, the metric appears automatically. Queries without needles work exactly as before.\n\n---\n\n## Commands\n\n| Command | Purpose |\n|---------|---------|\n| `ingest` | Load documents into vector store |\n| `explain` | Debug retrieval for a single query |\n| `simulate` | Batch benchmark with metrics, needle coverage + CI mode |\n| `compare` | Compare embedders or chunk sizes |\n| `audit` | Quick health check (pass/fail) |\n| `report` | Generate markdown reports |\n| `import-queries` | Import queries from CSV/JSON |\n\nSee [CLI Reference](docs/cli-reference.md) for all flags and options.\n\n---\n\n## CI/CD Quality Gates\n\n```yaml\n# .github/workflows/rag-quality.yml\n- name: RAG Quality Gate\n  run: |\n    ragtune ingest ./docs --collection ci-test --embedder ollama\n    ragtune simulate --collection ci-test --queries tests/golden-queries.json \\\n      --ci --min-recall 0.85 --min-coverage 0.90 --max-latency-p95 500\n```\n\nExit code 1 if thresholds fail. See [examples/github-actions.yml](examples/github-actions.yml) for complete setup.\n\n### Regression Testing\n\nCompare against a baseline to catch regressions before they reach production:\n\n```bash\n# Compare current run against baseline\nragtune simulate --collection prod --queries golden.json \\\n  --baseline runs/baseline.json --fail-on-regression\n```\n\nOutput shows deltas for each metric:\n\n```\nBASELINE COMPARISON\nComparing against: 2026-01-15T12:00:00Z\n─────────────────────────────────────────────────────────────\n  Recall@5:    0.900 → 0.850  ↓ 5.6%  (REGRESSED)\n  MRR:         0.800 → 0.820  ↑ 2.5%  (improved)\n  Coverage:    0.950 → 0.950  = 0.0%  (unchanged)\n  Latency p95: 100ms → 120ms  ↑ 20.0%  (REGRESSED)\n─────────────────────────────────────────────────────────────\n\n❌ REGRESSION DETECTED\n   The following metrics decreased: [Recall@5, Latency p95]\n```\n\n---\n\n## Why RagTune?\n\nRAG retrieval is a configuration problem: chunk size, embedding model, index type, top-k. Most teams tune by intuition. RagTune provides the measurement layer to make these decisions empirically, using standard IR metrics (Recall@k, MRR, NDCG) on your actual data.\n\n| What Matters | Impact |\n|--------------|--------|\n| Domain-appropriate chunking | 7%+ recall difference |\n| Embedding model choice | 5% difference |\n| Continuous monitoring | Catches data drift before users do |\n\n### RagTune vs. Other Tools\n\nRagTune focuses on **retrieval debugging, monitoring, and benchmarking**, not end-to-end answer evaluation.\n\n| | RagTune | Ragas / DeepEval | misbahsy/RAGTune |\n|---|---------|------------------|------------------|\n| **Focus** | Retrieval layer | Full pipeline | Full pipeline |\n| **LLM calls** | None required | Required | Required |\n| **Interface** | CLI (CI/CD-native) | Python library | Streamlit UI |\n| **Speed** | Fast (embedding only) | Slow (LLM inference) | Slow |\n| **CI/CD** | First-class | Manual setup | None |\n\n**Use RagTune when:** debugging retrieval, CI/CD quality gates, comparing embedders, deterministic benchmarks.\n\n**Use other tools when:** evaluating LLM answer quality, you need `answer_relevancy` metrics.\n\n---\n\n## Signs You Need This\n\nRetrieval failures are **silent**. No error, no exception. Just gradually worse answers.\n\n- Users complaining about \"wrong answers\" but you can't reproduce it\n- No idea if that embedding change made things better or worse\n- Retrieval was \"good\" in dev, failing in production\n- You added documents but answers got *worse*\n- Can't tell if the LLM is hallucinating or retrieval is broken\n\nIf any of these sound familiar:\n\n```bash\nragtune explain \"the query that's failing\" --collection prod\n```\n\n---\n\n## Installation\n\n```bash\n# Homebrew (macOS/Linux)\nbrew install metawake/tap/ragtune\n\n# Go Install\ngo install github.com/metawake/ragtune/cmd/ragtune@latest\n\n# Or download binary from GitHub Releases\n```\n\n**Prerequisites:** Docker (for Qdrant), Ollama or API key for embeddings.\n\n---\n\n## Embedders\n\n| Embedder | Setup | Best For |\n|----------|-------|----------|\n| `ollama` | Local, no API key | Development, privacy |\n| `openai` | `OPENAI_API_KEY` | General purpose |\n| `voyage` | `VOYAGE_API_KEY` | Legal, code (domain-tuned) |\n| `cohere` | `COHERE_API_KEY` | Multilingual |\n| `tei` | Docker container | High throughput |\n\n## Vector Stores\n\n| Store | Setup |\n|-------|-------|\n| Qdrant (default) | `docker run -p 6333:6333 qdrant/qdrant` |\n| pgvector | `--store pgvector --pgvector-url postgres://...` |\n| Weaviate | `--store weaviate --weaviate-host localhost:8080` |\n| Chroma | `--store chroma --chroma-url http://localhost:8000` |\n| Pinecone | `--store pinecone --pinecone-host HOST` |\n\n---\n\n## Included Benchmarks\n\n| Dataset | Documents | Purpose |\n|---------|-----------|---------|\n| `data/` | 9 | Quick testing |\n| `benchmarks/hotpotqa-1k/` | 398 | General knowledge |\n| `benchmarks/casehold-500/` | 500 | Legal domain |\n| `benchmarks/synthetic-50k/` | 50,000 | Scale testing |\n\n```bash\n# Try it\nragtune ingest ./benchmarks/hotpotqa-1k/corpus --collection demo --embedder ollama\nragtune simulate --collection demo --queries ./benchmarks/hotpotqa-1k/queries.json\n```\n\n---\n\n## Documentation\n\n| Guide | Description |\n|-------|-------------|\n| [Concepts](docs/concepts.md) | RAG basics, metrics explained |\n| [CLI Reference](docs/cli-reference.md) | All commands and flags |\n| [Quickstart](docs/articles/00-quickstart.md) | Step-by-step setup guide |\n| [Benchmarking Guide](docs/articles/03-benchmarking-guide.md) | Scale testing, runtimes |\n| [Deployment Patterns](docs/articles/04-deployment-patterns.md) | CI/CD, production |\n| [FAQ](docs/faq.md) | Common questions |\n| [Troubleshooting](docs/troubleshooting.md) | Common issues and fixes |\n\n---\n\n## Contributing\n\nContributions welcome. Please open an issue first to discuss significant changes.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetawake%2Fragtune","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetawake%2Fragtune","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetawake%2Fragtune/lists"}