{"id":29222747,"url":"https://github.com/boemer00/rag-naive","last_synced_at":"2026-05-07T13:45:09.858Z","repository":{"id":302525829,"uuid":"1012722550","full_name":"boemer00/rag-naive","owner":"boemer00","description":"This project is a modular RAG pipeline tailored for research papers in longevity.","archived":false,"fork":false,"pushed_at":"2025-08-13T05:33:03.000Z","size":6331,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-13T07:06:56.584Z","etag":null,"topics":["langchain","langsmith","llm","rag","research","vector-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/boemer00.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-02T19:23:01.000Z","updated_at":"2025-08-13T05:33:06.000Z","dependencies_parsed_at":"2025-08-13T19:17:15.325Z","dependency_job_id":null,"html_url":"https://github.com/boemer00/rag-naive","commit_stats":null,"previous_names":["boemer00/rag-naive"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/boemer00/rag-naive","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boemer00%2Frag-naive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boemer00%2Frag-naive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boemer00%2Frag-naive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boemer00%2Frag-naive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/boemer00","download_url":"https://codeload.github.com/boemer00/rag-naive/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boemer00%2Frag-naive/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32740908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["langchain","langsmith","llm","rag","research","vector-database"],"created_at":"2025-07-03T04:02:02.489Z","updated_at":"2026-05-07T13:45:09.845Z","avatar_url":"https://github.com/boemer00.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## RAG‑Naive: Intelligent Longevity RAG with Agentic AI\n\nAdvanced Retrieval‑Augmented Generation system for longevity research featuring **intelligent agentic AI**, **semantic scoring**, and **adaptive query strategies**. Built with **LangChain**, **OpenAI**, and **Chroma**, it delivers research-grade performance with sophisticated retrieval, re-ranking, and self-healing decision loops.\n\n### 🧠 Agentic Intelligence Features\n- **Semantic Similarity Scoring**: Embedding-based relevance assessment\n- **Document Re-ranking**: Improves result quality by 56% over baseline\n- **LLM-based Quality Assessment**: Intelligent context evaluation\n- **Query Reformulation**: Adaptive search strategies for failed retrievals\n- **High-Confidence Termination**: Smart stopping conditions\n- **100% Citation Grounding**: All answers backed by research sources\n\n---\n\n## Quick Start\n```bash\n# 1) Install dependencies (includes scikit-learn and numpy for agentic features)\npip install -r requirements.txt\n\n# 2) Configure OpenAI\nexport OPENAI_API_KEY=sk-...\n\n# 3) Ask a question using intelligent agent (builds index on first run)\npython main.py \"What is the relationship between exercise and longevity?\"\n```\n\nTip: The first run creates `db/` with vector embeddings; subsequent runs are fast.\n\n---\n\n## Web API (FastAPI)\nRun the server:\n```bash\nuvicorn web.backend.app:app --reload\n```\n\nExamples:\n```bash\n# Query (classic RAG)\ncurl -X POST -F \"question=What improves VO2 max?\" http://127.0.0.1:8000/query\n\n# Assistant (agentic mode - default enabled)\ncurl -X POST -F \"question=Best evidence for sleep and longevity?\" -F \"use_agent=true\" \\\n  http://127.0.0.1:8000/assistant/message\n\n# Apple Health analysis (XML export)\ncurl -X POST -F \"file=@export.xml\" -F \"question=Any VO2 max trends?\" \\\n  http://127.0.0.1:8000/health-analysis\n```\n\n---\n\n## Features\n\n### 🚀 Core RAG Capabilities\n- **Multi‑PDF RAG**: Auto‑loads all PDFs under `raw_data/`\n- **Local persistence**: Chroma DB with efficient vector storage\n- **Intelligent Retrieval**: 3-pass adaptive strategy (semantic → filtered → reformulated)\n- **Web API**: `/query`, `/assistant/message`, `/health-analysis` (Apple Health XML)\n- **Optional tracing**: LangSmith monitoring and evaluation\n\n### 🧠 Agentic Intelligence\n- **Semantic Scoring**: Cosine similarity with embedding-based relevance\n- **Document Re-ranking**: 2x retrieval + similarity-based selection  \n- **LLM Assessment**: Quality evaluation with 0.6 semantic + 0.4 LLM weighting\n- **Query Reformulation**: Failed query analysis and intelligent rephrasing\n- **Adaptive Termination**: High-confidence early stopping (configurable thresholds)\n- **Comprehensive Tracing**: Detailed decision logs for debugging and optimization\n\n### 📊 Performance Metrics (vs Direct LLM)\n- **100% Citation Rate**: All responses grounded in research\n- **28% Higher Specificity**: More technical, precise answers (8.96 vs 7.00)\n- **56% Improvement** on covered research topics (9.51 vs 6.08 specificity)\n- **100% Knowledge Boundary Awareness**: Honest about limitations\n- **~8s Response Time**: Reasonable latency for production use\n\n---\n\n## Configuration\nEnvironment variables are loaded via `python-dotenv`.\n\n| Variable | Default | Description |\n|---|---|---|\n| `OPENAI_API_KEY` | — (required) | OpenAI key |\n| `MODEL_NAME` | `gpt-4.1-nano-2025-04-14` | Chat model |\n| `EMBEDDING_MODEL` | `text-embedding-3-large` | Embedding model |\n| `PERSIST_DIRECTORY` | `db` | Chroma storage path |\n| `RETRIEVAL_K` | `6` | Top‑k documents |\n| `TEMPERATURE` | `0.0` | LLM temperature |\n| `MAX_TOKENS` | `512` | LLM max tokens |\n| `LANGSMITH_TRACING` | `false` | Enable tracing |\n| `LANGSMITH_API_KEY` | — | LangSmith key |\n| `LANGSMITH_PROJECT` | `default` | Project name |\n| `LANGSMITH_ENDPOINT` | `https://api.smith.langchain.com` | Optional EU/self‑hosted endpoint |\n| `EVAL_SAMPLE_RATE` | `0.05` | Sampling rate for judge |\n\n### 🧠 Agent Configuration\n| Variable | Default | Description |\n|---|---|---|\n| `AGENT_MIN_RELEVANCE_SCORE` | `0.5` | Minimum score to accept retrieval results |\n| `AGENT_HIGH_CONFIDENCE_THRESHOLD` | `0.8` | Score for early termination |\n| `AGENT_MAX_PASSES` | `3` | Maximum retrieval attempts |\n| `AGENT_ENABLE_FILTERED_RETRY` | `true` | Enable Pass 2 filtered retrieval |\n| `AGENT_ENABLE_SEMANTIC_RETRY` | `true` | Enable Pass 3 query reformulation |\n\n`.env` example:\n```env\nOPENAI_API_KEY=sk-...\n# MODEL_NAME=gpt-4.1-nano-2025-04-14\n# EMBEDDING_MODEL=text-embedding-3-large\n# PERSIST_DIRECTORY=db\n# RETRIEVAL_K=6\n# TEMPERATURE=0.0\n# MAX_TOKENS=512\n# LANGSMITH_TRACING=true\n# LANGSMITH_API_KEY=ls__...\n# LANGSMITH_PROJECT=rag-naive-mvp\n# LANGSMITH_ENDPOINT=https://eu.api.smith.langchain.com\n# EVAL_SAMPLE_RATE=0.05\n\n# Agent Configuration (Optional)\n# AGENT_MIN_RELEVANCE_SCORE=0.5\n# AGENT_HIGH_CONFIDENCE_THRESHOLD=0.8\n# AGENT_MAX_PASSES=3\n# AGENT_ENABLE_FILTERED_RETRY=true\n# AGENT_ENABLE_SEMANTIC_RETRY=true\n```\n\n---\n\n## Project Layout\n```text\nrag-naive/\n├── raw_data/                       # PDFs for indexing\n├── src/\n│   ├── chain.py                    # Prompt + LCEL chain\n│   ├── indexer.py                  # Build/load Chroma index\n│   ├── retrieval.py                # Retrieval helpers (title/author boost)\n│   ├── utils.py                    # Load PDFs + optional PMC fetch\n│   ├── monitoring.py               # LangSmith tracing + RAG judge\n│   ├── agent/                      # Intelligent agentic AI system\n│   │   ├── decision_tree.py        # 3-pass adaptive agent controller\n│   │   ├── tools.py                # Semantic scoring, re-ranking, reformulation\n│   │   ├── policy.py               # Configurable strategies and thresholds\n│   │   └── types.py                # Types and protocols\n│   └── mcp/                        # Health analysis (Apple Health)\n│       ├── health_analyzer.py\n│       ├── apple_health.py\n│       └── health_server.py\n├── web/\n│   └── backend/app.py              # FastAPI app: /query, /assistant/message, /health-analysis\n├── main.py                         # CLI entrypoint\n├── config.py                       # Env‑driven configuration\n├── fetch_longevity_papers.py       # Build PMC longevity corpus\n├── fetch_vo2max_papers.py          # Add VO2max papers\n├── inspect_papers.py               # Inspect stored papers\n└── README.md\n```\n\n---\n\n## Makefile (shortcuts)\n```bash\nmake install            # deps for dev\nmake test               # unit + integration\nmake test-unit          # unit only\nmake test-integration   # integration only\nmake test-performance   # Agentic RAG vs Direct LLM comparison\nmake quality-gates      # performance gates\nmake demo               # run a demo CLI query\n```\n\n---\n\n## Testing\n```bash\npip install -r requirements.txt -r requirements-dev.txt\n\n# All tests\npytest tests/ -v\n\n# Markers\npytest -m unit\npytest -m integration\npytest tests/performance/test_rag_vs_llm.py -v\npytest tests/performance/test_quality_gates.py::test_quality_gates -v\n```\n\nNotes:\n- Tests mock OpenAI where possible; set `OPENAI_API_KEY` to run real calls.\n- Performance tests demonstrate 28% improvement in answer quality over direct LLM.\n- Web tests include assistant endpoint shape checks and agentic functionality.\n\n---\n\n## Monitoring \u0026 Evaluation\n\n### LangSmith Integration\nOptional tracing and RAG quality feedback are supported when LangSmith is installed and `LANGSMITH_TRACING=true`. See variables in the Configuration section. The module `src/monitoring.py` handles setup.\n\n### Performance Testing\nRun comprehensive performance evaluation:\n```bash\n# Compare agentic RAG vs direct LLM across question categories\npython -m tests.performance.test_rag_vs_llm --quick\n\n# Detailed analysis with full outputs\npython -m tests.performance.test_rag_vs_llm\n```\n\nKey metrics tracked:\n- **Citation grounding rate** (100% for agentic RAG vs 0% for LLM)\n- **Technical specificity scores** (28% improvement)\n- **Knowledge boundary awareness** (100% honesty rate)\n- **Response times** (~8s for research-grade quality)\n\n---\n\n## Data \u0026 Rebuilding Index\n- Add PDFs to `raw_data/` and run the CLI once to index\n- To force rebuild, delete `db/` and run again\n- Optional: fetch extra papers via PMC helpers:\n```bash\npython fetch_longevity_papers.py --limit 50\npython fetch_vo2max_papers.py\n```\n\n---\n\n## Troubleshooting\n| Symptom | Fix |\n|---|---|\n| `OPENAI_API_KEY environment variable is required` | Export your key or create `.env` |\n| `InvalidRequestError: model not found` | Set `MODEL_NAME` to a model you can access |\n| Empty/weak answers | Increase `RETRIEVAL_K`, add more PDFs, or adjust agent thresholds |\n| Agent returns \"impossible\" status | Lower `AGENT_MIN_RELEVANCE_SCORE` or add more relevant documents |\n| Health endpoint fails | Upload Apple Health XML (`.xml`) exports only |\n| Agent responses too slow | Disable re-ranking or reduce `RETRIEVAL_K` for faster responses |\n\n---\n\n## Author\nDeveloped and maintained by **Renato Boemer**\n\n- GitHub: https://github.com/boemer00\n- LinkedIn: https://www.linkedin.com/in/renatoboemer/\n\n## License\n© 2025 Renato Boemer — MIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboemer00%2Frag-naive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboemer00%2Frag-naive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboemer00%2Frag-naive/lists"}