{"id":30524753,"url":"https://github.com/ashebbar-dev/ojas_eb","last_synced_at":"2026-04-15T10:35:06.943Z","repository":{"id":294209443,"uuid":"985859213","full_name":"ashebbar-dev/Ojas_EB","owner":"ashebbar-dev","description":"Multi-path RAG agent designed to provide support to dementia caregivers.","archived":false,"fork":false,"pushed_at":"2025-08-25T15:46:05.000Z","size":428,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-25T17:41:29.712Z","etag":null,"topics":["agent","chatbot","cohere","dementia-care","flask","langchain","llm","pgvector","python","rag","reranking","retrieval-augmented-generation","supabase","voyage-ai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashebbar-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-18T17:09:53.000Z","updated_at":"2025-08-25T15:46:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"0efe8dea-5f95-4c11-8b5a-edb20ee337c9","html_url":"https://github.com/ashebbar-dev/Ojas_EB","commit_stats":null,"previous_names":["enigmatulipgarde00n/ojas_eb","ashebbar-dev/ojas_eb"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ashebbar-dev/Ojas_EB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashebbar-dev%2FOjas_EB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashebbar-dev%2FOjas_EB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashebbar-dev%2FOjas_EB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashebbar-dev%2FOjas_EB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashebbar-dev","download_url":"https://codeload.github.com/ashebbar-dev/Ojas_EB/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashebbar-dev%2FOjas_EB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272255803,"owners_count":24901329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-26T02:00:07.904Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","chatbot","cohere","dementia-care","flask","langchain","llm","pgvector","python","rag","reranking","retrieval-augmented-generation","supabase","voyage-ai"],"created_at":"2025-08-26T21:01:54.255Z","updated_at":"2026-04-15T10:35:06.909Z","avatar_url":"https://github.com/ashebbar-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OJAS -  Advanced RAG Agent for Dementia Caregivers\n![alt text](image-1.png)\n\nThis repository contains the complete source code and documentation for a state-of-the-art, multi-path Retrieval-Augmented Generation (RAG) agent. It is designed to function as an empathetic, accurate, and verifiable AI assistant for caregivers of people with dementia, using relaiable knowledge source.\n\nThis is not a simple RAG implementation. It is an advanced, multi-stage system designed to overcome common RAG pitfalls by employing query deconstruction, parallel-path retrieval, state-of-the-art reranking, and a robust, streaming-first web interface.\n\n---\n\n## 🌟 Core Features \u0026 Technical Highlights\n![alt text](image-2.png)\n\n- **Sophisticated Agentic Workflow:** Implements a \"Decompose -\u003e Retrieve -\u003e Synthesize\" pipeline, allowing the agent to break down complex, multi-intent user queries into focused sub-queries before retrieving information.\n- **Novel Parallel-Path Retrieval:** The core of the system is a custom retrieval tool that executes two search strategies in parallel for each sub-query:\n    1.  **Broad Hybrid Search:** A \"wide net\" combining semantic (vector) and lexical (keyword) search to maximize recall across the entire knowledge base.\n    2.  **Title-First Entity Search:** A \"sniper rifle\" that first identifies the most relevant articles by title and then retrieves their full context, maximizing precision.\n- **State-of-the-Art Reranking:** Utilizes Cohere's `rerank-english-v3.0` model as a final, crucial quality gate to re-order candidate chunks based on true contextual relevance, significantly improving the signal-to-noise ratio of the retrieved context.\n- **Specialized \u0026 Robust Data Ingestion:** The data pipeline is not a one-size-fits-all solution. It uses multiple, specialized scripts tailored to handle the inconsistent HTML structures found across different sections of the source website (e.g., informational pages vs. blog posts).\n- **Automated Content Curation \u0026 QC:** The pipeline automatically filters content by category (e.g., only \"Advice,\" \"Research\") and flags pages with abnormal structures for manual review, ensuring a high-quality, relevant knowledge base.\n- **Production-Grade Backend \u0026 UI:** A non-blocking Flask backend serves a real-time streaming API using Server-Sent Events (SSE). The vanilla JS/Tailwind CSS frontend consumes this stream, providing an immediate, token-by-token response for an excellent user experience.\n- **Secure \u0026 Performant Database:** Built on Supabase (PostgreSQL with `pgvector`), the database schema is fully indexed for vector search (`ivfflat`), full-text search (`gin`), and metadata filtering (`btree`), ensuring high performance at scale.\n\n---\n\n## 🏗️ System Architecture Deep Dive\n\nThe agent operates on a sophisticated pipeline designed for maximum reliability and answer quality.\n\n1.  **User Interface (`ui/index.html`):** A user submits a query through the web interface.\n2.  **API Server (`agent/server.py`):** The Flask server receives the query at the `/ask_stream` endpoint. It dispatches the main agent task to a background thread to keep the server responsive.\n3.  **Decomposition (`agent/chatbot_agent_claw4.py`):** The agent's first action is a dedicated LLM call. It uses a \"Strategist\" prompt to analyze the user's query and break it down into a list of focused, self-contained sub-queries.\n    - *Example:* \"How do I handle my dad's wandering and eating problems?\" -\u003e `[\"managing wandering behavior in dementia\", \"addressing eating problems in dementia\"]`\n4.  **Parallel Retrieval (`agent/chatbot_agent_claw4.py`):** The agent makes a single call to the `parallel_comprehensive_search` tool, passing the entire list of sub-queries.\n    - This tool uses a `ThreadPoolExecutor` to run a `execute_dual_track_search` worker for each sub-query simultaneously.\n    - Each worker executes two database searches in parallel: `simple_hybrid_search` and `title_filtered_search`.\n    - The results from both paths are combined, de-duplicated, and put through Cohere Reranking.\n5.  **Meta-Reranking \u0026 Context Assembly:** The results from all parallel workers are collected. A final reranking pass is performed on this entire pool of context to find the most relevant chunks related to the user's *original* query.\n6.  **Synthesis \u0026 Streaming:** The final, high-quality context is fed into a dedicated \"Synthesizer\" LLM call. This LLM's only job is to write a comprehensive, empathetic answer. As it generates tokens, a custom `SSECallbackHandler` puts them into a queue, which the Flask server streams directly to the user's browser.\n\n---\n\n## 🛠️ Setup and Installation Guide\n\nFollow these steps meticulously to set up the project environment and run the application.\n\n### 1. Prerequisites\n\n- Python 3.10 or newer.\n- An account on [Supabase](https://supabase.com) to create a new PostgreSQL project.\n- API keys from the following services:\n    - [Voyage AI](https://www.voyageai.com/) (for embeddings)\n    - [Cohere](https://cohere.com/) (for reranking)\n    - [OpenRouter](https://openrouter.ai/) or [Groq](https://groq.com/) (for LLM access)\n\n### 2. Environment Setup\n\n**Clone the Repository:**\n```bash\ngit clone https://github.com/enigmatulipgarde00n/Ojas_EB.git\ncd Ojas_EB\n```\n\n**Create and Activate a Virtual Environment (Highly Recommended):**\nA virtual environment isolates your project's dependencies from your system's global Python installation.\n\n```bash\n# Create the virtual environment (named 'venv')\npython -m venv venv\n\n# Activate it\n# On Windows:\nvenv\\Scripts\\activate\n# On macOS/Linux:\nsource venv/bin/activate\n```\nYou will see `(venv)` at the beginning of your terminal prompt, indicating it's active.\n\n**Install Python Dependencies:**\nInstall all required libraries from the `requirements.txt` file.\n\n```bash\npip install -r requirements.txt\n```\n\n**Set Up Environment Variables:**\nThis project uses a `.env` file to manage secret API keys.\n\n1.  Copy the example file:\n    ```bash\n    cp .env.example .env\n    ```\n2.  Open the newly created `.env` file in a text editor.\n3.  Fill in the placeholder values with your actual keys from Supabase, Voyage, Cohere, and your chosen LLM provider.\n\n### 3. Database \u0026 Data Ingestion Pipeline\n\nThis multi-step process populates your Supabase database with the knowledge base. Run these steps in order.\n\n**Step 3.1: Initialize the Database Schema**\n- **Action:** Go to your Supabase project dashboard.\n- **Navigate:** Find the \"SQL Editor\" in the left-hand menu.\n- **Execute:** Open the `database/schema.sql` file from this repository, copy its entire content, paste it into the SQL Editor, and click \"RUN\".\n- **Purpose:** This one-time setup creates the `dementia_chunks` table, all necessary performance indexes (`ivfflat`, `gin`), and the three custom SQL functions (`simple_hybrid_search`, `title_filtered_search`, `find_relevant_pages`) required for advanced retrieval.\n\n**Step 3.2: Crawl and Chunk the Content**\nThe data is gathered using two specialized scripts to handle the website's different layouts.\n\n- **Action 1: Crawl Informational Pages**\n    ```bash\n    python ingestion/trial_nollm_crawl1.py \"https://www.alzheimers.org.uk/sitemap.xml\" \"https://www.alzheimers.org.uk/about-dementia\" -o about_dementia.json\n    ```\n    - **Purpose:** This script specifically targets the main informational sections. It uses a simpler chunking strategy based on `\u003ch2\u003e` tags, which is optimal for these pages. It outputs `about_dementia.json`.\n\n- **Action 2: Crawl and Curate Blog Pages**\n    ```bash\n    python ingestion/curated_chunker_with_log1.py \"https://www.alzheimers.org.uk/sitemap.xml\" \"https://www.alzheimers.org.uk/blog\" -o blog_posts.json\n    ```\n    - **Purpose:** This script targets the blog. It first verifies each article belongs to an approved category (e.g., \"Advice\") and then uses a different chunking strategy optimized for the blog's HTML structure. It also logs any pages with abnormal chunk counts to `outliers_for_review.csv` for quality control. It outputs `blog_posts.json`.\n\n**Step 3.3: Refine and Prepare the Final Dataset**\n- **Action 1: Combine Files:** Manually or programmatically merge the contents of `about_dementia.json` and `blog_posts.json` into a single file named `knowledge_base.json`.\n- **Action 2: Refine Chunks**\n    ```bash\n    python ingestion/refine_chunks.py knowledge_base.json -o final_knowledge_base.json\n    ```\n    - **Purpose:** This critical script takes the structurally-chunked data and processes it for optimal LLM performance. It uses `tiktoken` to ensure every chunk is within a target token range (e.g., 300-600 tokens) by intelligently merging small chunks and splitting large ones.\n\n**Step 3.4: Embed and Upload to Supabase**\n- **Action:**\n    ```bash\n    python ingestion/embed_and_upload_idempotent.py final_knowledge_base.json\n    ```\n    - **Purpose:** This is the final ingestion step. The script is **idempotent**, meaning it first deletes all existing data from the `dementia_chunks` table to prevent duplicates. It then reads `final_knowledge_base.json`, generates embeddings for each chunk using the Voyage AI API, and uploads the content, metadata, and embeddings to your Supabase database in efficient batches.\n\nYour knowledge base is now live and ready for querying.\n\n### 4. Running the Chatbot Application\n\nWith the data pipeline complete, you can now start the application.\n\n**Step 4.1: Start the Backend Server**\n- **Action:**\n    ```bash\n    python agent/server.py\n    ```\n    - **Purpose:** This starts the Flask web server on `http://127.0.0.1:5000`. It listens for API requests from the frontend, manages chat sessions, and orchestrates the agent's work in background threads.\n\n**Step 4.2: Open the User Interface**\n- **Action:** Open the `ui/index.html` file in your web browser. You don't need to serve it; you can open it directly from your file system.\n- **Interact:** The UI will connect to your local Flask server. You can now start a new chat and ask questions. Watch your terminal where the server is running to see the agent's detailed reasoning trace in real-time.\n\n---\n\n## 📜 License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for more details.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashebbar-dev%2Fojas_eb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashebbar-dev%2Fojas_eb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashebbar-dev%2Fojas_eb/lists"}