{"id":31559546,"url":"https://github.com/avijitshil/sydney","last_synced_at":"2026-04-11T19:04:08.567Z","repository":{"id":316460516,"uuid":"1063466870","full_name":"AvijitShil/Sydney","owner":"AvijitShil","description":"Sydney – Multimodal Offline AI for medical guidance. CPU-friendly, powered by Gemma 3 1B, Whisper, Glow-TTS \u0026 Granite 47M R2 embeddings. RAG + memory = rapid, accurate, context-aware responses. ","archived":false,"fork":false,"pushed_at":"2025-10-03T08:57:14.000Z","size":65,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T10:33:47.748Z","etag":null,"topics":["agentic-ai","ai-research-and-development","faster-whisper","gen-ai","gradio","gradio-python-llm","health-assistant","langchain","llm","medical-ai","medical-ai-healthcare-platform","medical-chatbot","multimodal-ai","ollama","python","rag","voice-agent","voice-assistant-ai","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AvijitShil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-24T17:02:08.000Z","updated_at":"2025-10-03T08:57:18.000Z","dependencies_parsed_at":"2025-10-03T10:25:11.025Z","dependency_job_id":"536b7654-9e02-4894-a71b-f95cc385d858","html_url":"https://github.com/AvijitShil/Sydney","commit_stats":null,"previous_names":["avijitshil/sydney"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AvijitShil/Sydney","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AvijitShil%2FSydney","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AvijitShil%2FSydney/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AvijitShil%2FSydney/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AvijitShil%2FSydney/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AvijitShil","download_url":"https://codeload.github.com/AvijitShil/Sydney/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AvijitShil%2FSydney/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278399690,"owners_count":25980332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","ai-research-and-development","faster-whisper","gen-ai","gradio","gradio-python-llm","health-assistant","langchain","llm","medical-ai","medical-ai-healthcare-platform","medical-chatbot","multimodal-ai","ollama","python","rag","voice-agent","voice-assistant-ai","whisper"],"created_at":"2025-10-05T01:53:55.788Z","updated_at":"2025-12-30T21:18:50.422Z","avatar_url":"https://github.com/AvijitShil.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🩺 Sydney – The Ultimate Local (Offline) AI Medical Assistant\r\n\r\n![Sydney](https://img.shields.io/badge/status-Alpha-yellow)\r\n\r\n**Sydney** is an **advanced multimodal Local (offline) AI assistant** capable of **understanding text and voice**, delivering **highly accurate, context-aware medical guidance**, and working **fully offline with CPU optimization**. Powered by **Gemma 3 !B, Glow-TTS, Whisper, LangChain LLMs, and the Granite 47M R2 embedding model**, Sydney merges **retrieval-augmented generation (RAG)**, **conversation memory**, and **multimodal interaction** to provide **rapid, reliable responses**.\r\n\r\n⚠️ **Medical Disclaimer:** Sydney is **not a substitute for professional medical advice**. For emergencies, always contact certified healthcare providers.\r\n\r\n---\r\n\r\n## **🚀 Key Highlights**\r\n\r\n* 🧠 **Multimodal**: Supports both **text** and **voice input/output** in the same version.\r\n* 📚 **Offline-first**: Entirely functional without internet once models and embeddings are downloaded.\r\n* ⚡ **CPU-optimized**: Fast inference even on standard CPUs without compromising response quality.\r\n* 🛠 **Context-aware memory**: Stores user queries and AI responses, allowing nuanced conversations.\r\n* 🔍 **Multiple RAG pipelines**: Integrates **local medical documents** and **Granite 47M R2 embeddings** for factual, precise, and comprehensive answers.\r\n* 💡 **Problem-solving powerhouse**: Capable of **complex multi-topic medical reasoning**, combining retrieval and generative capabilities.\r\n* 🎯 **Clean \u0026 structured outputs**: Markdown removal, concise formatting, and speech-ready text.\r\n* 🚨 **Emergency detection**: Flags urgent situations and provides immediate guidance to contact professionals.\r\n\r\n---\r\n\r\n## **🧩 Multimodal \u0026 Offline Architecture**\r\n\r\nSydney is designed as a **unified multimodal AI assistant**:\r\n\r\n```\r\n                +--------------------+\r\n                |   User Input       |\r\n                |  Text / Voice      |\r\n                +---------+----------+\r\n                          |\r\n                  [Whisper Speech-to-Text]\r\n                          |\r\n        +-----------------+-----------------+\r\n        |                                   |\r\n[Memory Manager]                     [RAG Pipelines]\r\n- Tracks last N turns               - Local medical docs\r\n- Maintains context                 - Granite 47M R2 embeddings\r\n- Preserves conversation            - Combines results for reasoning\r\n        |                                   |\r\n        +-----------------+-----------------+\r\n                          |\r\n                   [LLM Processing]\r\n                  - Concise, context-aware\r\n                  - Multi-topic reasoning\r\n                          |\r\n                 [Post-processing \u0026 Cleanup]\r\n                  - Markdown removal\r\n                  - Bullet point formatting\r\n                          |\r\n                  [Glow-TTS Speech Synthesis]\r\n                  - Natural, expressive audio\r\n                          |\r\n                 +---------------------+\r\n                 | Gradio UI / Output  |\r\n                 | Text + Voice        |\r\n                 +---------------------+\r\n```\r\n\r\n---\r\n\r\n## **💾 Memory \u0026 Context Management**\r\n\r\nSydney’s **conversation memory** is **persistent, intelligent, and context-aware**:\r\n\r\n* Stores **queries and AI responses** in `memory.json`.\r\n* Limits memory to **configurable recent turns** (default: 10) for performance.\r\n* Each query is augmented with **recent conversation context** for:\r\n\r\n  * Coherent multi-turn dialogue\r\n  * Avoiding repetition\r\n  * Tailored answers based on user history\r\n\r\n**Example Context-Aware Query:**\r\n\r\n```\r\nRecent context:\r\nUser: I have diabetes and high blood pressure. Can I exercise daily?\r\nAssistant: Light cardio 3-5 times/week, strength training, and regular monitoring of blood sugar levels.\r\n\r\nCurrent query: What dietary changes should I implement alongside exercise?\r\nAssistant: - Low glycemic index foods, high fiber intake...\r\n           - Reduce sodium and processed foods...\r\n           - Maintain protein balance for muscle health...\r\n```\r\n\r\n---\r\n\r\n## **📚 Multiple RAG Pipelines for Robust Knowledge Retrieval**\r\n\r\nSydney leverages **multiple RAG systems** for precise answers:\r\n\r\n1. **Local Medical Knowledge Base RAG**\r\n\r\n   * Curated offline documents on diseases, symptoms, nutrition, and lifestyle.\r\n2. **Granite 47M R2 Embeddings RAG**\r\n\r\n   * Embedding-based vector search for **fast, semantic retrieval** of relevant information.\r\n   * Handles **rare, multi-faceted medical queries** with high accuracy.\r\n\r\n**Benefits:**\r\n\r\n* Multi-topic problem-solving\r\n* Context-enhanced recommendations\r\n* Factually grounded AI responses\r\n\r\n---\r\n\r\n## **💨 Offline \u0026 CPU-Optimized Performance**\r\n\r\nSydney is engineered for **offline, CPU-friendly operation**:\r\n\r\n* **Glow-TTS + Multi-band MelGAN** ensures smooth text-to-speech without GPU.\r\n* **Whisper small/int8** allows fast audio transcription on CPU.\r\n* **Precomputed Granite embeddings** for instant retrieval.\r\n* **Chunking long responses** prevents CPU overload.\r\n* **Async threading** ensures non-blocking, fast UI.\r\n* **Persistent memory caching** reduces repeated computation.\r\n\r\n\u003e Result: **High-speed inference and audio response**, even on standard CPUs, with **zero reliance on cloud services**.\r\n\r\n---\r\n\r\n## **💡 Problem-Solving Capabilities**\r\n\r\nSydney is more than a chatbot—it’s a **problem-solving AI assistant**:\r\n\r\n* **Multi-step reasoning**: Can combine multiple symptoms, conditions, and treatment options.\r\n* **Cross-topic retrieval**: Uses **RAG + memory + LLM** to synthesize complex guidance.\r\n* **Structured outputs**: Provides bullet points, warnings, and key takeaways for clarity.\r\n\r\n---\r\n\r\n## **🌟 Why Sydney Stands Out**\r\n\r\n* **Multimodal**: Single version supports **text and voice** seamlessly.\r\n* **Offline \u0026 CPU-Optimized**: Works anywhere, no internet needed, fast even on standard CPUs.\r\n* **Memory + Multi-RAG**: Maintains **context**, retrieves **accurate information**, solves **complex problems**.\r\n* **Granite 47M R2 Embeddings**: State-of-the-art retrieval model for **high-fidelity, semantic medical reasoning**.\r\n* **Problem-Solving Ready**: Handles **multi-topic queries, stepwise reasoning, and actionable suggestions**.\r\n**Example Use Case:**\r\n\r\n```\r\nUser: I have hypertension and mild kidney issues. Can I exercise, and what should I eat?\r\nAssistant: \r\n- Exercise: Low-impact cardio, yoga, 3-5 times/week\r\n- Diet: Low sodium, moderate protein, avoid processed foods\r\n- Monitor: Blood pressure and kidney function regularly\r\n- Warning: Avoid strenuous exercises that elevate blood pressure rapidly\r\n```\r\n\r\n---\r\n\r\n## **⚙ Installation**\r\n\r\n### **1. Clone Repository**\r\n\r\n```bash\r\ngit clone https://github.com/AvijitShil/Sydney.git\r\ncd Sydney\r\n```\r\n\r\n### **2. Setup Python Environment**\r\n\r\n```bash\r\npython -m venv venv\r\nsource venv/bin/activate   # Linux/Mac\r\nvenv\\Scripts\\activate      # Windows\r\n```\r\n\r\n### **3. Install Dependencies**\r\n\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n\r\n\u003e Or manually if `requirements.txt` is missing:\r\n\r\n```bash\r\npip install torch numpy gradio soundfile coqui-tts faster-whisper langchain_ollama\r\n```\r\n\r\n---\r\n\r\n## **🚀 Usage**\r\n\r\n### **Launch Sydney**\r\n\r\n```bash\r\npython main.py\r\n```\r\n\r\n### **Gradio Interface**\r\n\r\n* **💬 Chat Tab**: Type questions → get **text + audio responses**.\r\n* **🎤 Speak Tab**: Record questions → get **transcribed text + audio response**.\r\n\r\n---\r\n\r\n## **🔧 Customization**\r\n\r\n* **Memory Size**: Adjust `CONFIG[\"max_memory_turns\"]`.\r\n* **TTS Speed / Style**: Change `speed` parameter in `tts_to_file()`.\r\n* **Knowledge Base Expansion**: Add new documents to `CONFIG[\"medical_docs\"]`.\r\n* **Emergency Handling**: Customize `is_medical_emergency()` and `get_emergency_response()`.\r\n\r\n---\r\n\r\n## **📂 Folder Structure**\r\n\r\n```\r\n.\r\n├─ main.py                # Core application\r\n├─ rag_system.py          # RAG \u0026 emergency handling\r\n├─ requirements.txt       # Dependencies\r\n├─ memory.json            # Persistent conversation memory\r\n└─ README.md\r\n```\r\n\r\n---\r\n\r\n## **💡 Contributing**\r\n\r\n* Fork \u0026 PR for improvements:\r\n\r\n  * Fine-tune TTS or LLM for voice/naturalness\r\n  * Add medical knowledge or RAG sources\r\n  * Enhance offline performance\r\n* Ensure **no real medical advice is hard-coded**.\r\n\r\n---\r\n\r\n## **📜 License**\r\n\r\nMIT License – see [LICENSE](LICENSE) for details.\r\n\r\n\r\n### ⚕️ **IMPORTANT MEDICAL DISCLAIMER**\r\n\r\nThis AI assistant is for **educational and informational purposes only**.\r\nIt cannot and should not replace professional medical advice, diagnosis, or treatment.\r\n**Always consult qualified healthcare professionals for medical concerns.**\r\n\r\n---\r\n\r\n### 🚀 Features\r\n\r\n* **🎤 Speech-to-Text (STT)**: Uses **Faster-Whisper** for accurate local transcription\r\n* **🧠 LLM Processing**: Runs **Gemma 1B** locally via Ollama\r\n* **🔊 Text-to-Speech (TTS)**: **GlowTTS** for natural, expressive voice output\r\n* **📚 Retrieval-Augmented Generation (RAG)**: Knowledge base with vector embeddings\r\n* **💭 Memory Management**: Maintains conversation context across turns\r\n* **🚨 Emergency Detection**: Detects critical medical keywords automatically\r\n* **⚡ Optimized Performance**: Async processing + model caching\r\n\r\n---\r\n\r\n### 📋 Requirements\r\n\r\n* **Python**: 3.8 or higher\r\n* **RAM**: 8GB minimum (16GB recommended)\r\n* **Storage**: \\~5GB free space for models\r\n* **Dependencies**:\r\n\r\n  * PyTorch (CPU or CUDA if GPU available)\r\n  * Ollama (for hosting LLMs)\r\n  * Audio drivers (microphone \u0026 speaker support)\r\n\r\n---\r\n\r\n### 📁 Project Structure\r\n\r\n```\r\nsydney/\r\n├── main.py                 # Core application\r\n├── rag_system.py           # RAG implementation\r\n├── config.py               # Configuration management\r\n├── requirements.txt        # Dependencies\r\n├── memory.json             # Conversation memory\r\n├── response.json           # Response cache\r\n└── medical_knowledge_db/   # Vector database (auto-created)\r\n```\r\n\r\n---\r\n\r\n### 🔒 Privacy \u0026 Safety\r\n\r\n* **100% Local Processing**: No internet or cloud dependency\r\n* **Conversation Memory**: Stored locally in JSON (can be encrypted)\r\n* **Emergency Safety**: Detects medical emergencies and prompts urgent actions\r\n* **Medical Disclaimers**: Auto-injected into sensitive health responses\r\n\r\n---\r\n\r\n### 🚀 Performance Optimizations\r\n\r\n* **Async Processing**: Non-blocking STT, LLM, and TTS pipeline\r\n* **Model Caching**: Preloads models for faster response\r\n* **Memory Management**: Maintains rolling context window\r\n* **Vector Search**: Efficient retrieval with embeddings\r\n\r\n---\r\n\r\n### 🛡️ Safety Features\r\n\r\n1. **Emergency Detection**: Auto flagging of life-threatening terms\r\n2. **Medical Disclaimers**: Always included in health-related answers\r\n3. **Graceful Fallbacks**: Handles low-resource environments (CPU-only, int8 quantization)\r\n4. **Local-Only**: Keeps sensitive queries private\r\n\r\n---\r\n\r\n\r\n\r\n\r\n\r\n\r\nSydney is **not just an assistant—it’s an offline, multimodal AI medical companion**, capable of **massive-scale problem solving** while preserving **privacy, speed, and accuracy**.\r\n\r\n---\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favijitshil%2Fsydney","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Favijitshil%2Fsydney","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favijitshil%2Fsydney/lists"}