{"id":28411448,"url":"https://github.com/evangks/vision-voice-multimodal-app","last_synced_at":"2026-05-02T05:43:44.789Z","repository":{"id":295307810,"uuid":"989750946","full_name":"EvanGks/vision-voice-multimodal-app","owner":"EvanGks","description":"An AI-powered chatbot that combines image understanding, voice input, and multilingual speech output. Users can upload images, ask questions by voice, and receive intelligent spoken answers. Showcases state-of-the-art models (Gemini, Whisper, Kokoro TTS) and robust Python engineering.","archived":false,"fork":false,"pushed_at":"2025-06-10T07:49:40.000Z","size":229,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-10T08:35:51.966Z","etag":null,"topics":["ai-chatbot","computer-vision","deep-learning","flask-application","full-stack","gemini","gradio","kokoro","machine-learning","multimodal","nlp","portfolio","python","stt","tts","whisper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EvanGks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-24T18:46:41.000Z","updated_at":"2025-06-10T07:49:43.000Z","dependencies_parsed_at":"2025-05-24T20:29:24.171Z","dependency_job_id":"c39e1405-1d67-410b-bb25-4c055d83357a","html_url":"https://github.com/EvanGks/vision-voice-multimodal-app","commit_stats":null,"previous_names":["evangks/vision-voice-multimodal-app"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/EvanGks/vision-voice-multimodal-app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanGks%2Fvision-voice-multimodal-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanGks%2Fvision-voice-multimodal-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanGks%2Fvision-voice-multimodal-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanGks%2Fvision-voice-multimodal-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EvanGks","download_url":"https://codeload.github.com/EvanGks/vision-voice-multimodal-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EvanGks%2Fvision-voice-multimodal-app/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261575873,"owners_count":23179585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-chatbot","computer-vision","deep-learning","flask-application","full-stack","gemini","gradio","kokoro","machine-learning","multimodal","nlp","portfolio","python","stt","tts","whisper"],"created_at":"2025-06-02T16:00:41.023Z","updated_at":"2026-05-02T05:43:44.750Z","avatar_url":"https://github.com/EvanGks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧠 Multimodal AI Visual Assistant\n\nA full-stack, production-grade multimodal AI assistant that enables users to interact with images using natural language and voice. Get instant, multilingual spoken answers powered by state-of-the-art AI models.\n\n## 🚀 Features\n\n- **Image Upload \u0026 Analysis**: Ask questions about any image using your voice.\n- **Voice Input \u0026 Transcription**: Record queries, transcribed in real-time with OpenAI Whisper.\n- **Advanced Multimodal Reasoning**: Google Gemini 2.5 Flash analyzes images and queries for detailed, factual answers.\n- **Multilingual Voice Output**: Hear responses in your chosen language, accent, and gender using Kokoro TTS.\n- **Voice Customization**: Select from dozens of voices (male/female), languages, and adjust speech rate.\n- **Accessible, Responsive UI**: Gradio interface designed for keyboard navigation, screen readers, and all devices.\n- **Automated Asset Management**: All TTS models and voices are auto-downloaded on first run.\n- **Robust Error Handling**: User-friendly error messages and backend logging.\n- **Comprehensive Testing**: Pytest suite covers backend, frontend, and model logic.\n\n## 📸 Screenshots\n\nBelow are some screenshots of the application in action. (Add your images to the `assets/` directory and reference them here.)\n\n![Main UI](assets/UI_demo.png)\n![Example](assets/example_demo.png)\n\n## 🏗️ Architecture\n\n```\nUser\n  │\n  ▼\nGradio UI (app/frontend/gradio_app.py)\n  │\n  ▼\nFlask API (app/backend/services/flask_app.py)\n  │\n  ▼\nModelManager (app/backend/utils/model_manager.py)\n  ├─ Whisper (Speech-to-Text)\n  ├─ Gemini (Image+Text Reasoning)\n  └─ Kokoro TTS (Text-to-Speech)\n```\n\n- **Concurrent Backend \u0026 Frontend**: Both run together via `main.py` (threaded).\n- **All assets and models are managed automatically.**\n\n## 🌍 Supported Languages \u0026 Voices\n\n- **English** (American, British,): Multiple male/female voices\n- **Japanese**: Male/female voices\n- **Mandarin Chinese**: Multiple voices\n- **French**: Female voice\n- **Spanish**: Male/female voices\n- **Italian**: Male/female voices\n- **Brazilian Portuguese**: Male/female voices\n- **Hindi**: Male/female voices\n\nSee [`app/backend/utils/kokoro_voices.py`](app/backend/utils/kokoro_voices.py) for the full list.\n\n## 🛠️ Installation\n\n1. **Clone the repository:**\n   ```bash\n   git clone https://github.com/EvanGks/vision-voice-multimodal-app.git\n   cd multimodal-ai\n   ```\n\n2. **Create and activate a virtual environment:**\n   ```bash\n   python -m venv .venv\n   .venv\\Scripts\\activate  # On Windows\n   source .venv/bin/activate  # On macOS/Linux\n   ```\n\n3. **Install dependencies:**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. **Configure environment variables:**\n   - Copy `.env.example` to `.env` and fill in your Google API key and Flask secret.\n   - **All configuration variables (API keys, model names, upload folders, etc.) are loaded exclusively from the `.env` file in the root directory.**\n   - The application does **not** read configuration from system environment variables—**all configuration must be set in `.env`**.\n   - All model assets will be auto-downloaded.\n\n5. **Run the application:**\n   ```bash\n   python main.py\n   ```\n   - Access the UI at [http://localhost:7860](http://localhost:7860)\n\n## 🗂️ Project Structure\n\n```\nMultimodal_AI/\n├── app/\n│   ├── backend/\n│   │   ├── services/           # Flask API endpoints\n│   │   ├── utils/              # Model management, voice metadata, text utils\n│   │   └── kokoro_assets/      # TTS model assets (auto-downloaded)\n│   ├── frontend/               # Gradio UI implementation\n│   └── uploads/                # Uploaded files (auto-cleaned)\n├── assets/                     # Screenshots and static assets\n├── tests/                      # Test suite (API, UI, models)\n├── .env, .env.example          # Environment variables\n├── .gitignore                  # Git ignore file\n├── LICENSE                     # MIT License\n├── main.py                     # Application entry point\n├── README.md                   # Project documentation\n├── requirements.txt            # Python dependencies\n```\n\n- All code is modular and organized for clarity and extensibility.\n- The `assets/` directory is the recommended place for screenshots and static files.\n\n## 🧪 Testing\n\nRun all tests with:\n```bash\npytest\n```\n- Tests cover API endpoints, UI workflow, and model logic.\n- Test data and assets are auto-managed and cleaned up.\n\n## ♿ Accessibility \u0026 UX\n\n- **Keyboard navigation** and **screen reader** support.\n- **High color contrast** and **responsive design**.\n- **Clear feedback** for all user actions and errors.\n- **Semantic HTML** and ARIA attributes for assistive technologies.\n- **Resizable text** and mobile-friendly layout.\n\n## 🔒 Security\n\n- **No secrets in code**—all credentials via `.env`.\n- **All configuration is managed via the `.env` file only. System environment variables are not used for configuration.**\n- **Strict file upload validation** and privacy cleanup.\n- **Sensitive/model files are git-ignored.**\n- **API key management** and environment-based configuration.\n\n## 🧩 Extensibility\n\n- Add new voices/languages by updating `kokoro_voices.py`.\n- Swap or extend models via `ModelManager` and configure them in `.env`.\n- **All configuration is centralized in `.env` for easy reproducibility and sharing.**\n- Modular, testable codebase for rapid prototyping.\n\n## ✨ Why This Project Stands Out\n\n- **End-to-end AI workflow**: From voice to vision to speech, all in one app.\n- **Production best practices**: Security, error handling, accessibility, and testing.\n- **Portfolio-ready**: Demonstrates full-stack AI, modern Python, and real-world deployment skills.\n- **Comprehensive documentation and code comments** for maintainability.\n- **Automated asset management** for seamless setup.\n\n## 🚀 Future Enhancements\n\n- Add support for additional languages and regional accents in TTS and STT.\n- Implement real-time streaming responses for faster feedback.\n- Develop a mobile-friendly or native mobile UI.\n- Integrate more advanced image analysis models (e.g., OCR, object detection).\n- Add user authentication and personalized settings.\n- Enable cloud deployment with scalable infrastructure.\n- Provide downloadable audio/text transcripts for user queries.\n- Add progress indicators and better feedback for long-running operations.\n- Expand accessibility features (e.g., high-contrast mode, localization).\n\n## 📄 License\n\nThis project is licensed under the MIT License (c) 2025 Evan GKS. See the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- [OpenAI Whisper](https://huggingface.co/openai/whisper-tiny)\n- [Google Gemini](https://ai.google.dev/)\n- [Kokoro TTS](https://github.com/hexgrad/Kokoro)\n- [Gradio](https://gradio.app/)\n- [Flask](https://flask.palletsprojects.com/)\n\n## 📬 Contact\nFor questions or feedback, please reach out via:\n\n- **GitHub:** [EvanGks](https://github.com/EvanGks)\n- **X (Twitter):** [@Evan6471133782](https://x.com/Evan6471133782)\n- **LinkedIn:** [Evangelos Gakias](https://www.linkedin.com/in/evangelos-gakias-346a9072)\n- **Kaggle:** [evangelosgakias](https://www.kaggle.com/evangelosgakias)\n- **Email:** [evangks88@gmail.com](mailto:evangks88@gmail.com)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevangks%2Fvision-voice-multimodal-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevangks%2Fvision-voice-multimodal-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevangks%2Fvision-voice-multimodal-app/lists"}