{"id":27065990,"url":"https://github.com/monarch1108/advanced-semantic-search-engine","last_synced_at":"2025-06-10T16:10:40.541Z","repository":{"id":284754031,"uuid":"955955048","full_name":"MONARCH1108/Advanced-Semantic-Search-Engine","owner":"MONARCH1108","description":"An advanced subtitle search engine using Whisper, ChromaDB, and NLP for semantic search, filtering, and visualizations with audio transcription.","archived":false,"fork":false,"pushed_at":"2025-05-23T12:43:32.000Z","size":491,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-23T14:00:33.839Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MONARCH1108.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-27T13:19:05.000Z","updated_at":"2025-05-23T12:43:36.000Z","dependencies_parsed_at":"2025-03-27T14:40:13.360Z","dependency_job_id":null,"html_url":"https://github.com/MONARCH1108/Advanced-Semantic-Search-Engine","commit_stats":null,"previous_names":["monarch1108/advanced-semantic-search-engine"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FAdvanced-Semantic-Search-Engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FAdvanced-Semantic-Search-Engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FAdvanced-Semantic-Search-Engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FAdvanced-Semantic-Search-Engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MONARCH1108","download_url":"https://codeload.github.com/MONARCH1108/Advanced-Semantic-Search-Engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MONARCH1108%2FAdvanced-Semantic-Search-Engine/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259106728,"owners_count":22805941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-05T18:34:40.896Z","updated_at":"2025-06-10T16:10:40.530Z","avatar_url":"https://github.com/MONARCH1108.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🎬 Movie Subtitle Search Engine\n\nAn advanced semantic search engine that identifies movies based on memorable quotes and dialogue lines using Natural Language Processing and vector embeddings.\n\n## Documentation :\nhttps://medium.com/@abhayemani8/building-a-movie-subtitle-search-engine-from-ai-assisted-to-self-coded-development-8a789407471d\n\n## 🌟 Features\n\n- **Semantic Search**: Find movies using natural language queries, not just exact matches\n- **Multiple Versions**: Two complete implementations (Streamlit \u0026 Flask)\n- **Vector Embeddings**: Uses sentence transformers for accurate semantic matching\n- **Real-time Search**: Fast similarity search with confidence scoring\n- **Interactive UI**: Clean, modern web interface\n- **Audio Support**: Convert speech to text for voice-based searches (Version 1)\n- **Extensible**: Easy to add new movie subtitles and expand the database\n\n## 🚀 Demo\n\nSimply enter a movie quote like:\n\n- _\"I can do this all day\"_ → Captain America\n- _\"May the force be with you\"_ → Star Wars\n- _\"I'll be back\"_ → Terminator\n\n## 📁 Project Structure\n\n```\nsubtitle-search-engine/\n├── version_1/                    # Streamlit Implementation (AI-Assisted)\n│   ├──AI_SEO.py                  # Advanced subtitle search engine\n│   └── requirements.txt\n├── version_2/                    # Flask Implementation (Self-Coded)\n│   ├── app.py                   # Flask web application\n│   ├── templates/\n│   │   └── index.html          # Frontend interface\n│   ├── subtitles/              # Movie subtitle files (.txt)\n│   ├── chroma_subtitles/       # Vector database storage\n│   └── app.ipynb              # Jupyter notebook demo\n└── README.md\n```\n\n## 🛠️ Technologies Used\n\n### Core Technologies\n\n- **Python 3.8+**\n- **ChromaDB** - Vector database for embeddings\n- **Sentence Transformers** - Semantic text embeddings\n- **Langchain** - Document processing and retrieval\n\n### Version 1 (Streamlit)\n\n- **Streamlit** - Web interface\n- **Whisper** - Speech-to-text conversion\n- **SQLite** - Local database\n- **Matplotlib/Seaborn** - Data visualization\n\n### Version 2 (Flask)\n\n- **Flask** - Web framework\n- **HTML/CSS/JavaScript** - Frontend\n- **HuggingFace Transformers** - NLP models\n\n## ⚡ Quick Start\n\n### Version 2 (Flask - Recommended)\n\n1. **Clone the repository**\n    \n    bash\n    \n    ```bash\n    git clone https://github.com/MONARCH1108/Advanced-Semantic-Search-Engine\n    cd subtitle-search-engine/version_2\n    ```\n    \n2. **Install dependencies**\n    \n    bash\n    \n    ```bash\n    pip install flask langchain-chroma langchain-huggingface chromadb sentence-transformers\n    ```\n    \n3. **Prepare subtitle data**\n    - Create a `subtitles/` directory\n    - Add movie subtitle files as `.txt` files\n    - Or use the provided Marvel dataset\n4. **Run the application**\n    \n    bash\n    \n    ```bash\n    python app.py\n    ```\n    \n5. **Open your browser**\n    \n    ```\n    http://localhost:5000\n    ```\n    \n\n### Version 1 (Streamlit)\n\n1. **Navigate to version 1**\n    \n    bash\n    \n    ```bash\n    cd version_1\n    ```\n    \n2. **Install dependencies**\n    \n    bash\n    \n    ```bash\n    pip install -r requirements.txt\n    ```\n    \n3. **Run the application**\n    \n    bash\n    \n    ```bash\n    python main.py\n    ```\n    \n\n## 📊 Dataset\n\nThe project uses movie subtitle files in plain text format. You can:\n\n1. **Use the Marvel Cinematic Universe dataset** (demonstrated in `app.ipynb`)\n2. **Add your own subtitle files** to the `subtitles/` directory\n3. **Download from subtitle websites** like OpenSubtitles\n\n### Supported Formats\n\n- `.txt` files with UTF-8, Latin-1, or CP1252 encoding\n- One subtitle file per movie\n- Automatic text chunking for better search performance\n\n## 🔍 How It Works\n\n### 1. Text Processing\n\n- Subtitle files are read and processed with multiple encoding fallbacks\n- Text is split into meaningful chunks using Langchain's text splitter\n- Each chunk maintains metadata about its source movie\n\n### 2. Vector Embeddings\n\n- Uses `sentence-transformers/paraphrase-MiniLM-L6-v2` model\n- Converts text chunks into high-dimensional vectors\n- Stores embeddings in ChromaDB for fast similarity search\n\n### 3. Semantic Search\n\n- User queries are converted to embeddings\n- Cosine similarity calculated against stored vectors\n- Results ranked by confidence score\n\n### 4. Web Interface\n\n- Real-time search with loading states\n- Confidence scoring for match quality\n- Responsive design for mobile and desktop\n\n## 🎯 API Endpoints (Version 2)\n\n|Endpoint|Method|Description|\n|---|---|---|\n|`/`|GET|Main search interface|\n|`/search`|POST|Perform subtitle search|\n|`/health`|GET|System health check|\n\n### Search API Example\n\njavascript\n\n```javascript\nPOST /search\n{\n  \"query\": \"I can do this all day\",\n  \"top_k\": 5\n}\n```\n\nResponse:\n\njavascript\n\n```javascript\n{\n  \"results\": [\n    {\n      \"movie\": \"Captain.America.The.First.Avenger\",\n      \"score\": 0.1234,\n      \"matched_text\": \"I can do this all day...\",\n      \"confidence\": 87.7\n    }\n  ]\n}\n```\n\n## 🔧 Configuration\n\n### Embedding Model\n\nChange the embedding model in the code:\n\npython\n\n```python\nembedding_function = HuggingFaceEmbeddings(\n    model_name=\"sentence-transformers/all-MiniLM-L6-v2\"  # Alternative model\n)\n```\n\n### Search Parameters\n\n- `top_k`: Number of results to return (default: 5)\n- `chunk_size`: Text chunk size for processing (default: 500)\n- `chunk_overlap`: Overlap between chunks (default: 50)\n\n## 📈 Performance\n\n- **Search Speed**: ~100-500ms per query\n- **Memory Usage**: ~200MB for 50 movies\n- **Accuracy**: 85-95% for exact quotes, 70-85% for paraphrased queries\n- **Scalability**: Handles 1000+ movies efficiently\n\n## 🤖 Development Journey\n\n### Version 1: AI-Assisted Development\n\n- **Approach**: Heavily relied on ChatGPT and AI tools\n- **Features**: Advanced analytics, audio processing, comprehensive search\n- **Learning**: Understanding AI capabilities and limitations\n\n### Version 2: Self-Coded Implementation\n\n- **Approach**: Minimal AI assistance, focused on core functionality\n- **Features**: Clean architecture, efficient search, modern UI\n- **Learning**: Deep understanding of semantic search principles\n\n## 🚀 Future Enhancements\n\n- [ ]  **Multi-language Support** - Support for non-English subtitles\n- [ ]  **Advanced Filtering** - Filter by genre, year, rating\n- [ ]  **User Accounts** - Save favorite searches and movies\n- [ ]  **Batch Processing** - Upload multiple subtitle files\n- [ ]  **REST API** - Full API for integration with other apps\n- [ ]  **Docker Support** - Containerized deployment\n- [ ]  **Cloud Deployment** - Deploy on AWS/GCP/Azure\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fadvanced-semantic-search-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonarch1108%2Fadvanced-semantic-search-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonarch1108%2Fadvanced-semantic-search-engine/lists"}