{"id":30193930,"url":"https://github.com/siddhigate/video-query-ai","last_synced_at":"2025-08-13T02:06:45.710Z","repository":{"id":305238991,"uuid":"1021462400","full_name":"siddhigate/video-query-ai","owner":"siddhigate","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-18T22:26:26.000Z","size":125,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-19T03:14:14.081Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/siddhigate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-17T12:37:53.000Z","updated_at":"2025-07-18T22:26:29.000Z","dependencies_parsed_at":"2025-07-19T03:25:39.920Z","dependency_job_id":null,"html_url":"https://github.com/siddhigate/video-query-ai","commit_stats":null,"previous_names":["siddhigate/video-query-ai"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/siddhigate/video-query-ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhigate%2Fvideo-query-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhigate%2Fvideo-query-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhigate%2Fvideo-query-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhigate%2Fvideo-query-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/siddhigate","download_url":"https://codeload.github.com/siddhigate/video-query-ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/siddhigate%2Fvideo-query-ai/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270166108,"owners_count":24538445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-13T02:06:42.332Z","updated_at":"2025-08-13T02:06:45.615Z","avatar_url":"https://github.com/siddhigate.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Video Query AI: Search your videos like you search text 🧠🎥\n\n## Overview\nA fully local, privacy-first app that helps you semantically search through videos using natural language. Simply upload a video, and we’ll do the heavy lifting to process frames, generate descriptions using AI, embed them, and allow fast search.\n\n## ✨ Features\n- 🔍 Semantic search: Find scenes by describing them in plain English\n- 📤 Drag-and-drop video upload\n- 🖼️ Instant timestamps \u0026 thumbnails for search results\n- ⚡ Real-time resumable processing updates\n- 🔒 100% local, privacy-first architecture\n\n\n## App Walkthrough ✨\n\n### 1️⃣ 🏠 Home \u0026 Video Library\n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862225260/b5994750-d236-4ac6-a28c-e84afeea6d0b.gif)\n\n- Land on a clean dashboard with a sidebar listing all uploaded videos.\n- Browse or search through your video collection easily.\n\n### 2️⃣ 📤 Video Upload \u0026 Processing\n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862242679/4adcc684-5210-454a-a633-b8f2e79408d8.gif)\n\n- Drag and drop a video file for processing.\n- The backend kicks off a background job to:\n    - Extract frames using `ffmpeg`\n    - Generate descriptions via **LLaVA**\n    - Generate vector embeddings of those descriptions for semantic search\n- Real-time progress updates\n    - Updates are streamed live via WebSockets.\n    - Progress persists across reloads using Redis Pub/Sub for state sync\n\n### 3️⃣ 🔍 Natural Language Search\n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862256793/877b3735-0fa8-4f64-8764-7eddb6003a5e.gif)\n\n- Enter a query like *\"Where is the elephant?\"* or *\"Chef chopping onions\"*\n- The app performs a vector similarity search against frame captions.\n- You’ll get timestamps + thumbnails of the best matching moments in the video.\n\n\n## 🛠 Under the Hood \n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862278277/faa90937-b973-4ebd-9da4-1961d47714c4.png)\n\nVideo Query AI follows a modular architecture built by:\n\n- **Frontend:** React + TypeScript with Vite and React Router\n- **Backend:** FastAPI serving REST and WebSocket endpoints\n- **Job Queue:** Redis + RQ for background processing\n- **Embedding Store:** ChromaDB for vector search\n- **Realtime Updates:** WebSockets with Redis Pub/Sub for progress tracking and resumable streams\n\n### 📼 Video Processing Flow\n\nHere’s what happens behind the scenes when you upload a video:\n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862439792/7a198ef5-81a6-44ca-9307-a09c63521697.png)\n\n1. **Upload**\n    - File is saved to disk and its metadata is stored in ChromaDB.\n2. **Job Queuing**\n    - A video processing job is pushed to a **Redis Queue** and handled asynchronously by a worker.\n3. **Frame Extraction**\n    - Frames are extracted from the video using `ffmpeg`.\n4. **Frame Analysis**\n    - Each frame is sent through **LLaVA (via Ollama)** to describe it.\n    - Description is embedded into a vector using a sentence transformer.\n5. **Storage**\n    - Vector embeddings + metadata is stored in ChromaDB.\n6. **Progress Updates**\n    - Real-time progress is sent to the frontend via WebSockets + Redis PubSub.\n\n### 🔍 Search Flow\n\nUsers can search across:\n\n- **All uploaded videos**\n- **A single selected video**\n\n![](https://cdn.hashnode.com/res/hashnode/image/upload/v1752862430476/7144c89f-148b-4b72-8a15-99b255f4163a.png)\n\nWhen a query is made:\n\n1. The backend embeds the query using the same embedding model.\n2. A **vector similarity search** is performed in ChromaDB.\n3. Top 10 closest matches (timestamps + thumbnails) are returned.\n\n### 🔁 Real-Time Progress \u0026 Resumability\n\nEven if the user refreshes the page mid-processing:\n\n- The frontend reconnects via WebSocket.\n- The backend reads the current job state from Redis and resumes updates seamlessly.\n\n## 🛠 Useful Links\n- [Blog](https://blog.siddhigate.com/video-query-ai-search-your-videos-like-you-search-text)\n- [Demo](https://www.loom.com/share/10c1299c709545598d2f533761b57972?sid=e3dd6104-85a3-433b-81e7-1807d689eead)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiddhigate%2Fvideo-query-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsiddhigate%2Fvideo-query-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsiddhigate%2Fvideo-query-ai/lists"}