{"id":21649477,"url":"https://github.com/htanh2003/llm_powered_video_search","last_synced_at":"2025-08-21T02:07:23.193Z","repository":{"id":263086383,"uuid":"887714717","full_name":"HTAnh2003/LLM_Powered_Video_Search","owner":"HTAnh2003","description":"The LLM-Powered Video Search System is an advanced multimodal video search solution that leverages Large Language Models (LLMs) to enhance video retrieval through text, image, and metadata queries.","archived":false,"fork":false,"pushed_at":"2025-01-07T08:24:47.000Z","size":5549,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T20:29:09.014Z","etag":null,"topics":["clip","django","docker","faiss","multimodal","retrieval","retrieval-augmented-generation","text-image-retrieval","tf-idf","yolo"],"latest_commit_sha":null,"homepage":"https://faster-united.info/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HTAnh2003.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-13T06:44:25.000Z","updated_at":"2025-01-07T08:24:51.000Z","dependencies_parsed_at":"2025-01-25T05:32:26.467Z","dependency_job_id":null,"html_url":"https://github.com/HTAnh2003/LLM_Powered_Video_Search","commit_stats":null,"previous_names":["htanh2003/llm_powered_video_search"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HTAnh2003/LLM_Powered_Video_Search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FLLM_Powered_Video_Search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FLLM_Powered_Video_Search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FLLM_Powered_Video_Search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FLLM_Powered_Video_Search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HTAnh2003","download_url":"https://codeload.github.com/HTAnh2003/LLM_Powered_Video_Search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HTAnh2003%2FLLM_Powered_Video_Search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271415496,"owners_count":24755639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","django","docker","faiss","multimodal","retrieval","retrieval-augmented-generation","text-image-retrieval","tf-idf","yolo"],"created_at":"2024-11-25T07:31:13.570Z","updated_at":"2025-08-21T02:07:23.186Z","avatar_url":"https://github.com/HTAnh2003.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./figs/image/AIC2024-Banner.png\" width=\"1080\"\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003e 🧠 LLM-Powered Video Search System for AIC2024 \u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cem\u003eAn intelligent video retrieval system leveraging Large Language Models (LLMs) and multimodal search, developed for the AIC2024 competition and accepted at the international SOICT 2024 conference.\u003c/em\u003e\n\u003c/p\u003e\n\n![Static Badge](https://img.shields.io/badge/python-\u003e=3.10-blue)\n![Static Badge](https://img.shields.io/badge/django-3.x-blue)\n![Static Badge](https://img.shields.io/badge/clip-v1.0-blue)\n![Static Badge](https://img.shields.io/badge/tfidf-1.5.2-blue)\n\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n\n  - [📍 Overview](#-overview)\n  - [🎯 Features](#-features)\n  - [🤖 Tech Stack](#-tech-stack)\n  - [🚀 Setup and Usage](#-setup-and-usage)\n  - [🎬 Demo](#-demo)\n  - [👣 Workflow](#-workflow)\n  - [📐 App Structure](#-app-structure)\n  - [🧑‍💻 Contributors](#-contributors)\n  - [📚 Citation](#-citation)\n\n\u003c/details\u003e\n\n## 📍 Overview \nThe `LLM-Powered Video Search System` is an advanced multimodal video search solution that leverages Large Language Models (LLMs) to enhance video retrieval through text, image, and metadata queries. This project was developed for the [AIC2024](https://aichallenge.hochiminhcity.gov.vn/) competition and has been accepted at the international [SOICT 2024](https://soict.org/) conference, aiming to provide an intelligent and efficient video search system. Details about the paper can be found on [Springer](https://link.springer.com/chapter/10.1007/978-981-96-4291-5_25).\n\n## 🎯 Features\n\n1. **Multimodal Search Capabilities**\n   - **Text-based search:** Supports ASR (Automatic Speech Recognition), OCR, captions, and descriptive image queries for improved accuracy.\n   - **Image-based search:** Enables users to find specific video segments based on images.\n   - **Metadata-based search:** Provides a 7x7 matrix for tagging objects and color attributes for contextual search.\n\n2. **LLM-Powered Interaction**\n   - Integrates LLMs (e.g., GPT-4) to handle natural language queries and deliver relevant search results tailored to the context.\n\n3. **User-Friendly Interface**\n   - A responsive user interface allows users to view results as keyframes or full video segments and interact with detailed metadata.\n\n## 🤖 Tech Stack\n\n- **Back-end**: Django\n- **Core Technologies**: CLIP, Faiss, TFIDF\n- **Supporting Technologies**: OpenCV, PyTorch, Transformers\n- **Development Tools**: Docker, Git, Jupyter Notebook\n\n## 🚀 Setup and Usage\n\n1. **Clone Repository**\n   ```bash\n   git clone https://github.com/HTAnh2003/LLM_Powered_Video_Search.git\n   cd LLM_Powered_Video_Search\n   ```\n\n2. **Install Dependencies**\n   Ensure Python and Django are installed. Then, install other dependencies from `requirements.txt`:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. **Configure `MEDIA_ROOT`**\n   Open [settings.py](./AIC/settings.py) in the `AIC/` folder and set `MEDIA_ROOT` to point to your local `media` directory:\n\n   ```python\n   MEDIA_ROOT = '/path/to/your/media'\n   ```\n   You can download the dataset from [Google Drive](https://drive.google.com/drive/folders/17Yab4iMAEzok0pO_czgbAkKBlaQ2ptqU) or [Kaggle](https://www.kaggle.com/datasets/tienanh2003/keyframes-v1-aic2024).\n\n4. **Verify Paths in `viewAPI.py`**\n   Ensure paths in [app/viewAPI.py](./app/viewAPI.py) are correct.\n\n5. **Run Migrations**\n   Update the database with migrations:\n\n   ```bash\n   python manage.py migrate\n   ```\n\n6. **Run the Application**\n   To start the application, use:\n\n   ```bash\n   python manage.py runserver\n   ```\n\n   The app will run by default at `http://127.0.0.1:8000/`.\n\n## 🎬 Demo\n\n- **Screenshots**: ![image](./figs/image/demo.png)\n\n## 👣 Workflow\n![Pipeline](./figs/image/Pipeline.png)\n- **Data Processing**: Video data is processed using ASR or extracted via TransnetV2, then converted into image features and metadata.\n![Data Processing](./figs/image/data_processing.png)\n- **LLM Powered Interaction**: Natural language queries are processed by the LLM and combined with image features and metadata for relevant video retrieval.\n![LLM Interaction](./figs/image/LLM.png)\n\n\n## 📐 App Structure\n```\n├── LLM_Powered_Video_Search/\n│   ├── AIC/\n│   │   ├── settings.py\n│   ├── app/\n│   │   ├── admin.py\n│   │   ├── data_utils.py\n│   │   ├── migrations/\n│   │   ├── static/\n│   │   ├── templates/\n│   │   ├── viewAPI.py \n│   ├── data_extraction/\n│   │   ├── TransnetV2/\n│   │   ├── audio/\n│   │   ├── metadata/\n│   ├── docker-compose.yml\n│   ├── figs/\n│   ├── manage.py\n│   ├── requirements.txt\n│   ├── utils/\n│       ├── LLM/\n│       ├── video_retrieval/\n│       ├── faiss_search.py\n│       ├── combine_search.py\n|       |...\n```\n\n## 🧑‍💻 Contributors\n\n- [Hoàng Tiến Anh](https://github.com/HTAnh2003)\n- [Trần Xuân Diện](https://github.com/xndien2004)\n- [Dương Văn Tài](https://github.com/TaiDuongRepo)\n\n## 📚 Citation\n\nIf you use this system in your research or publications, please cite it using the following format:\n\n```bibtex\n@InProceedings{10.1007/978-981-96-4291-5_25,\n  author    = {Tran, Dien X. and Hoang, Anh T. and Duong, Tai V. and Nguyen, Kien C.},\n  editor    = {Buntine, Wray and Fjeld, Morten and Tran, Truyen and Tran, Minh-Triet and Huynh Thi Thanh, Binh and Miyoshi, Takumi},\n  title     = {LLM-Powered Video Search: A Comprehensive Multimedia Retrieval System},\n  booktitle = {Information and Communication Technology},\n  year      = {2025},\n  publisher = {Springer Nature Singapore},\n  address   = {Singapore},\n  pages     = {305--315},\n  isbn      = {978-981-96-4291-5}\n}\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtanh2003%2Fllm_powered_video_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhtanh2003%2Fllm_powered_video_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhtanh2003%2Fllm_powered_video_search/lists"}