https://github.com/kabachuha/video2scenario
Recursively writes descriptions of video scenes using Large Language Models and Image Captioners
https://github.com/kabachuha/video2scenario
gradio image-captioning large-language-models video-labeling
Last synced: over 1 year ago
JSON representation
Recursively writes descriptions of video scenes using Large Language Models and Image Captioners
- Host: GitHub
- URL: https://github.com/kabachuha/video2scenario
- Owner: kabachuha
- License: agpl-3.0
- Created: 2023-06-05T20:30:29.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-03-12T17:25:54.000Z (over 2 years ago)
- Last Synced: 2025-03-10T18:07:23.688Z (over 1 year ago)
- Topics: gradio, image-captioning, large-language-models, video-labeling
- Language: Python
- Homepage:
- Size: 167 KB
- Stars: 13
- Watchers: 2
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# video2scenario
Forms a tree-like folder dataset with L parts at the lowest level.
Then recursively writes descriptions of scenes with Large Language Models and Image Captioning Models.
The lowest level clips are [captioned with VideoLLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA).
The descriptions are gathered in a list and then the LLM is asked to describe the overall scene. Then the process continutes until the top level.
Any OpenAI-like text completion model can be used for this. In my tests [Oobabooga's text generation webui](https://github.com/oobabooga/text-generation-webui) is used as the API endpoint.
User can also provide the master prompt to help the model and edit the resulting descriptions with a Gradio demo interface.
There is also an option to store the resulting corrected output for better fine-tuning the models, for example, using a LoRA.
The Gradio interface has a dropdown to select each description - clip pair, on each level.
---
The goal of this subproject is to make a DiffusionOverDiffusion dataset to train [InfiNet](https://github.com/kabachuha/InfiNet) and the future complex script-based text2video models with minimal human labeling efforts.