{"id":16228688,"url":"https://github.com/kabachuha/video2scenario","last_synced_at":"2025-03-19T13:31:29.921Z","repository":{"id":175215773,"uuid":"649871545","full_name":"kabachuha/video2scenario","owner":"kabachuha","description":"Recursively writes descriptions of video scenes using Large Language Models and Image Captioners","archived":false,"fork":false,"pushed_at":"2024-03-12T17:25:54.000Z","size":171,"stargazers_count":13,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-10T18:07:23.688Z","etag":null,"topics":["gradio","image-captioning","large-language-models","video-labeling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kabachuha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-05T20:30:29.000Z","updated_at":"2024-08-12T20:31:37.000Z","dependencies_parsed_at":"2024-03-03T13:28:32.994Z","dependency_job_id":"205ba390-5a68-49c0-85ed-547f11705557","html_url":"https://github.com/kabachuha/video2scenario","commit_stats":null,"previous_names":["kabachuha/video2scenario"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabachuha%2Fvideo2scenario","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabachuha%2Fvideo2scenario/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabachuha%2Fvideo2scenario/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabachuha%2Fvideo2scenario/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kabachuha","download_url":"https://codeload.github.com/kabachuha/video2scenario/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243992408,"owners_count":20380199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gradio","image-captioning","large-language-models","video-labeling"],"created_at":"2024-10-10T12:56:08.663Z","updated_at":"2025-03-19T13:31:29.675Z","avatar_url":"https://github.com/kabachuha.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# video2scenario\n\nForms a tree-like folder dataset with L parts at the lowest level.\n\nThen recursively writes descriptions of scenes with Large Language Models and Image Captioning Models.\n\nThe lowest level clips are [captioned with VideoLLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA).\n\nThe descriptions are gathered in a list and then the LLM is asked to describe the overall scene. Then the process continutes until the top level.\n\nAny OpenAI-like text completion model can be used for this. In my tests [Oobabooga's text generation webui](https://github.com/oobabooga/text-generation-webui) is used as the API endpoint.\n\nUser can also provide the master prompt to help the model and edit the resulting descriptions with a Gradio demo interface.\n\nThere is also an option to store the resulting corrected output for better fine-tuning the models, for example, using a LoRA.\n\nThe Gradio interface has a dropdown to select each description - clip pair, on each level.\n\n---\n\nThe goal of this subproject is to make a DiffusionOverDiffusion dataset to train [InfiNet](https://github.com/kabachuha/InfiNet) and the future complex script-based text2video models with minimal human labeling efforts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabachuha%2Fvideo2scenario","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkabachuha%2Fvideo2scenario","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabachuha%2Fvideo2scenario/lists"}