{"id":19845982,"url":"https://github.com/vchitect/vbench","last_synced_at":"2025-04-12T16:33:55.301Z","repository":{"id":209520507,"uuid":"724118604","full_name":"Vchitect/VBench","owner":"Vchitect","description":"[CVPR2024 Highlight] VBench - We Evaluate Video Generation","archived":false,"fork":false,"pushed_at":"2025-04-05T14:09:10.000Z","size":68831,"stargazers_count":883,"open_issues_count":54,"forks_count":49,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-05T14:28:53.435Z","etag":null,"topics":["aigc","benchmark","dataset","evaluation-kit","gen-ai","stable-diffusion","text-to-video","video-generation"],"latest_commit_sha":null,"homepage":"https://vchitect.github.io/VBench-project/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Vchitect.png","metadata":{"files":{"readme":"README-pypi.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-27T12:41:46.000Z","updated_at":"2025-04-05T14:09:13.000Z","dependencies_parsed_at":"2024-02-04T09:34:40.680Z","dependency_job_id":"9250a564-2440-4c76-b575-fb3cbc3b1204","html_url":"https://github.com/Vchitect/VBench","commit_stats":{"total_commits":335,"total_committers":14,"mean_commits":"23.928571428571427","dds":0.7253731343283583,"last_synced_commit":"5448ffa97fa43c9db9f6d70d138e071914a24d14"},"previous_names":["vchitect/vbench"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FVBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FVBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FVBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FVBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Vchitect","download_url":"https://codeload.github.com/Vchitect/VBench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322867,"owners_count":21084395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","benchmark","dataset","evaluation-kit","gen-ai","stable-diffusion","text-to-video","video-generation"],"created_at":"2024-11-12T13:09:59.561Z","updated_at":"2025-04-12T16:33:50.291Z","avatar_url":"https://github.com/Vchitect.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![vbench_logo](https://raw.githubusercontent.com/Vchitect/VBench/master/asset/vbench_logo_short.jpg)\n\n**VBench** is a comprehensive benchmark suite for video generative models. You can use **VBench** to evaluate video generation models from 16 different ability aspects.\n\nThis project is the PyPI implementation of the following research:\n\u003e **VBench: Comprehensive Benchmark Suite for Video Generative Models**\u003cbr\u003e\n\u003e [Ziqi Huang](https://ziqihuangg.github.io/)\u003csup\u003e∗\u003c/sup\u003e, [Yinan He](https://github.com/yinanhe)\u003csup\u003e∗\u003c/sup\u003e, [Jiashuo Yu](https://scholar.google.com/citations?user=iH0Aq0YAAAAJ\u0026hl=zh-CN)\u003csup\u003e∗\u003c/sup\u003e, [Fan Zhang](https://github.com/zhangfan-p)\u003csup\u003e∗\u003c/sup\u003e, [Chenyang Si](https://chenyangsi.top/), [Yuming Jiang](https://yumingj.github.io/), [Yuanhan Zhang](https://zhangyuanhan-ai.github.io/),  [Tianxing Wu](https://tianxingwu.github.io/), [Qingyang Jin](https://github.com/Vchitect/VBench), [Nattapol Chanpaisit](https://nattapolchan.github.io/me), [Yaohui Wang](https://wyhsirius.github.io/), [Xinyuan Chen](https://scholar.google.com/citations?user=3fWSC8YAAAAJ), [Limin Wang](https://wanglimin.github.io), [Dahua Lin](http://dahua.site/)\u003csup\u003e+\u003c/sup\u003e, [Yu Qiao](http://mmlab.siat.ac.cn/yuqiao/index.html)\u003csup\u003e+\u003c/sup\u003e, [Ziwei Liu](https://liuziwei7.github.io/)\u003csup\u003e+\u003c/sup\u003e\u003cbr\u003e\n\n[![Paper](https://img.shields.io/badge/cs.CV-Paper-b31b1b?logo=arxiv\u0026logoColor=red)](https://arxiv.org/abs/2311.17982)\n[![Project Page](https://img.shields.io/badge/VBench-Website-green?logo=googlechrome\u0026logoColor=green)](https://vchitect.github.io/VBench-project/)\n[![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Leaderboard-blue)](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard)\n[![Video](https://img.shields.io/badge/YouTube-Video-c4302b?logo=youtube\u0026logoColor=red)](https://www.youtube.com/watch?v=7IhCC8Qqn8Y)\n[![Visitor](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FVchitect%2FVBench\u0026count_bg=%23FFA500\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=visitors\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\n## Installation\n```\npip install vbench\n```\n\nTo evaluate some video generation ability aspects, you need to install [detectron2](https://github.com/facebookresearch/detectron2) via:\n   ```\n   pip install detectron2@git+https://github.com/facebookresearch/detectron2.git\n   ```\n    \nIf there is an error during [detectron2](https://github.com/facebookresearch/detectron2) installation, see [here](https://detectron2.readthedocs.io/en/latest/tutorials/install.html).\n\n## Usage\n\n### Evaluate Your Own Videos\nWe support evaluating any video. Simply provide the path to the video file, or the path to the folder that contains your videos. There is no requirement on the videos' names.\n- Note: We support customized videos / prompts for the following dimensions: `'subject_consistency', 'background_consistency', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality'`\n\n\nTo evaluate videos with customed input prompt, run our script with `--mode=custom_input`:\n```\npython evaluate.py \\\n    --dimension $DIMENSION \\\n    --videos_path /path/to/folder_or_video/ \\\n    --mode=custom_input\n```\nalternatively you can use our command:\n```\nvbench evaluate \\\n    --dimension $DIMENSION \\\n    --videos_path /path/to/folder_or_video/ \\\n    --mode=custom_input\n```\n\n### Evaluation on the Standard Prompt Suite of VBench\n\n##### command line \n```bash\n    vbench evaluate --videos_path $VIDEO_PATH --dimension $DIMENSION\n```\nFor example:\n```bash\n    vbench evaluate --videos_path \"sampled_videos/lavie/human_action\" --dimension \"human_action\"\n```\n##### python\n```python\n    from vbench import VBench\n    my_VBench = VBench(device, \u003cpath/to/VBench_full_info.json\u003e, \u003cpath/to/save/dir\u003e)\n    my_VBench.evaluate(\n        videos_path = \u003cvideo_path\u003e,\n        name = \u003cname\u003e,\n        dimension_list = [\u003cdimension\u003e, \u003cdimension\u003e, ...],\n    )\n```\nFor example: \n```python\n    from vbench import VBench\n    my_VBench = VBench(device, \"vbench/VBench_full_info.json\", \"evaluation_results\")\n    my_VBench.evaluate(\n        videos_path = \"sampled_videos/lavie/human_action\",\n        name = \"lavie_human_action\",\n        dimension_list = [\"human_action\"],\n    )\n```\n\n### Evaluation on a specific category from VBench\n\n##### command line \n```bash\nvbench evaluate \\\n    --videos_path $VIDEO_PATH \\\n    --dimension $DIMENSION \\\n    --mode=vbench_category \\\n    --category=$CATEGORY\n```\nor \n```\npython evaluate.py \\\n    --dimension $DIMENSION \\\n    --videos_path /path/to/folder_or_video/ \\\n    --mode=vbench_category\n```\n\n## Prompt Suite\n\nWe provide prompt lists are at `prompts/`. \n\nCheck out [details of prompt suites](https://github.com/Vchitect/VBench/tree/master/prompts), and instructions for [**how to sample videos for evaluation**](https://github.com/Vchitect/VBench/tree/master/prompts).\n\n## Citation\n\n   If you find this package useful for your reports or publications, please consider citing the VBench paper:\n\n   ```bibtex\n    @article{huang2023vbench,\n        title={{VBench}: Comprehensive Benchmark Suite for Video Generative Models},\n        author={Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei},\n        journal={arXiv preprint arXiv:2311.17982},\n        year={2023}\n    }\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Fvbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvchitect%2Fvbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Fvbench/lists"}