{"id":20161904,"url":"https://github.com/ailab-cvc/videogen-eval","last_synced_at":"2026-01-27T23:48:01.170Z","repository":{"id":258579797,"uuid":"864610689","full_name":"AILab-CVC/VideoGen-Eval","owner":"AILab-CVC","description":"VideoGen-Eval: Agent-based System for Video Generation Evaluation","archived":false,"fork":false,"pushed_at":"2025-12-16T03:20:59.000Z","size":17772,"stargazers_count":253,"open_issues_count":5,"forks_count":14,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-12-19T14:08:40.654Z","etag":null,"topics":["aigc","benchmark","image-to-video","sora-video-ai","text-to-video","video-evaluation","video-generation","video-to-video"],"latest_commit_sha":null,"homepage":"https://ailab-cvc.github.io/VideoGen-Eval/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AILab-CVC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-28T17:17:41.000Z","updated_at":"2025-12-16T03:21:03.000Z","dependencies_parsed_at":"2025-01-13T13:57:14.461Z","dependency_job_id":null,"html_url":"https://github.com/AILab-CVC/VideoGen-Eval","commit_stats":null,"previous_names":["ailab-cvc/videogen-eval"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AILab-CVC/VideoGen-Eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FVideoGen-Eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FVideoGen-Eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FVideoGen-Eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FVideoGen-Eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AILab-CVC","download_url":"https://codeload.github.com/AILab-CVC/VideoGen-Eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FVideoGen-Eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28827902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T23:29:49.665Z","status":"ssl_error","status_checked_at":"2026-01-27T23:25:58.379Z","response_time":168,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","benchmark","image-to-video","sora-video-ai","text-to-video","video-evaluation","video-generation","video-to-video"],"created_at":"2024-11-14T00:21:41.947Z","updated_at":"2026-01-27T23:48:01.151Z","avatar_url":"https://github.com/AILab-CVC.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- Improved compatibility of back to top link: See: https://github.com/othneildrew/Best-README-Template/pull/73 --\u003e\n\u003ca id=\"readme-top\"\u003e\u003c/a\u003e\n\n[![Contributors][contributors-shield]][contributors-url]\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n\n\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/teaser/teaser.png\" alt=\"Logo\"\u003e\n\n  \u003ch1 align=\"center\"\u003eVideoGen-Eval 1.0\u003c/h1\u003e\n  \n#### [\u003ccode\u003eProject Page 🚀\u003c/code\u003e](https://ailab-cvc.github.io/VideoGen-Eval/) | [\u003ccode\u003eTechnical Report 📝\u003c/code\u003e](http://arxiv.org/abs/2410.05227)  | [\u003ccode\u003ePrompt 🎬\u003c/code\u003e](https://ailab-cvc.github.io/VideoGen-Eval/specifc_model/prompt.html)  | [\u003ccode\u003eVideo Download 🤩\u003c/code\u003e](https://drive.google.com/drive/folders/11WxQudsVgqI-ETXQB5PQjd7dzhz41-E0?usp=sharing) | [\u003ccode\u003eJoin WeChat 💬\u003c/code\u003e](https://github.com/AILab-CVC/VideoGen-Eval/blob/main/docs/specifc_model/wechat.md)\n\n  \u003cp align=\"center\"\u003e\n    To observe and compare the video quality of recent video generative models!\n    \u003cbr /\u003e\n    \u003ca href=\"https://ailingzeng.site/\"\u003eAiling Zeng\u003csup\u003e1\u003c/sup\u003e\u003csup\u003e*\u003c/sup\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://yyvhang.github.io/\"\u003eYuhang Yang\u003csup\u003e2\u003c/sup\u003e\u003csup\u003e*\u003c/sup\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"\"\u003eWeidong Chen\u003csup\u003e1\u003c/sup\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=AjxoEpIAAAAJ\u0026hl=en\"\u003eWei Liu\u003csup\u003e1\u003c/sup\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003cp\u003e \u003csub\u003e\u003csup\u003e1\u003c/sup\u003e Tencent AI Lab, \u003csup\u003e2\u003c/sup\u003e USTC. *Equal contribution\u003c/sub\u003e\u003c/p\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\n\n## 🔥 Project Updates\n- **News**: ```2024/12/21```: We update results of [PramidalFlow](https://pyramid-flow.github.io/), please check our website.\n- **News**: ```2024/12/10```: We update results of [Sora](https://openai.com/sora/) and the comparison results of the latest 6 Sota models.\n- **News**: ```2024/12/04```: We update results of [Hunyuan](https://github.com/Tencent/HunyuanVideo), please check our website.\n- **News**: ```2024/11/21```: We update results of Seaweed2.0 Pro, PixelDance2.0 Pro, Pika1.5 and Vidu1.5, please check our website.\n- **News**: ```2024/11/12```: We update results of [Seaweed](https://jimeng.jianying.com/ai-tool/home) and [PixVerse-V3](https://pixverse.ai/), please check our website.\n- **News**: ```2024/11/01```: We update text-to-video results of [Mochi1](https://www.genmo.ai/blog), we use `cfg=6.0`, which is the same as their website.\n- **News**: ```2024/10/19```: We update 1k text-to-video results of [Meta-MovieGen](https://ai.meta.com/research/movie-gen/) (prompts are from MovieGenVideoBench); please check [here](https://ailab-cvc.github.io/VideoGen-Eval/specifc_model/MovieGen/MovieGen.html). **Plus, we make the pypi package `VGenEval` available, you can easily obtain all input prompts (text, image, video) corresponding to any ID through jsut one line of code.**\n-  **News**: ```2024/10/14```: We update results of Minimax image-to-video generation, please check [here](https://ailab-cvc.github.io/VideoGen-Eval/specifc_model/minimax/minimax.html).\n-  **News**: ```2024/10/08```: VideoGen-Eval-1.0 is available, please check the [Project Page](https://ailab-cvc.github.io/VideoGen-Eval/) and [Technical Report](http://arxiv.org/abs/2410.05227) for more details.\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#assets\"\u003eAssets\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#job-list\"\u003eJob List\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## 💡 About The Project \nHigh-quality video generation, such as text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation, holds considerable significance in content creation and world simulation. Models like SORA have advanced generating videos with higher resolution, more natural motion, better vision-language alignment, and increased controllability, particularly for long video sequences. These improvements have been driven by the evolution of model architectures, shifting from UNet to more scalable and parameter-rich DiT models, along with large-scale data expansion and refined training strategies. However, despite the emergence of several DiT-based closed-source and open-source models, a comprehensive investigation into their capabilities and limitations still needs to be completed. Additionally, existing evaluation metrics often fail to align with human preferences.\n\nThis report v1.0 studies a series of SORA-like T2V, I2V, and V2V models via to bridge the gap between academic research and industry practice and provide a more profound analysis of recent video generation advancements. This is achieved by demonstrating and comparing over 8,000 generated video cases from **ten closed-source and several open-source models** (Kling 1.0, Kling 1.5, Gen-3, Luma 1.0, Luma 1.6, Vidu, Qingying, MiniMax Hailuo, Tongyi Wanxiang, Pika1.5) via our 700 critical prompts. Seeing is believing. We encourage readers to visit our [Website](https://ailab-cvc.github.io/VideoGen-Eval/) to browse these results online. Our study systematically examines four core aspects: \n\n\n* Impacts on vertical-domain application models, such as human-centric animation and robotics;\n* Key objective capabilities, such as text alignment, motion diversity, composition, stability, etc.;\n* Video generation across ten real-life application scenarios;\n* In-depth discussions on potential usage scenarios and tasks, challenges, and future work.\n\n\nWe assign an ID to each case. The input text, the names of input images and videos correspond to the ID. The results generated by different models are named as `model_name+id.mp4`. Please refer to the [prompt](https://ailab-cvc.github.io/VideoGen-Eval/specifc_model/prompt.html). All the results are publicly accessible, and we will continuously update the results as new models are released and existing ones undergo version updates. \n\n## 🎞️ Assets\n\nThe inputs we introduced, including the input text, images, videos, and the generated results of all models, are available for download at [Google Drive](https://drive.google.com/drive/folders/11WxQudsVgqI-ETXQB5PQjd7dzhz41-E0?usp=sharing) and [Baidu](https://pan.baidu.com/s/16nhiiKIYn3EPRMpefEoEqw?pwd=rgha). You can also visit our [Website](https://ailab-cvc.github.io/VideoGen-Eval/) to browse these results online.\n\nGet VideoGen-Eval prompts:\n```bash\npip install VGenEval\n\n# example\n#id_list of the id e.g. [0,1,2,3]\n#test_model_name of the model name e.g. 'SORA'\nfrom VGenEval import load_prompt\nresults = load_prompt.get_prompts([id_list], 'test_model_name')\n\n# result is a dict, {\n#   'text prompt': [],\n#   'visual prompt': [], return the url of the input image or video\n#   'save name': [], We have standardized the save name\n# }\n\n# note: for the sample which takes the first-last frame for generation, visual prompt return urls of the two frames.\n```\n\n## 🦉 Job List\n\n- [x] VideoGen-Eval-1.0 released \n- [x] Add results of Seaweed, PixVerse.\n- [ ] Make the arena for video generation models.\n\n\u003c!-- CONTRIBUTING --\u003e\n## 💗 Contributing\nWelcome all contributions! If you have a suggestion to improve this project, please fork the repo and create a pull request. You can also open an issue with the tag \"enhancement.\"\nDon't forget to give the project a star! Thanks again!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some change'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n### 🏄 Top contributors:\n\n\u003ca href=\"https://github.com/AILab-CVC/VideoGen-Eval/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=AILab-CVC/VideoGen-Eval\" alt=\"contrib.rocks image\" /\u003e\n\u003c/a\u003e\n\n\u003c!-- LICENSE --\u003e\n## ✏️ License\n\nDistributed under the MIT License. See `LICENSE.txt` for more information.\n\n\u003c!-- CONTACT --\u003e\n## 📢 Contact\n\nAiling Zeng - [ailingzengzzz@gmail.com](mailto:ailingzengzzz@gmail.com)\n\nYuhang Yang - [yyuhang@mail.ustc.edu.cn](mailto:yyuhang@mail.ustc.edu.cn)\n\n## 💌 Citation\n```\n@article{zeng2024dawn,\n  title={The Dawn of Video Generation: Preliminary Explorations with SORA-like Models},\n  author={Zeng, Ailing and Yang, Yuhang and Chen, Weidong and Liu, Wei},\n  journal={arXiv preprint arXiv:2410.05227},\n  year={2024}\n}\n```\n[contributors-shield]: https://img.shields.io/github/contributors/AILab-CVC/VideoGen-Eval.svg?style=for-the-badge\n[contributors-url]: https://github.com/AILab-CVC/VideoGen-Eval/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/AILab-CVC/VideoGen-Eval.svg?style=for-the-badge\n[forks-url]: https://github.com/othneildrew/Best-README-Template/network/members\n[stars-shield]: https://img.shields.io/github/stars/AILab-CVC/VideoGen-Eval.svg?style=for-the-badge\n[stars-url]: https://github.com/AILab-CVC/VideoGen-Eval/stargazers\n[issues-shield]: https://img.shields.io/github/issues/AILab-CVC/VideoGen-Eval.svg?style=for-the-badge\n[issues-url]: https://github.com/AILab-CVC/VideoGen-Eval/issues\n[product-screenshot]: images/screenshot.png\n[Next.js]: https://img.shields.io/badge/next.js-000000?style=for-the-badge\u0026logo=nextdotjs\u0026logoColor=white\n[Next-url]: https://nextjs.org/\n[React.js]: https://img.shields.io/badge/React-20232A?style=for-the-badge\u0026logo=react\u0026logoColor=61DAFB\n[React-url]: https://reactjs.org/\n[Vue.js]: https://img.shields.io/badge/Vue.js-35495E?style=for-the-badge\u0026logo=vuedotjs\u0026logoColor=4FC08D\n[Vue-url]: https://vuejs.org/\n[Angular.io]: https://img.shields.io/badge/Angular-DD0031?style=for-the-badge\u0026logo=angular\u0026logoColor=white\n[Angular-url]: https://angular.io/\n[Svelte.dev]: https://img.shields.io/badge/Svelte-4A4A55?style=for-the-badge\u0026logo=svelte\u0026logoColor=FF3E00\n[Svelte-url]: https://svelte.dev/\n[Laravel.com]: https://img.shields.io/badge/Laravel-FF2D20?style=for-the-badge\u0026logo=laravel\u0026logoColor=white\n[Laravel-url]: https://laravel.com\n[Bootstrap.com]: https://img.shields.io/badge/Bootstrap-563D7C?style=for-the-badge\u0026logo=bootstrap\u0026logoColor=white\n[Bootstrap-url]: https://getbootstrap.com\n[JQuery.com]: https://img.shields.io/badge/jQuery-0769AD?style=for-the-badge\u0026logo=jquery\u0026logoColor=white\n[JQuery-url]: https://jquery.com \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Failab-cvc%2Fvideogen-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Failab-cvc%2Fvideogen-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Failab-cvc%2Fvideogen-eval/lists"}