{"id":19846008,"url":"https://github.com/vchitect/lavie","last_synced_at":"2025-05-15T03:02:48.560Z","repository":{"id":196756766,"uuid":"692725736","full_name":"Vchitect/LaVie","owner":"Vchitect","description":"[IJCV 2024] LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models","archived":false,"fork":false,"pushed_at":"2024-11-13T03:02:48.000Z","size":168407,"stargazers_count":924,"open_issues_count":24,"forks_count":62,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-04-14T01:53:15.953Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Vchitect.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-17T11:51:44.000Z","updated_at":"2025-04-11T14:18:20.000Z","dependencies_parsed_at":"2023-09-30T23:18:29.983Z","dependency_job_id":"347388e9-0609-480c-92f3-5ca1e5dfb549","html_url":"https://github.com/Vchitect/LaVie","commit_stats":null,"previous_names":["vchitect/lavie"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLaVie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLaVie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLaVie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLaVie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Vchitect","download_url":"https://codeload.github.com/Vchitect/LaVie/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264737,"owners_count":22041791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T13:10:03.015Z","updated_at":"2025-05-15T03:02:48.531Z","avatar_url":"https://github.com/Vchitect.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models\n\nThis repository is the official PyTorch implementation of [LaVie](https://arxiv.org/abs/2309.15103).\n\n**LaVie** is a Text-to-Video (T2V) generation framework, and main part of video generation system [Vchitect](http://vchitect.intern-ai.org.cn/). You can also check our fine-tuned Image-to-Video (I2V) model [SEINE](https://github.com/Vchitect/SEINE).\n\n\n[![arXiv](https://img.shields.io/badge/arXiv-2309.15103-b31b1b.svg)](https://arxiv.org/abs/2309.15103)\n[![Project Page](https://img.shields.io/badge/Project-Website-green)](https://vchitect.github.io/LaVie-project/)\n[![Replicate](https://replicate.com/cjwbw/lavie/badge)](https://replicate.com/cjwbw/lavie)\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://huggingface.co/spaces/Vchitect/LaVie)\n[![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)](https://openxlab.org.cn/apps/detail/houshaowei/LaVie)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FVchitect%2FLaVie%2F\u0026count_bg=%23368ED7\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=Visitors\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\n\n\u003c!--\n[![Open in OpenXLab](https://cdn-static.openxlab.org.cn/app-center/openxlab_app.svg)]()\n--\u003e\n\n## News\n__[2024.07.08]__: **LaVie-2** will be released soon, stay tuned!\n\u003cvideo poster=\"https://github.com/Vchitect/LaVie/assets/7929326/f54662e9-7641-4173-99ca-db7592d086c3\" preload=\"auto\" src=\"https://github.com/Vchitect/LaVie/assets/7929326/196ee3ca-f106-433a-8edc-7803f2f11aa4\" width=\"800\"\u003e\u003c/video\u003e\n\n\u003c!--\n\u003cimg src=\"lavie.gif\" width=\"800\"\u003e\n--\u003e\n\n## Installation\n```\nconda env create -f environment.yml \nconda activate lavie\n```\n\n## Download Pre-Trained models\nDownload pre-trained [LaVie models](https://huggingface.co/YaohuiW/LaVie/tree/main), [Stable Diffusion 1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/main), [stable-diffusion-x4-upscaler](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler/tree/main) to `./pretrained_models`. You should be able to see the following:\n```\n├── pretrained_models\n│   ├── lavie_base.pt\n│   ├── lavie_interpolation.pt\n│   ├── lavie_vsr.pt\n│   ├── stable-diffusion-v1-4\n│   │   ├── ...\n└── └── stable-diffusion-x4-upscaler\n        ├── ...\n```\nGallery:\n\n\u003ctable class=\"center\"\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/two_teddy_bears_playing_poker_under_water,_highly_detailed,_oil_painting_style.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_teddy_bear_skating_under_water,_highly_detailed.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_cat_reading_a_book_on_the_table,__Van_Gogh_style.gif\"\u003e\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003etwo teddy bears playing poker under water, highly detailed, oil painting style\u003c/td\u003e\n  \u003ctd\u003ea teddy bear skateboarding under water, highly detailed\u003c/td\u003e\n  \u003ctd\u003ea cat reading a book on the table, Van Gogh style\u003c/td\u003e      \n\u003c/tr\u003e\n        \n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_cute_raccoon_playing_guitar_in_the_park_at_sunrise,_oil_painting_style.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_teddy_bear_walking_in_the_park_at_sunrise_oil_painting_style.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/laviea_teddy_bear_reading_a_book_near_a_small_river,_oil_painting_style-.gif\"\u003e\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003ea cute raccoon playing guitar in the park at sunrise, oil painting style\u003c/td\u003e\n  \u003ctd\u003ea teddy bear walking in the park at sunrise, oil painting style\u003c/td\u003e\n  \u003ctd\u003ea teddy bear reading a book near a small river, oil painting style\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/Elon_Musk_in_spacesuit_standing_besides_a_rocket,_high_quality05.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_teddy_bear_in_a__suit_having_dinner.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/Iron_Man_flying_in_the_sky.gif\"\u003e\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003eElon Musk in a space suit standing besides a rocket, high quality\u003c/td\u003e\n  \u003ctd\u003ea teddy bear in a suit having dinner in a well-decorated house\u003c/td\u003e\n  \u003ctd\u003eIron Man flying in the sky, 4k, high quality\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003c/table\u003e\n\nFeel free to try different prompts, and share with us which one you like the most!\n\n\n## Inference\nThe inference contains **Base T2V**, **Video Interpolation** and **Video Super-Resolution** three steps. We provide several options to generate videos:\n\n|       |Step1|Step2|Step3|Resolution|Length|\n|-------|-----|-----|-----|----------|------|\n|option1|  ✔  |     |     | 320x512  |  16  |\n|option2|  ✔  |  ✔  |     | 320x512  |  61  |\n|option3|  ✔  |     |  ✔  | 1280x2048|  16  |\n|option4|  ✔  |  ✔  |  ✔  | 1280x2048|  61  |\n\nFeel free to try different options :)\n\n\n### Step1. Base T2V\nRun following command to generate videos from base T2V model. \n```\ncd base\npython pipelines/sample.py --config configs/sample.yaml\n```\nIn **configs/sample.yaml**, arguments for inference:\n\n- **ckpt_path:** Path to the downloaded LaVie base model, default is `../pretrained_models/lavie_base.pt`\n\n- **pretrained_models:** Path to the downloaded SD1.4, default is `../pretrained_models`\n\n- **output_folder:** Path to save generated results, default is `../res/base`\n\n- **seed:** Seed to be used, `None` for random generation\n\n- **sample_method:** Scheduler to use, default is `ddpm`, options are `ddpm`, `ddim` and `eulerdiscrete`\n\n- **guidance_scale:** CFG scale to use, default is `7.5`\n\n- **num_sampling_steps:** Denoising steps, default is `50`\n\n- **text_prompt:** Prompt for generation\n\nFollowing results were generated with the arguments:\n\nseed: `400`, sample_method: `ddpm`, guidance_scale: `7.0`, num_sampling_steps: `50`\n\n(you might obtain different results on different device)\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_Corgi_walking_in_the_park_at_sunrise,_oil_painting_style.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_panda_taking_a_selfie,_2k,_high_quality.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_polar_bear_playing_drum_kit_in_NYC_Times_Square,_4k,_high_resolution.gif\"\u003e\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003ea Corgi walking in the park at sunrise, oil painting style\u003c/td\u003e\n  \u003ctd\u003ea panda taking a selfie, 2k, high quality\u003c/td\u003e\n  \u003ctd\u003ea polar bear playing drum kit in NYC Times Square, 4k, high resolution\u003c/td\u003e      \n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_shark_swimming_in_clear_Carribean_ocean,_2k,_high_quality.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/a_teddy_bear_walking_on_the_street,_2k,_high_quality.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"assets/jungle_river_at_sunset,_ultra_quality.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003ea shark swimming in clear Carribean ocean, 2k, high quality\u003c/td\u003e\n  \u003ctd\u003ea teddy bear walking on the street, 2k, high quality\u003c/td\u003e\n  \u003ctd\u003ejungle, river, at sunset, ultra quality\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n### Step2 (optional). Video Interpolation\nRun following command to conduct video interpolation.\n```\ncd interpolation\npython sample.py --config configs/sample.yaml\n```\nThe default input video path is `./res/base`, results will be saved under `./res/interpolation`. In `configs/sample.yaml`, you could modify default `input_folder` with `YOUR_INPUT_FOLDER` in `configs/sample.yaml`. Input videos should be named as `prompt1.mp4`, `prompt2.mp4`, ... and put under `YOUR_INPUT_FOLDER`. Launching the code will process all the input videos in `input_folder`.\n\n### Step3 (optional). Video Super-Resolution\nRun following command to conduct video super-resolution.\n```\ncd vsr\npython sample.py --config configs/sample.yaml\n```\nThe default input video path is `./res/base` and results will be saved under `./res/vsr`. You could modify default `input_path` with `YOUR_INPUT_FOLDER` in `configs/sample.yaml`. Similar to Step2, input videos should be named as `prompt1.mp4`, `prompt2.mp4`, ... and put under `YOUR_INPUT_FOLDER`. Launching the code will process all the input videos in `input_folder`.\n\n## BibTeX\n```bibtex\n@article{wang2023lavie,\n  title={LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models},\n  author={Wang, Yaohui and Chen, Xinyuan and Ma, Xin and Zhou, Shangchen and Huang, Ziqi and Wang, Yi and Yang, Ceyuan and He, Yinan and Yu, Jiashuo and Yang, Peiqing and others},\n  journal={IJCV},\n  year={2024}\n}\n```\n\n```bibtex\n@inproceedings{chen2023seine,\n  title={Seine: Short-to-long video diffusion model for generative transition and prediction},\n  author={Chen, Xinyuan and Wang, Yaohui and Zhang, Lingjun and Zhuang, Shaobin and Ma, Xin and Yu, Jiashuo and Wang, Yali and Lin, Dahua and Qiao, Yu and Liu, Ziwei},\n  booktitle={ICLR},\n  year={2023}\n}\n```\n\n## Disclaimer\nWe disclaim responsibility for user-generated content. The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model's capabilities. It is prohibited for pornographic, violent and bloody content generation, and to generate content that is demeaning or harmful to people or their environment, culture, religion, etc. Users are solely liable for their actions. The project contributors are not legally affiliated with, nor accountable for users' behaviors. Use the generative model responsibly, adhering to ethical and legal standards.\n\n## Contact Us\n**Yaohui Wang**: [wangyaohui@pjlab.org.cn](mailto:wangyaohui@pjlab.org.cn)  \n**Xinyuan Chen**: [chenxinyuan@pjlab.org.cn](mailto:chenxinyuan@pjlab.org.cn)  \n**Xin Ma**: [xin.ma1@monash.edu](mailto:xin.ma1@monash.edu)\n\n## Acknowledgements\nThe code is built upon [diffusers](https://github.com/huggingface/diffusers) and [Stable Diffusion](https://github.com/CompVis/stable-diffusion), we thank all the contributors for open-sourcing. \n\n\n## License\nThe code is licensed under Apache-2.0, model weights are fully open for academic research and also allow **free** commercial usage. To apply for a commercial license, please contact vchitect@pjlab.org.cn.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Flavie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvchitect%2Flavie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Flavie/lists"}