{"id":13993878,"url":"https://github.com/Vchitect/SEINE","last_synced_at":"2025-07-22T18:32:37.310Z","repository":{"id":204515997,"uuid":"695229234","full_name":"Vchitect/SEINE","owner":"Vchitect","description":"[ICLR 2024] SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction","archived":false,"fork":false,"pushed_at":"2024-11-13T03:01:30.000Z","size":53033,"stargazers_count":938,"open_issues_count":24,"forks_count":63,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-05-23T05:12:40.685Z","etag":null,"topics":["diffusion-model","image-to-video","stable-diffusion","video-generation"],"latest_commit_sha":null,"homepage":"https://vchitect.github.io/SEINE-project/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Vchitect.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-22T16:30:31.000Z","updated_at":"2025-05-23T04:10:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"5d8b4aca-4404-4677-a58a-431b12f28e3b","html_url":"https://github.com/Vchitect/SEINE","commit_stats":null,"previous_names":["vchitect/seine"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Vchitect/SEINE","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FSEINE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FSEINE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FSEINE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FSEINE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Vchitect","download_url":"https://codeload.github.com/Vchitect/SEINE/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FSEINE/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266552467,"owners_count":23947174,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-model","image-to-video","stable-diffusion","video-generation"],"created_at":"2024-08-09T14:02:36.448Z","updated_at":"2025-07-22T18:32:37.296Z","avatar_url":"https://github.com/Vchitect.png","language":"Python","readme":"# SEINE\n[![arXiv](https://img.shields.io/badge/arXiv-2310.20700-b31b1b.svg)](https://arxiv.org/abs/2310.20700)\n[![Project Page](https://img.shields.io/badge/SEINE-Website-green)](https://vchitect.github.io/SEINE-project/)\n[![Replicate](https://replicate.com/lucataco/seine/badge)](https://replicate.com/lucataco/seine) \n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://huggingface.co/spaces/Vchitect/SEINE)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FVchitect%2FSEINE\u0026count_bg=%23F59352\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=visitors\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\nThis repository is the official implementation of [SEINE](https://arxiv.org/abs/2310.20700):\n\n**[SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR2024)](https://arxiv.org/abs/2310.20700)**\n\n**SEINE** is a video diffusion model and is part of the video generation system [Vchitect](http://vchitect.intern-ai.org.cn/). \nYou can also check our Text-to-Video (T2V) framework [LaVie](https://github.com/Vchitect/LaVie).\n\n\n\n\u003cimg src=\"https://github.com/Vchitect/SEINE/blob/main/seine.gif?raw=true\" width=\"800\"\u003e\n\n\n##  Setup\n\n### Prepare Environment\n```\nconda create -n seine python==3.9.16\nconda activate seine\npip install -r requirement.txt\n```\n\n### Download our model and T2I base model\n\nOur model is based on Stable diffusion v1.4, you may download [Stable Diffusion v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4) to the director of ``` pretrained ```\n.\nDownload our model checkpoint (from [google drive](https://drive.google.com/drive/folders/1cWfeDzKJhpb0m6HA5DoMOH0_ItuUY95b?usp=sharing) or [hugging face](https://huggingface.co/xinyuanc91/SEINE/tree/main)) and save to the directory of ```pretrained```\n\n\nNow under `./pretrained`, you should be able to see the following:\n```\n├── pretrained\n│   ├── seine.pt\n│   ├── stable-diffusion-v1-4\n│   │   ├── ...\n└── └── ├── ...\n        ├── ...\n```\n## Usage\n### Inference for I2V \nRun the following command to get the I2V results:\n```python\npython sample_scripts/with_mask_sample.py --config configs/sample_i2v.yaml\n```\nThe generated video will be saved in ```./results/i2v```.\n\n#### More Details\nYou may modify ```./configs/sample_i2v.yaml``` to change the generation conditions.\nFor example:\n\n```ckpt``` is used to specify a model checkpoint.\n\n```text_prompt``` is used to describe the content of the video.\n\n```input_path``` is used to specify the path to the image.\n\n### Inference for Transition\n```python\npython sample_scripts/with_mask_sample.py --config configs/sample_transition.yaml\n```\nThe generated video will be saved in ```./results/transition```.\n\n\n\n\n## Results\n### I2V Results\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd style=\"text-align:center;width: 50%\" colspan=\"1\"\u003e\u003cb\u003eInput Image\u003c/b\u003e\u003c/td\u003e\n  \u003ctd style=\"text-align:center;width: 50%\" colspan=\"1\"\u003e\u003cb\u003eOutput Video\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"https://github.com/Vchitect/SEINE-project/blob/main/static/image-animation/more_results/Close-up_essence_is_poured_from_bottleKodak_Vision.png?raw=true\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"examples/Close-up essence is poured from bottleKodak Vision3 50,slow motion_0000_001.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"input/i2v/The_picture_shows_the_beauty_of_the_sea_and_at_the_same.png\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"examples/The picture shows the beauty of the sea and at the sam,slow motion_0000_11301.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"input/i2v/The_picture_shows_the_beauty_of_the_sea.png\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"examples/The picture shows the beauty of the sea and at the sam,slow motion_0000_6600.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\n### Transition Results\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003ctd style=\"text-align:center;width: 66%\" colspan=\"2\"\u003e\u003cb\u003eInput Images\u003c/b\u003e\u003c/td\u003e\n  \u003ctd style=\"text-align:center;width: 33%\" colspan=\"1\"\u003e\u003cb\u003eOutput Video\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"https://vchitect.github.io/SEINE-project/static/diverse/reference-scene/1-Close-up%20shot%20of%20a%20blooming%20cherry%20tree,%20realism-1.png\" \u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"https://vchitect.github.io/SEINE-project/static/diverse/reference-scene/2-Wide%20angle%20shot%20of%20an%20alien%20planet%20with%20cherry%20blossom%20forest-2.png\" \u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"examples/Travel from Earth's spring blossoms to the alien cherry blossom forestssmooth transition, slow motion_0000_003.gif\" \u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"https://vchitect.github.io/SEINE-project/static/transition/spiderman/spiderman.png\" \u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"https://vchitect.github.io/SEINE-project/static/transition/spiderman/sand.png\" \u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"examples/spiderman-becomes-a-sand-sculpture.gif\" \u003e\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n## BibTeX\n```bibtex\n@inproceedings{chen2023seine,\n  title={Seine: Short-to-long video diffusion model for generative transition and prediction},\n  author={Chen, Xinyuan and Wang, Yaohui and Zhang, Lingjun and Zhuang, Shaobin and Ma, Xin and Yu, Jiashuo and Wang, Yali and Lin, Dahua and Qiao, Yu and Liu, Ziwei},\n  booktitle={ICLR},\n  year={2023}\n}\n```\n\n```bibtex\n@article{wang2023lavie,\n  title={LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models},\n  author={Wang, Yaohui and Chen, Xinyuan and Ma, Xin and Zhou, Shangchen and Huang, Ziqi and Wang, Yi and Yang, Ceyuan and He, Yinan and Yu, Jiashuo and Yang, Peiqing and others},\n  journal={IJCV},\n  year={2024}\n}\n```\n\n## Disclaimer\nWe disclaim responsibility for user-generated content. The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model's capabilities. It is prohibited for pornographic, violent and bloody content generation, and to generate content that is demeaning or harmful to people or their environment, culture, religion, etc. Users are solely liable for their actions. The project contributors are not legally affiliated with, nor accountable for users' behaviors. Use the generative model responsibly, adhering to ethical and legal standards.\n\n## Contact Us\n**Xinyuan Chen**: [chenxinyuan@pjlab.org.cn](mailto:chenxinyuan@pjlab.org.cn)\n**Yaohui Wang**: [wangyaohui@pjlab.org.cn](mailto:wangyaohui@pjlab.org.cn)  \n\n## Acknowledgements\nThe code is built upon [LaVie](https://github.com/Vchitect/LaVie), [diffusers](https://github.com/huggingface/diffusers) and [Stable Diffusion](https://github.com/CompVis/stable-diffusion), we thank all the contributors for open-sourcing. \n\n\n## License\nThe code is licensed under Apache-2.0, model weights are fully open for academic research and also allow **free** commercial usage. To apply for a commercial license, please contact vchitect@pjlab.org.cn.\n","funding_links":[],"categories":["Python","Poster"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVchitect%2FSEINE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVchitect%2FSEINE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVchitect%2FSEINE/lists"}