{"id":20161923,"url":"https://github.com/ailab-cvc/cv-vae","last_synced_at":"2025-04-12T23:29:19.205Z","repository":{"id":241993002,"uuid":"807517499","full_name":"AILab-CVC/CV-VAE","owner":"AILab-CVC","description":"[NeurIPS 2024] CV-VAE: A Compatible Video VAE for Latent Generative Video Models","archived":false,"fork":false,"pushed_at":"2024-12-04T03:54:06.000Z","size":86823,"stargazers_count":269,"open_issues_count":17,"forks_count":9,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-04T02:09:50.041Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://ailab-cvc.github.io/cvvae/index.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AILab-CVC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-29T08:52:47.000Z","updated_at":"2025-03-23T15:01:19.000Z","dependencies_parsed_at":"2024-12-18T12:02:20.144Z","dependency_job_id":"ee8fb18f-d576-424d-bb8b-442e62f8916a","html_url":"https://github.com/AILab-CVC/CV-VAE","commit_stats":{"total_commits":14,"total_committers":3,"mean_commits":4.666666666666667,"dds":0.1428571428571429,"last_synced_commit":"f513bc3f6ecfbfbf593c7a00b9e8b7b6baa5942b"},"previous_names":["ailab-cvc/cv-vae"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FCV-VAE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FCV-VAE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FCV-VAE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AILab-CVC%2FCV-VAE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AILab-CVC","download_url":"https://codeload.github.com/AILab-CVC/CV-VAE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248646802,"owners_count":21139075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T00:21:53.739Z","updated_at":"2025-04-12T23:29:19.167Z","avatar_url":"https://github.com/AILab-CVC.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eCV-VAE: A Compatible Video VAE for Latent \nGenerative Video Models\u003c/h1\u003e\n\n[Sijie Zhao](https://scholar.google.com/citations?user=tZ3dS3MAAAAJ) · [Yong Zhang*](https://yzhang2016.github.io/) · [Xiaodong Cun](https://vinthony.github.io/academic/) · [Shaoshu Yang]() · [Muyao Niu]()\n\n[Xiaoyu Li](https://xiaoyu258.github.io/) · [Wenbo Hu](https://wbhu.github.io/) · [Ying Shan](https://scholar.google.com/citations?user=4oXBp9UAAAAJ\u0026hl=en)\n\n\u003csup\u003e*\u003c/sup\u003eCorresponding Authors\n\n\n\u003ca href='https://ailab-cvc.github.io/cvvae/index.html'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-green'\u003e\u003c/a\u003e\n\u003ca href='https://arxiv.org/abs/2405.20279'\u003e\u003cimg src='https://img.shields.io/badge/Technique-Report-red'\u003e\u003c/a\u003e\n\n\n\u003c/div\u003e\n\n\u003e **TL; DR:** A video VAE for latent generative video models, which is compatible with pretrained image and video models, e.g., SD 2.1 and SVD\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/i2v_and_t2v_results.gif\"\u003e\n\u003c/p\u003e\n\n\n## News\n\n- [x] **2024-10-14** :hugs: We have updated the training code of CV-VAE.\n- [x] **2024-10-14**  We have released the [inference code](cvvae_sd3_inference_video.py) and [model weights](https://huggingface.co/AILab-CVC/CV-VAE/tree/main/vae3d_sd3) of CV-VAE-SD3 which is compatible with [SD3](stabilityai/stable-diffusion-3-medium) and [SD3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium).\n- [x] **2024-10-14**  We have updated the CV-VAE with better performance, please check [cv-vae-v1-1](https://huggingface.co/AILab-CVC/CV-VAE/tree/main/vae3d_v1-1).\n- [x] **2024-09-25**  CV-VAE is accepted by NeurIPS 2024.\n- [x] **2024-06-03**  We have released the [inference code](cvvae_inference_video.py) and [model weights](https://huggingface.co/AILab-CVC/CV-VAE/tree/main/vae3d) of CV-VAE.\n\n- [x] **2024-05-30**  We have updated the [arXiv preprint](https://arxiv.org/abs/2405.20279).\n\n## Usage\n\n### Dependencies\n- Python \u003e= 3.8 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux))\n- [PyTorch \u003e= 1.13.0](https://pytorch.org/)\n- NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads)\n\n\n### Video reconstruction\n\nDownload the model weight from [Hugging Face](https://huggingface.co/AILab-CVC/CV-VAE/tree/main)\n\n```bash\npython3 cvvae_inference_video.py \\\n  --vae_path MODEL_PATH \\\n  --video_path INPUT_VIDEO_PATH \\\n  --save_path VIDEO_SAVE_PATH \\\n  --height HEIGHT \\\n  --width WIDTH \n```\n\n\n## 😉 Citation\n```\n@article{zhao2024cvvae,\n  title={CV-VAE: A Compatible Video VAE for Latent Generative Video Models},\n  author={Zhao, Sijie and Zhang, Yong and Cun, Xiaodong and Yang, Shaoshu and Niu, Muyao and Li, Xiaoyu and Hu, Wenbo and Shan, Ying},\n  journal={https://arxiv.org/abs/2405.20279},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Failab-cvc%2Fcv-vae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Failab-cvc%2Fcv-vae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Failab-cvc%2Fcv-vae/lists"}