{"id":15036197,"url":"https://github.com/vchitect/latte","last_synced_at":"2025-05-14T10:05:19.110Z","repository":{"id":204618113,"uuid":"711118194","full_name":"Vchitect/Latte","owner":"Vchitect","description":"[TMLR 2025] Latte: Latent Diffusion Transformer for Video Generation. ","archived":false,"fork":false,"pushed_at":"2025-03-26T23:56:07.000Z","size":37235,"stargazers_count":1803,"open_issues_count":4,"forks_count":186,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-03-28T15:02:55.429Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Vchitect.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-28T09:14:10.000Z","updated_at":"2025-03-28T12:40:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"95449dbc-4f13-46cb-ba5d-b9f00fe764e1","html_url":"https://github.com/Vchitect/Latte","commit_stats":null,"previous_names":["maxin-cn/lavita","maxin-cn/diffusynth","vchitect/latte"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLatte","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLatte/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLatte/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vchitect%2FLatte/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Vchitect","download_url":"https://codeload.github.com/Vchitect/Latte/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246049630,"owners_count":20715511,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:30:29.392Z","updated_at":"2025-04-11T20:33:06.241Z","avatar_url":"https://github.com/Vchitect.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Latte: Latent Diffusion Transformer for Video Generation\u003cbr\u003e\u003csub\u003eOfficial PyTorch Implementation\u003c/sub\u003e\r\n\r\n\u003c!-- ### [Paper](https://arxiv.org/abs/2401.03048v1) | [Project Page](https://maxin-cn.github.io/latte_project/) --\u003e\r\n\r\n\u003c!-- [![arXiv](https://img.shields.io/badge/arXiv-2401.03048-b31b1b.svg)](https://arxiv.org/abs/2401.03048) --\u003e\r\n[![arXiv](https://img.shields.io/badge/arXiv-2401.03048-b31b1b.svg)](https://arxiv.org/abs/2401.03048)\r\n[![Project Page](https://img.shields.io/badge/Project-Website-blue)](https://maxin-cn.github.io/latte_project/)\r\n[![HF Demo](https://img.shields.io/static/v1?label=Demo\u0026message=OpenBayes%E8%B4%9D%E5%BC%8F%E8%AE%A1%E7%AE%97\u0026color=green)](https://openbayes.com/console/public/tutorials/UOeU0ywVxl7) \r\n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-yellow)](https://huggingface.co/spaces/maxin-cn/Latte-1)\r\n[![slack badge](https://img.shields.io/badge/Discord-join-blueviolet?logo=discord\u0026amp)](https://discord.gg/RguYqhVU92)\r\n\r\n[![Static Badge](https://img.shields.io/badge/Latte--1%20checkpoint%20(T2V)-HuggingFace-yellow?logoColor=violet%20Latte-1%20checkpoint)](https://huggingface.co/maxin-cn/Latte-1)\r\n[![Static Badge](https://img.shields.io/badge/Latte%20checkpoint%20-HuggingFace-yellow?logoColor=violet%20Latte%20checkpoint)](https://huggingface.co/spaces/maxin-cn/Latte-1)\r\n\r\nThis repo contains the PyTorch model definitions, pre-trained weights, and training/sampling/evaluation codes for our paper \r\nLatte: Latent Diffusion Transformer for Video Generation. \r\n\r\n\u003e [**Latte: Latent Diffusion Transformer for Video Generation**](https://maxin-cn.github.io/latte_project/)\u003cbr\u003e\r\n\u003e [Xin Ma](https://maxin-cn.github.io/), [Yaohui Wang*](https://wyhsirius.github.io/), [Xinyuan Chen](https://scholar.google.com/citations?user=3fWSC8YAAAAJ), [Gengyun Jia](https://scholar.google.com/citations?user=_04pkGgAAAAJ\u0026hl=zh-CN), [Ziwei Liu](https://liuziwei7.github.io/), [Yuan-Fang Li](https://users.monash.edu/~yli/), [Cunjian Chen](https://cunjian.github.io/), [Yu Qiao](https://scholar.google.com.hk/citations?user=gFtI-8QAAAAJ\u0026hl=zh-CN)\r\n\u003e (*Corresponding Author \u0026 Project Lead)\r\n\u003c!-- \u003e \u003cbr\u003eMonash University, Shanghai Artificial Intelligence Laboratory,\u003cbr\u003e NJUPT, S-Lab, Nanyang Technological University \r\n\r\nWe propose a novel Latent Diffusion Transformer, namely Latte, for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to text-to-video generation (T2V) task, where Latte achieves comparable results compared to recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation.\r\n\r\n ![The architecture of Latte](visuals/architecture.svg){width=20}\r\n --\u003e\r\n\r\n\u003c!--\r\n\u003cdiv align=\"center\"\u003e\r\n    \u003cimg src=\"visuals/architecture.svg\" width=\"650\"\u003e\r\n\u003c/div\u003e\r\n\r\nThis repository contains:\r\n\r\n* 🪐 A simple PyTorch [implementation](models/latte.py) of Latte\r\n* ⚡️ **Pre-trained Latte models** trained on FaceForensics, SkyTimelapse, Taichi-HD and UCF101 (256x256). In addition, we provide a T2V checkpoint (512x512). All checkpoints can be found [here](https://huggingface.co/maxin-cn/Latte/tree/main). \r\n\r\n* 🛸 A Latte [training script](train.py) using PyTorch DDP.\r\n--\u003e\r\n\r\n\u003cvideo controls loop src=\"https://github.com/Vchitect/Latte/assets/7929326/a650cd84-2378-4303-822b-56a441e1733b\" type=\"video/mp4\"\u003e\u003c/video\u003e\r\n\r\n## News\r\n- 🔥 **Mar 23, 2025** 💥 Latte is accepted by Transactions on Machine Learning Research (TMLR) 2025.\r\n\r\n- 🔥 **Jul 11, 2024** 💥 **Latte-1 is now integrated into [diffusers](https://huggingface.co/docs/diffusers/main/en/api/pipelines/latte). Thanks to [@yiyixuxu](https://github.com/yiyixuxu), [@sayakpaul](https://github.com/sayakpaul), [@a-r-r-o-w](https://github.com/a-r-r-o-w) and [@DN6](https://github.com/DN6).** You can easily run Latte using the following code. We also support inference with 4/8-bit quantization, which can reduce GPU memory from 17 GB to 9 GB. Please refer to this [tutorial](docs/latte_diffusers.md) for more information.\r\n\r\n```\r\n# Please update the version of diffusers at leaset to 0.30.0\r\nfrom diffusers import LattePipeline\r\nfrom diffusers.models import AutoencoderKLTemporalDecoder\r\nfrom torchvision.utils import save_image\r\nimport torch\r\nimport imageio\r\n\r\ntorch.manual_seed(0)\r\n\r\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\r\nvideo_length = 16 # 1 (text-to-image) or 16 (text-to-video)\r\npipe = LattePipeline.from_pretrained(\"maxin-cn/Latte-1\", torch_dtype=torch.float16).to(device)\r\n\r\n# Using temporal decoder of VAE\r\nvae = AutoencoderKLTemporalDecoder.from_pretrained(\"maxin-cn/Latte-1\", subfolder=\"vae_temporal_decoder\", torch_dtype=torch.float16).to(device)\r\npipe.vae = vae\r\n\r\nprompt = \"a cat wearing sunglasses and working as a lifeguard at pool.\"\r\nvideos = pipe(prompt, video_length=video_length, output_type='pt').frames.cpu()\r\n```\r\n\r\n- 🔥 **Jun 26, 2024** 💥 Latte is supported by [VideoSys](https://github.com/NUS-HPC-AI-Lab/VideoSys), which is a user-friendly, high-performance infrastructure for video generation.\r\n\r\n- 🔥 **May 23, 2024** 💥 **Latte-1** is released! Pre-trained model can be downloaded [here](https://huggingface.co/maxin-cn/Latte-1/tree/main/transformer). **We support both T2V and T2I**. Please run `bash sample/t2v.sh` and `bash sample/t2i.sh` respectively.\r\n\r\n\u003c!--\r\n\u003cdiv align=\"center\"\u003e\r\n    \u003cimg src=\"visuals/latteT2V.gif\" width=88%\u003e\r\n\u003c/div\u003e\r\n--\u003e\r\n\r\n- 🔥 **Feb 24, 2024** 💥 We are very grateful that researchers and developers like our work. We will continue to update our LatteT2V model, hoping that our efforts can help the community develop. Our Latte discord channel \u003ca href=\"https://discord.gg/RguYqhVU92\" style=\"text-decoration:none;\"\u003e\r\n\u003cimg src=\"https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png\" width=\"3%\" alt=\"\" /\u003e\u003c/a\u003e is created for discussions. Coders are welcome to contribute.\r\n\r\n- 🔥 **Jan 9, 2024** 💥 An updated LatteT2V model initialized with the [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha) is released, the checkpoint can be found [here](https://huggingface.co/maxin-cn/Latte-0/tree/main/transformer).\r\n\r\n- 🔥 **Oct 31, 2023** 💥 The training and inference code is released. All checkpoints (including FaceForensics, SkyTimelapse, UCF101, and Taichi-HD) can be found [here](https://huggingface.co/maxin-cn/Latte/tree/main). In addition, the LatteT2V inference code is provided.\r\n\r\n\r\n## Setup\r\n\r\nFirst, download and set up the repo:\r\n\r\n```bash\r\ngit clone https://github.com/Vchitect/Latte\r\ncd Latte\r\n```\r\n\r\nWe provide an [`environment.yml`](environment.yml) file that can be used to create a Conda environment. If you only want \r\nto run pre-trained models locally on CPU, you can remove the `cudatoolkit` and `pytorch-cuda` requirements from the file.\r\n\r\n```bash\r\nconda env create -f environment.yml\r\nconda activate latte\r\n```\r\n\r\n\r\n## Sampling \r\n\r\nYou can sample from our **pre-trained Latte models** with [`sample.py`](sample/sample.py). Weights for our pre-trained Latte model can be found [here](https://huggingface.co/maxin-cn/Latte).  The script has various arguments to adjust sampling steps, change the classifier-free guidance scale, etc. For example, to sample from our model on FaceForensics, you can use:\r\n\r\n```bash\r\nbash sample/ffs.sh\r\n```\r\n\r\nor if you want to sample hundreds of videos, you can use the following script with Pytorch DDP:\r\n\r\n```bash\r\nbash sample/ffs_ddp.sh\r\n```\r\n\r\nIf you want to try generating videos from text, just run `bash sample/t2v.sh`. All related checkpoints will download automatically.\r\n\r\nIf you would like to measure the quantitative metrics of your generated results, please refer to [here](docs/datasets_evaluation.md).\r\n\r\n## Training\r\n\r\nWe provide a training script for Latte in [`train.py`](train.py). The structure of the datasets can be found [here](docs/datasets_evaluation.md). This script can be used to train class-conditional and unconditional\r\nLatte models. To launch Latte (256x256) training with `N` GPUs on the FaceForensics dataset \r\n:\r\n\r\n```bash\r\ntorchrun --nnodes=1 --nproc_per_node=N train.py --config ./configs/ffs/ffs_train.yaml\r\n```\r\n\r\nor If you have a cluster that uses slurm, you can also train Latte's model using the following scripts:\r\n\r\n ```bash\r\nsbatch slurm_scripts/ffs.slurm\r\n```\r\n\r\nWe also provide the video-image joint training scripts [`train_with_img.py`](train_with_img.py). Similar to [`train.py`](train.py) scripts, these scripts can be also used to train class-conditional and unconditional\r\nLatte models. For example, if you want to train the Latte model on the FaceForensics dataset, you can use:\r\n\r\n```bash\r\ntorchrun --nnodes=1 --nproc_per_node=N train_with_img.py --config ./configs/ffs/ffs_img_train.yaml\r\n```\r\n\r\nIf you are familiar with `PyTorch Lightning`, you can also use the training script [`train_pl.py`](train_pl.py) and [`train_with_img_pl.py`](train_with_img_pl.py) provided by [@zhang.haojie](https://github.com/zhang-haojie),\r\n\r\n```bash\r\npython train_pl.py --config ./configs/ffs/ffs_train.yaml\r\n```\r\n\r\nor\r\n\r\n```bash\r\npython train_with_img_pl.py --config ./configs/ffs/ffs_img_train.yaml\r\n```\r\n\r\nThis script automatically detects available GPUs and uses distributed training.\r\n\r\n## Contact Us\r\n**Yaohui Wang**: [wangyaohui@pjlab.org.cn](mailto:wangyaohui@pjlab.org.cn)\r\n**Xin Ma**: [xin.ma1@monash.edu](mailto:xin.ma1@monash.edu)\r\n\r\n## Citation\r\nIf you find this work useful for your research, please consider citing it.\r\n```bibtex\r\n@article{ma2025latte,\r\n  title={Latte: Latent Diffusion Transformer for Video Generation},\r\n  author={Ma, Xin and Wang, Yaohui and Chen, Xinyuan and Jia, Gengyun and Liu, Ziwei and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},\r\n  journal={Transactions on Machine Learning Research},\r\n  year={2025}\r\n}\r\n```\r\n\r\n\r\n## Acknowledgments\r\nLatte has been greatly inspired by the following amazing works and teams: [DiT](https://github.com/facebookresearch/DiT) and [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha), we thank all the contributors for open-sourcing.\r\n\r\n\r\n## License\r\nThe code and model weights are licensed under [LICENSE](LICENSE).\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Flatte","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvchitect%2Flatte","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvchitect%2Flatte/lists"}