https://github.com/siliconflow/dit_latte
https://github.com/siliconflow/dit_latte
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/siliconflow/dit_latte
- Owner: siliconflow
- License: apache-2.0
- Created: 2024-06-02T08:14:07.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-12T07:07:24.000Z (10 months ago)
- Last Synced: 2025-01-10T20:42:10.534Z (4 months ago)
- Language: Python
- Size: 35.4 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Note this is a fork from: https://github.com/Vchitect/Latte
## Latte: Latent Diffusion Transformer for Video Generation
Official PyTorch Implementation
[](https://arxiv.org/abs/2401.03048)
[](https://maxin-cn.github.io/latte_project/)
[](https://openbayes.com/console/public/tutorials/UOeU0ywVxl7)
This repo contains PyTorch model definitions, pre-trained weights, and training/sampling code for our paper exploring
latent diffusion models with transformers (Latte). You can find more visualizations on our [project page](https://maxin-cn.github.io/latte_project/).
> [**Latte: Latent Diffusion Transformer for Video Generation**](https://maxin-cn.github.io/latte_project/)
> [Xin Ma](https://maxin-cn.github.io/), [Yaohui Wang*](https://wyhsirius.github.io/), [Xinyuan Chen](https://scholar.google.com/citations?user=3fWSC8YAAAAJ), [Gengyun Jia](https://scholar.google.com/citations?user=_04pkGgAAAAJ&hl=zh-CN), [Ziwei Liu](https://liuziwei7.github.io/), [Yuan-Fang Li](https://users.monash.edu/~yli/), [Cunjian Chen](https://cunjian.github.io/), [Yu Qiao](https://scholar.google.com.hk/citations?user=gFtI-8QAAAAJ&hl=zh-CN)
> (*Corresponding Author & Project Lead)
## News
- (🔥 New) **May 23, 2024** 💥 **Latte-1** is released! Pre-trained model can be downloaded [here](https://huggingface.co/maxin-cn/Latte-1/tree/main/transformer). **We support both T2V and T2I**. Please run `bash sample/t2v.sh` and `bash sample/t2i.sh` respectively.
- (🔥 New) **Feb 24, 2024** 💥 We are very grateful that researchers and developers like our work. We will continue to update our LatteT2V model, hoping that our efforts can help the community develop. Our Latte discord channel
is created for discussions. Coders are welcome to contribute.
- (🔥 New) **Jan 9, 2024** 💥 An updated LatteT2V model initialized with the [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha) is released, the checkpoint can be found [here](https://huggingface.co/maxin-cn/Latte-0/tree/main/transformer).
- (🔥 New) **Oct 31, 2023** 💥 The training and inference code is released. All checkpoints (including FaceForensics, SkyTimelapse, UCF101, and Taichi-HD) can be found [here](https://huggingface.co/maxin-cn/Latte/tree/main). In addition, the LatteT2V inference code is provided.
## Setup
First, download and set up the repo:
```bash
git clone https://github.com/Vchitect/Latte
cd Latte
```
We provide an [`environment.yml`](environment.yml) file that can be used to create a Conda environment. If you only want
to run pre-trained models locally on CPU, you can remove the `cudatoolkit` and `pytorch-cuda` requirements from the file.
```bash
conda env create -f environment.yml
conda activate latte
```
## Sampling
You can sample from our **pre-trained Latte models** with [`sample.py`](sample/sample.py). Weights for our pre-trained Latte model can be found [here](https://huggingface.co/maxin-cn/Latte). The script has various arguments to adjust sampling steps, change the classifier-free guidance scale, etc. For example, to sample from our model on FaceForensics, you can use:
```bash
bash sample/ffs.sh
```
or if you want to sample hundreds of videos, you can use the following script with Pytorch DDP:
```bash
bash sample/ffs_ddp.sh
```
If you want to try generating videos from text, just run `bash sample/t2v.sh`. All related checkpoints will download automatically.
## Training
We provide a training script for Latte in [`train.py`](train.py). This script can be used to train class-conditional and unconditional
Latte models. To launch Latte (256x256) training with `N` GPUs on the FaceForensics dataset
:
```bash
torchrun --nnodes=1 --nproc_per_node=N train.py --config ./configs/ffs/ffs_train.yaml
```
or If you have a cluster that uses slurm, you can also train Latte's model using the following scripts:
```bash
sbatch slurm_scripts/ffs.slurm
```
We also provide the video-image joint training scripts [`train_with_img.py`](train_with_img.py). Similar to [`train.py`](train.py) scripts, these scripts can be also used to train class-conditional and unconditional
Latte models. For example, if you want to train the Latte model on the FaceForensics dataset, you can use:
```bash
torchrun --nnodes=1 --nproc_per_node=N train_with_img.py --config ./configs/ffs/ffs_img_train.yaml
```
## Contact Us
**Yaohui Wang**: [[email protected]](mailto:[email protected])
**Xin Ma**: [[email protected]](mailto:[email protected])
## Citation
If you find this work useful for your research, please consider citing it.
```bibtex
@article{ma2024latte,
title={Latte: Latent Diffusion Transformer for Video Generation},
author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Liu, Ziwei and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
journal={arXiv preprint arXiv:2401.03048},
year={2024}
}
```
## Acknowledgments
Latte has been greatly inspired by the following amazing works and teams: [DiT](https://github.com/facebookresearch/DiT) and [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha), we thank all the contributors for open-sourcing.
## License
The code and model weights are licensed under [LICENSE](LICENSE).