https://github.com/bryandlee/Tune-A-Video

Unofficial implementation of Tune-A-Video
https://github.com/bryandlee/Tune-A-Video

Last synced: 3 months ago
JSON representation

Unofficial implementation of Tune-A-Video

• Host: GitHub
• URL: https://github.com/bryandlee/Tune-A-Video
• Owner: bryandlee
• Created: 2023-01-02T11:02:12.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-01-12T23:06:51.000Z (almost 2 years ago)
• Last Synced: 2024-06-30T09:17:27.700Z (5 months ago)
• Language: Python
• Homepage:
• Size: 3.13 MB
• Stars: 188
• Watchers: 11
• Forks: 14
• Open Issues: 6
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Tune-A-Video

Unofficial implementation of [Tune-A-Video](https://arxiv.org/abs/2212.11565)

![step_500_resized](https://user-images.githubusercontent.com/26464535/211755110-94ba1fbf-a55a-4f71-b665-b7ab2a26121c.gif)

### Training

```

accelerate launch train.py --config config/sample.yml

```

### Notes

* Learning rate and training steps in the [sample config](config/sample.yml) are different from the paper.

* The model seems to memorize a video clip very quickly. `prior_preservation` is added for the regularization.

* Requires > 20GB GPU memory with xformers enabled.

### References

* Pseudo 3D Conv & Temporal Attention: https://github.com/lucidrains/make-a-video-pytorch

* Training Code: https://github.com/huggingface/diffusers

* Video Source: https://youtu.be/RC-24Nfr7fc?t=100