https://github.com/SooLab/Free-Bloom

[NeurIPS 2023] Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
https://github.com/SooLab/Free-Bloom

free-bloom stable-diffusion text-to-video zero-shot

Last synced: 3 months ago
JSON representation

[NeurIPS 2023] Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator

• Host: GitHub
• URL: https://github.com/SooLab/Free-Bloom
• Owner: SooLab
• License: apache-2.0
• Created: 2023-09-22T06:13:57.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-03-18T12:48:05.000Z (11 months ago)
• Last Synced: 2024-07-24T01:59:58.875Z (7 months ago)
• Topics: free-bloom, stable-diffusion, text-to-video, zero-shot
• Language: Python
• Homepage:
• Size: 14.7 MB
• Stars: 81
• Watchers: 8
• Forks: 4
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-diffusion-categorized - [Code

README

        
# Free-Bloom

This repository is the official implementation of [Free-Bloom](https://arxiv.org/abs/2309.14494).

**[Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator](https://arxiv.org/abs/2309.14494)**

[Hanzhuo Huang∗]() , [Yufan Feng∗](), [Cheng Shi](https://chengshiest.github.io/), [Lan Xu](https://www.xu-lan.com/), [Jingyi Yu](https://vic.shanghaitech.edu.cn/vrvc/en/people/jingyi-yu/), [Sibei Yang†](https://faculty.sist.shanghaitech.edu.cn/yangsibei/)

*Equal contribution; †Corresponding Author

[![arXiv](https://img.shields.io/badge/arXiv-FreeBloom-b31b1b.svg)](https://arxiv.org/abs/2309.14494) ![Pytorch](https://img.shields.io/badge/PyTorch->=1.10.0-Red?logo=pytorch)

![image-20230924124604776](__assets__/teaser.png)

## Setup

### Requirements

```cmd

conda create -n fb python=3.10

conda activate fb

pip install -r requirements.txt

```

Installing [xformers](https://github.com/facebookresearch/xformers) is highly recommended for more efficiency and speed on GPUs. 

To enable xformers, set `enable_xformers_memory_efficient_attention=True` (default).

## Usage

### Generate

```cmd

python main.py --config configs/flowers.yaml

```

Change the path of diffusion models to your own for  the `pretrained_model_path` key in config yaml file.

## Results

**A Flower is blooming**

    

    

    

    

    

    

**Volcano eruption**

    

    

    

    

    

    

**A rainbow is forming**

    

    

    

    

    

    

## Citation

```

@article{freebloom,

	title={Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator},

	author={Huang, Hanzhuo and Feng, Yufan and Shi, Cheng and Xu, Lan and Yu, Jingyi and Yang, Sibei},

	journal={arXiv preprint arXiv:2309.14494},

	year={2023}

}

```