Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/IamCreateAI/Ruyi-Models

Last synced: about 1 month ago
JSON representation
Host: GitHub
URL: https://github.com/IamCreateAI/Ruyi-Models
Owner: IamCreateAI
License: apache-2.0
Created: 2024-12-16T12:20:44.000Z (about 1 month ago)
Default Branch: main
Last Pushed: 2024-12-16T12:46:16.000Z (about 1 month ago)
Last Synced: 2024-12-16T13:54:52.149Z (about 1 month ago)
Language: Python
Size: 6.04 MB
Stars: 2
Watchers: 4
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

ai-game-devtools - Ruyi - to-video model capable of generating cinematic-quality videos at a resolution of 768, with a frame rate of 24 frames per second, totaling 5 seconds and 120 frames. | | | Video | (<span id="video">Video</span> / <span id="tool">Tool (AI LLM)</span>)
awesome-comfyui - **ComfyUI-Ruyi** - to-video model by CreateAI. (All Workflows Sorted by GitHub Stars)
README

        # Ruyi-Models

English | [简体中文](./README_CN.md)

Welcome to Ruyi-Models!

Ruyi is an image-to-video model capable of generating cinematic-quality videos at a **resolution of 768**, with a frame rate of **24 frames per second**, totaling **5 seconds and 120 frames**. It supports **lens control** and **motion amplitude control**. Using a **RTX 3090 or RTX 4090**, you can generate 512 resolution, 120 frames (or 768 resolution, ~72 frames) videos **without any loss of quality**.

## Table of Contents

- [Installation Instructions](#installation-instructions)

- [Download Model (Optional)](#download-model-optional)

- [How to Use](#how-to-use)

- [Showcase](#showcase)

- [GPU Memory Optimization](#gpu-memory-optimization)

- [License](#license)

## Installation Instructions

The installation instructions are simple. Just clone the repo and install the requirements.

```shell

git clone https://github.com/IamCreateAI/Ruyi-Models

cd Ruyi-Models

pip install -r requirements.txt

```

### For ComfyUI Users

#### Method (1): Installation via ComfyUI Manager

Download and install [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager).

```shell

cd ComfyUI/custom_nodes/

git clone https://github.com/ltdrdata/ComfyUI-Manager.git

# install requirements

pip install -r ComfyUI-Manager/requirements.txt

```

Next, start ComfyUI and open the Manager. Select Custom Nodes Manager, then search for "Ruyi". You should see ComfyUI-Ruyi as shown in the screenshot below. Click "Install" to proceed.



  



Finally, search for "ComfyUI-VideoHelperSuite" and install it as well.

#### Method (2): Manual Installation

Download and save this repository to the path *ComfyUI/custom_nodes/Ruyi-Models*.

```shell

# download the repo

cd ComfyUI/custom_nodes/

git clone https://github.com/IamCreateAI/Ruyi-Models.git

# install requirements

pip install -r Ruyi-Models/requirements.txt

```

Install the dependency [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite) to display video output (skip this step if already installed).

```shell

# download ComfyUI-VideoHelperSuite

cd ComfyUI/custom_nodes/

git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git

# install requirements

pip install -r ComfyUI-VideoHelperSuite/requirements.txt

```

##### For Windows Users

When using the Windows operating system, a common distribution is [ComfyUI_windows_portable_nvidia](https://github.com/comfyanonymous/ComfyUI/releases). When launched with `run_nvidia_gpu.bat`, it utilizes the embedded Python interpreter included with the package. Therefore, the environment needs to be set up within this built-in Python.

For example, if the extracted directory of the distribution is ComfyUI_windows_portable, you can typically use the following command to download the repository and install the runtime environment:

```shell

# download the repo

cd ComfyUI_windows_portable\ComfyUI\custom_nodes

git clone https://github.com/IamCreateAI/Ruyi-Models.git

# install requirements using embedded Python interpreter

..\..\python_embeded\python.exe -m pip install -r Ruyi-Models\requirements.txt

```

## Download Model (Optional)

Download the model and save it to certain path. To directly run our model, it is recommand to save the models into _Ruyi-Models/models_ folder. For ComfyUI users, the path should be _ComfyUI/models/Ruyi_.

| Model Name | Type | Resolution | Max Frames | Frames per Second | Storage Space | Download |

| :---: | :---: | :---: | :---: | :---: | :---: | :---: |

| Ruyi-Mini-7B | Image to Video | 512 & 768 | 120 | 24 | 17 GB | [🤗](https://huggingface.co/IamCreateAI/Ruyi-Mini-7B) |

For example, after downloading Ruyi-Mini-7B, the file path structure should be:

```

📦 Ruyi-Models/models/ or ComfyUI/models/Ruyi/

├── 📂 Ruyi-Mini-7B/

│   ├── 📂 transformers/

│   ├── 📂 vae/

│   └── 📂 ...

```

> This repository **supports automatic model downloading**, but manual downloading provides more control. For instance, you can download the model to another location and then link it to the *ComfyUI/models/Ruyi* path using symbolic links or similar methods.

## How to Use

We provide two ways to run our model. The first is directly using python code.

```

python3 predict_i2v.py

```

Specifically, the script downloads the model to the _Ruyi-Models/models_ folder and uses images from the [_assets_](./assets/) folder as the start and end frames for video inference. You can modify the variables in the script to replace the input images and set parameters such as video length and resolution.

For users with more than 24GB of GPU memory, you can use predict_i2v_80g.py to enhance generation speed. For those with less GPU memory, we offer parameters to optimize memory usage, enabling the generation of higher resolution and longer videos by extending the inference time. The effects of these parameters can be found in the [GPU memory optimization section](#gpu-memory-optimization) section below.

Or use ComfyUI wrapper in our github repo, the detail of ComfyUI nodes is described in [_comfyui/README.md_](./comfyui/README.md).

## Showcase

### Image to Video Effects

    

        

        

        

        

    

### Camera Control

    

        input

        left

        right

    

    

        static

        up

        down

    

### Motion Amplitude Control

    

        motion 1

        motion 2

        motion 3

        motion 4

    

## GPU Memory Optimization

We provide the options **`GPU_memory_mode` and `GPU_offload_steps` to reduce GPU memory usage**, catering to different user needs.

Generally speaking, using **less GPU memory requires more RAM and results in longer generation times**. Below is a reference table of expected GPU memory usage and generation times. Note that, the GPU memory reported below is the `max_memory_allocated()` value. The values read from nvidia-smi may be higher than the reported values because CUDA occupies some GPU memory (usually between 500 - 800 MiB), and PyTorch's caching mechanism also requests additional GPU memory.

### A100 Results

- Resolution of 512

| Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps |

| :---: | :---: | :---: | :---: | :---: | :---: | :---: |

| 24 frames  | 16119MiB 
 _01:01s_ | 15535MiB 
 _01:07s_ | 15340MiB 
 _01:13s_ | 15210MiB 
 _01:20s_ | 14950MiB 
 _01:32s_ |  4216MiB 
 _05:14s_ |

| 48 frames  | 18398MiB 
 _01:53s_ | 17230MiB 
 _02:15s_ | 16840MiB 
 _02:29s_ | 16580MiB 
 _02:32s_ | 16060MiB 
 _02:54s_ |  4590MiB 
 _09:59s_ |

| 72 frames  | 20678MiB 
 _03:00s_ | 18925MiB 
 _03:31s_ | 18340MiB 
 _03:53s_ | 17951MiB 
 _03:57s_ | 17171MiB 
 _04:25s_ |  6870MiB 
 _14:42s_ |

| 96 frames  | 22958MiB 
 _04:11s_ | 20620MiB 
 _04:54s_ | 19841MiB 
 _05:10s_ | 19321MiB 
 _05:14s_ | 18281MiB 
 _05:47s_ |  9150MiB 
 _19:17s_ |

| 120 frames | 25238MiB 
 _05:42s_ | 22315MiB 
 _06:34s_ | 21341MiB 
 _06:59s_ | 20691MiB 
 _07:07s_ | 19392MiB 
 _07:41s_ | 11430MiB 
 _24:08s_ |

- Resolution of 768

| Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps |

| :---: | :---: | :---: | :---: | :---: | :---: | :---: |

| 24 frames  | 18971MiB 
 _02:06s_ | 17655MiB 
 _02:40s_ | 17217MiB 
 _02:39s_ | 16925MiB 
 _02:41s_ | 16339MiB 
 _03:13s_ |  5162MiB 
 _13:42s_ |

| 48 frames  | 24101MiB 
 _04:52s_ | 21469MiB 
 _05:44s_ | 20592MiB 
 _05:51s_ | 20008MiB 
 _06:00s_ | 18837MiB 
 _06:49s_ | 10292MiB 
 _20:58s_ |

| 72 frames  | 29230MiB 
 _08:24s_ | 25283MiB 
 _09:45s_ | 25283MiB 
 _09:45s_ | 23091MiB 
 _10:10s_ | 21335MiB 
 _11:10s_ | 15421MiB 
 _39:12s_ |

| 96 frames  | 34360MiB 
 _12:49s_ | 29097MiB 
 _14:41s_ | 27343MiB 
 _15:33s_ | 26174MiB 
 _15:44s_ | 23834MiB 
 _16:33s_ | 20550MiB 
 _43:47s_ |

| 120 frames | 39489MiB 
 _18:21s_ | 32911MiB 
 _20:39s_ | 30719MiB 
 _21:34s_ | 29257MiB 
 _21:48s_ | 26332MiB 
 _23:02s_ | 25679MiB 
 _63:01s_ |

### RTX 4090 Results

The values marked with `---` in the table indicate that an out-of-memory (OOM) error occurred, preventing generation.

- Resolution of 512

| Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps |

| :---: | :---: | :---: | :---: | :---: | :---: | :---: |

| 24 frames  | 16366MiB 
 _01:18s_ | 15805MiB 
 _01:26s_ | 15607MiB 
 _01:37s_ | 15475MiB 
 _01:36s_ | 15211MiB 
 _01:39s_ |  4211MiB 
 _03:57s_ |

| 48 frames  | 18720MiB 
 _02:21s_ | 17532MiB 
 _02:49s_ | 17136MiB 
 _02:55s_ | 16872MiB 
 _02:58s_ | 16344MiB 
 _03:01s_ |  4666MiB 
 _05:01s_ |

| 72 frames  | 21036MiB 
 _03:41s_ | 19254MiB 
 _04:25s_ | 18660MiB 
 _04:34s_ | 18264MiB 
 _04:36s_ | 17472MiB 
 _04:51s_ |  6981MiB 
 _06:36s_ |

| 96 frames  | -----MiB 
 _--:--s_ | 20972MiB 
 _06:18s_ | 20180MiB 
 _06:24s_ | 19652MiB 
 _06:36s_ | 18596MiB 
 _06:56s_ |  9298MiB 
 _10:03s_ |

| 120 frames | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | 21704MiB 
 _08:50s_ | 21044MiB 
 _08:53s_ | 19724MiB 
 _09:08s_ | 11613MiB 
 _13:57s_ |

- Resolution of 768

| Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps |

| :---: | :---: | :---: | :---: | :---: | :---: | :---: |

| 24 frames  | 19223MiB 
 _02:38s_ | 17900MiB 
 _03:06s_ | 17448MiB 
 _03:18s_ | 17153MiB 
 _03:23s_ | 16624MiB 
 _03:34s_ |  5251MiB 
 _05:54s_ |

| 48 frames  | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | 20946MiB 
 _07:28s_ | 20352MiB 
 _07:35s_ | 19164MiB 
 _08:04s_ | 10457MiB 
 _10:55s_ |

| 72 frames  | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | -----MiB 
 _--:--s_ | 15671MiB 
 _18:52s_ |

## License

We’re releasing the model under a permissive **Apache 2.0 license**.

## BibTeX

```

@misc{createai2024ruyi,

      title={Ruyi-Mini-7B},

      author={CreateAI Team},

      year={2024},

      publisher = {GitHub},

      journal = {GitHub repository},

      howpublished={\url{https://github.com/IamCreateAI/Ruyi-Models}}

}

```

## Welcome Feedback and Collaborative Optimization

We sincerely welcome everyone to actively provide valuable feedback and suggestions, and we hope to work together to optimize our services and products. Your words will help us better understand user needs, allowing us to continuously enhance the user experience. Thank you for your support and attention to our work!

You are welcomed to join our [Discord](https://discord.com/invite/nueQFQwwGw) or Wechat Group (Scan QR code) for further discussion!

![wechat](assets/wechat_group.png)