Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ArtVentureX/comfyui-animatediff
AnimateDiff for ComfyUI
https://github.com/ArtVentureX/comfyui-animatediff
Last synced: about 2 months ago
JSON representation
AnimateDiff for ComfyUI
- Host: GitHub
- URL: https://github.com/ArtVentureX/comfyui-animatediff
- Owner: ArtVentureX
- License: apache-2.0
- Created: 2023-07-22T08:18:09.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-05-22T18:16:43.000Z (4 months ago)
- Last Synced: 2024-06-04T07:35:02.343Z (4 months ago)
- Language: Python
- Size: 121 KB
- Stars: 620
- Watchers: 10
- Forks: 39
- Open Issues: 24
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-comfyui - animatediff
README
# AnimateDiff for ComfyUI
[AnimateDiff](https://github.com/guoyww/AnimateDiff/) integration for ComfyUI, adapts from [sd-webui-animatediff](https://github.com/continue-revolution/sd-webui-animatediff). Please read the original repo README for more information.
## How to Use
1. Clone this repo into `custom_nodes` folder.
2. Download motion modules and put them under `comfyui-animatediff/models/`.
- Original modules: [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y)
- Community modules: [manshoety/AD_Stabilized_Motion](https://huggingface.co/manshoety/AD_Stabilized_Motion) | [CiaraRowles/TemporalDiff](https://huggingface.co/CiaraRowles/TemporalDiff)
- AnimateDiff v2 [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt)
## Update 2023/09/25
#### **Motion LoRA** is now supported!
Download [motion LoRAs](https://huggingface.co/guoyww/animatediff/tree/main) and put them under `comfyui-animatediff/loras/` folder.
Note: LoRAs only work with **AnimateDiff v2** [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt) module.
#### New node: `AnimateDiffLoraLoader`
Example workflow:
Workflow: [lora.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/lora.json)
Samples:
## Update 2023/09/21
#### **Sliding Window** is now available!
The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the `SlidingWindowOptions` node. See the [sample workflow](#long-duration-with-sliding-window) bellow.
## Nodes
#### AnimateDiffLoader
#### AnimateDiffSampler
- Mostly the same with `KSampler`
- `motion_module`: use `AnimateDiffLoader` to load the motion module
- `inject_method`: should left default
- `frame_number`: animation length
- `latent_image`: You can pass an `EmptyLatentImage`
- `sliding_window_opts`: custom sliding window options
#### AnimateDiffCombine
- Combine GIF frames and produce the GIF image
- `frame_rate`: number of frame per second
- `loop_count`: use 0 for infinite loop
- `save_image`: should GIF be saved to disk
- `format`: supports `image/gif`, `image/webp` (better compression), `video/webm`, `video/h264-mp4`, `video/h265-mp4`. To use video formats, you'll need [ffmpeg](https://ffmpeg.org/download.html) installed and available in **`PATH`**
#### SlidingWindowOptions
Custom sliding window options
- `context_length`: number of frame per _window_. Use **16** to get the best results. Reduce it if you have low VRAM.
- `context_stride`:
- 1: sampling every frame
- 2: sampling every frame then every second frame
- 3: sampling every frame then every second frame then every third frames
- ...
- `context_overlap`: overlap frames between each window slice
- `closed_loop`: make the GIF a closed loop, will add more sampling step
#### LoadVideo
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
- `frame_start`: Skip some begining frames and start at `frame_start`
- `frame_limit`: Only take `frame_limit` frames
## Workflows
### Simple txt2gif
Workflow: [simple.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/simple.json)
Samples:
### Long duration with sliding window
Workflow: [sliding-window.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/sliding-window.json)
Samples:
### Latent upscale
Upscale latent output using `LatentUpscale` then do a 2nd pass with `AnimateDiffSampler`.
Workflow: [latent-upscale.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/latent-upscale.json)
Samples:
### Using with ControlNet
You will need following additional nodes:
- [Kosinkadink/ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet): Apply different weight for each latent in batch
- [Fannovel16/comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux): ControlNet preprocessors
#### Animate with starting and ending images
- Use `LatentKeyframe` and `TimestampKeyframe` from [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet) to apply diffrent weights for each latent index.
- Use 2 controlnet modules for two images with weights reverted.
Workflow: [cn-2images.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/cn-2images.json)
Samples:
#### Using GIF as ControlNet input
Using a GIF (or video, or a list of images) as ControlNet input.
Workflow: [cn-vid2vid.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/cn-vid2vid.json)
Samples:
## Known Issues
### CUDA error: invalid configuration argument
It's an `xformers` bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with `--disable-xformers` when booting ComfyUI.
### GIF split into multiple scenes
Work around:
- Shorter your prompt and negative prompt
- Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
- Disable xformers with `--disable-xformers`
### GIF has Wartermark (especially when using mm_sd_v15)
See: https://github.com/continue-revolution/sd-webui-animatediff/issues/31
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.