https://github.com/yvann-ba/ComfyUI_Yvann-Nodes
Audio Reactivity Nodes for ComfyUI 🔊 Create AI generated audio-driven animations. Compatible with IPAdapter, ControlNets, AnimateDiff...
https://github.com/yvann-ba/ComfyUI_Yvann-Nodes
audioreactive comfyui generative-ai
Last synced: 4 months ago
JSON representation
Audio Reactivity Nodes for ComfyUI 🔊 Create AI generated audio-driven animations. Compatible with IPAdapter, ControlNets, AnimateDiff...
- Host: GitHub
- URL: https://github.com/yvann-ba/ComfyUI_Yvann-Nodes
- Owner: yvann-ba
- License: gpl-3.0
- Created: 2024-08-26T20:52:14.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-11-19T21:42:23.000Z (5 months ago)
- Last Synced: 2024-12-14T13:13:07.867Z (4 months ago)
- Topics: audioreactive, comfyui, generative-ai
- Language: Python
- Homepage:
- Size: 41.8 MB
- Stars: 284
- Watchers: 3
- Forks: 9
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-comfyui - **ComfyUI_Yvann-Nodes** - driven visuals. Produce weight graphs and audio masks. Compatible with IPAdapter, ControlNets and more. Features audio scheduling and waveform analysis. Tutorials to use this pack: [a/Yvann Youtube](https://www.youtube.com/@yvann.mp4) (All Workflows Sorted by GitHub Stars)
README
# 👁️ ComfyUI_Yvann-Nodes [](https://www.instagram.com/yvann.mp4/) [](https://www.youtube.com/channel/yvann.mp4)
#### Made with the help of [Lilien](https://x.com/Lilien_RIG) 
#### **With this pack of nodes, you can analyze audio, extract drums, bass, vocal tracks, and use the scheduled masks and weights to create AI-generated audio-reactive animations in ComfyUI**
#### **Works with IPAdapter, AnimateDiff, ControlNets, Prompts Schedules**
- [Workflows](#Workflows)
- [Nodes](#Nodes)
- [Installation](#Installation)
---
## Workflows
### Audio Reactive Images To Video :
##### Workflow File (click on the link, then on "download raw file" button at the right and drop the file into ComfyUI) 📜
[AudioReactive_ImagesToVideo_Yvann.json](WORKFLOW_AUDIO_REACTIVE/AudioReactive_ImagesToVideo_Yvann.json)
##### Youtube Tutorial (Click on the image)
[](https://www.youtube.com/watch?v=O2s6NseXlMc)
##### Workflow Demo Render *(ENABLE VIDEO SOUND)* 🔊
https://github.com/user-attachments/assets/1e6590fc-e0d7-42d7-a205-433adf6c405c
>
> AudioReactive VideoToVideo
>
>##### Workflow File (click on the link, then on "download raw file" button at the right and drop the file into ComfyUI) 📜
>[AudioReactive_VideoToVideo_Yvann.json](./WORKFLOW_AUDIO_REACTIVE/AudioReactive_VideoToVideo_Yvann.json)
>
>##### Workflow Demo Render *(ENABLE VIDEO SOUND)* 🔊
>
>https://github.com/user-attachments/assets/6b0aa544-aa20-4257-b6be-28673082c7ef
>
>##### Youtube Tutorial (Click on the image)
>[](https://www.youtube.com/watch?v=BiQHWKP3q0c)
>##### Workflow Preview
>
>[CIVITAI Workflow Page](https://civitai.com/models/867298)
>
>
---
## Nodes
### Audio Analysis 🔍
Analyzes audio to generate reactive weights and graph. Can extract specific elements like drums, vocals, bass. Parameters allow manual control over audio weights
>
> Node Parameters
>
> - **audio_sep_model**: Loaded model from "Load Audio Separation Model"
> - **audio**: Input audio file
> - **batch_size**: Number of frames to associate with audio weights
> - **fps**: Frames per second for processing audio weights
>
> **Parameters:**
>
> - **analysis_mode**: Select audio component to analyze
> - **threshold**: Minimum weight value to pass through
> - **multiply**: Amplification factor for weights before normalization
>
> **Outputs:**
>
> - **graph_audio**: Graph image of audio weights over frames
> - **processed_audio**: Separated or processed audio (e.g., drums vocals)
> - **original_audio**: Original unmodified audio input
> - **audio_weights**: List of audio-reactive weights based on processed audio
>
>
---
### Load Audio Separation Model 🎧
Load an audio separation model, If unavailable downloads to `ComfyUI/models/audio_separation_model/
>
> Node Parameters
>
> - **model**: Audio separation model to load
> - [HybridDemucs](https://github.com/facebookresearch/demucs): Most accurate fastest and lightweight
> - [OpenUnmix](https://github.com/sigsep/open-unmix-pytorch): Alternative model
>
> **Outputs:**
>
> - **audio_sep_model**: Loaded audio separation model
> Connect it to "Audio Analysis" or "Audio Remixer"
>
---
### Audio Peaks Detection 📈
Detects peaks in audio weights based on a threshold and minimum distance. Identifies significant audio events to trigger visual changes or actions.
>
> Node Parameters
>
> - **peaks_threshold**: Threshold for peak detection
> - **min_peaks_distance**: Minimum frames between consecutive peaks help remove close unwanted peaks around big peaks
>
> **Outputs:**
>
> - **peaks_weights**: Binary list indicating peak presence (1 for peak 0 otherwise)
> - **peaks_alternate_weights**: Alternating binary list based on detected peaks
> - **peaks_index**: String of peak indices
> - **peaks_count**: Total number of detected peaks
> - **graph_peaks**: Visualization image of detected peaks over audio weights
>
>
---
### Audio IP Adapter Transitions 🔄
Uses "peaks_weights" from "Audio Peaks Detection" to control image transitions based on audio peaks. Outputs images and weights for two IPAdapter batches, logic from "IPAdapter Weights", [IPAdapter_Plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus)
>
> Node Parameters
> - **images**: Batch of images for transitions, Loops images to match peak count
> - **peaks_weights**: List of audio peaks from "Audio Peaks Detection"
>
> **Parameters:**
>
> - **blend_mode**: transition method applied to weights
> - **transitions_length**: Frames used to blend between images
> - **min_IPA_weight**: Minimum weight applied by IPAdapter per frame
> - **max_IPA_weight**: Maximum weight applied by IPAdapter per frame
>
> **Outputs:**
>
> - **image_1**: Starting image for transition Connect to first IPAdapter batch "image"
> - **weights**: Blending weights for transitions Connect to first IPAdapter batch "weight"
> - **image_2**: Ending image for transition Connect to second IPAdapter batch "image"
> - **weights_invert**: Inversed weights Connect to second IPAdapter batch "weight"
> - **graph_transitions**: Visualization of weight transitions over frames
>
---
### Audio Prompt Schedule 📝
Associates "prompts" with "peaks_index" into a scheduled format. Connect output to "batch prompt schedule" of [Fizz Nodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes) add an empty line between each individual prompts
>
> Node Parameters
> - **peaks_index**: frames where peaks occurs from "Audio Peaks Detections"
> - **prompts**: Multiline string of prompts for each index
>
> **Outputs:**
>
> - **prompt_schedule**: String mapping each audio index to a prompt
>
---
### Audio Remixer 🎛️
Modify input audio by adjusting the intensity of drums bass vocals or others elements
>
> Node Parameters
> - **audio_sep_model**: Loaded model from "Load Audio Separation Model"
> - **audio**: Input audio file
>
> **Parameters:**
>
> - **bass_volume**: Adjusts bass volume
> - **drums_volume**: Adjusts drums volume
> - **others_volume**: Adjusts others elements' volume
> - **vocals_volume**: Adjusts vocals volume
>
> **Outputs:**
>
> - **merged_audio**: Composition of separated tracks with applied modifications
>
---
### Repeat Image To Count 🔁
Repeats images N times, Cycles inputs if N > images
>
> Node Parameters
> - **mask**: Mask input to convert
>
> **Outputs:**
>
> - **float**: Float value
>
---
### Invert Floats 🔄
Inverts each value in a list of floats
>
> Node Parameters
>
> - **floats**: List of float values to invert.
>
> **Outputs**:
> - **inverted_floats**: Inverted list of float values.
>
>
---
### Floats Visualizer 📈
Generates a graph from floats for visual data comparison
Useful to compare audio weights
>
> Node Parameters
>
> - **floats**: Primary list of floats to visualize
> - **floats_optional1**: (Optional) Second list of floats
> - **floats_optional2**: (Optional) Third list of floats
>
> **Parameters:**
>
> - **title**: Graph title
> - **x_label**: Label for the x-axis
> - **y_label**: Label for the y-axis
>
> **Outputs:**
>
> - **visual_graph**: Visual graph of provided floats
>
---
### Mask To Float 🎭
Converts mask into float works with batch of mask
>
> Node Parameters
>
> - **mask**: Mask input to convert.
>
> **Outputs**:
> - **float**: Float value representing the average value of the mask.
>
>
---
### Floats To Weights Strategy 🏋️
Converts a list of floats into an IPAdapter weights strategy format. Use with "IPAdapter Weights From Strategy" or "Prompt Schedule From Weights Strategy" to integrate output into [IPAdapter](https://github.com/cubiq/ComfyUI_IPAdapter_plus) pipeline
>
> Node Parameters
>
> **Inputs:**
>
> - **floats**: List of float values to convert
>
> **Outputs:**
>
> - **WEIGHTS_STRATEGY**: Dictionary of the weights strategy
---
### Installation
1. Install [ComfyUI](https://github.com/comfyanonymous/ComfyUI) & [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager)
2. Launch ComfyUI
3. Click on "🧩 Manager" -> "Custom Nodes Manager"
4. Search for `ComfyUI_Yvann-Nodes` in the manager and install it
---
#### *Giving a ⭐ to this repo is the best way to support us (:*