https://github.com/janiluuk/defora
An Audiovisual instrument controllable by Web UI / Midi Controller
https://github.com/janiluuk/defora
deforum
Last synced: about 4 hours ago
JSON representation
An Audiovisual instrument controllable by Web UI / Midi Controller
- Host: GitHub
- URL: https://github.com/janiluuk/defora
- Owner: janiluuk
- Created: 2023-08-07T10:31:55.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2026-05-31T11:06:37.000Z (3 days ago)
- Last Synced: 2026-05-31T12:20:43.686Z (3 days ago)
- Topics: deforum
- Language: JavaScript
- Homepage:
- Size: 121 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Roadmap: ROADMAP.md
Awesome Lists containing this project
README
# Defora — audio-visual instrument for Stable Diffusion Forge/Deforumation
Defora turns Stable Diffusion Forge + Deforumation into a playable instrument: live visuals, prompt morphing, camera motion, beat-synced controls, and a neon-styled web UI for performance.
https://defora.dudeisland.eu/
## Current Browser Tour
These screenshots were recaptured from the current web UI after generating fresh material in the **MOTION → Animation Sequencer** view. The shared preview therefore shows a real recent render instead of placeholder content.
LIVE
The stage-first performance view keeps the latest render front and center, with floating morph and modulation HUDs for quick changes during playback.
PROMPTS
Prompt morphing, img2img, and plugin surfaces stay grouped in one place so narrative control and prompt blending stay close to the live preview.
MOTION
The motion view exposes the XY performance pad and camera presets for fast spatial moves. Use the Animation Sequencer sub-tab for timeline edits, story status, and sequencer controls.
MODULATION
LFO cards, routed targets, and beat macros make modulation ownership visible at a glance so you can see what is driving each parameter.
LIBRARY
Library focuses on runs + frame inspection, with an optional fullscreen expand mode for browsing and review.
TOOLS
VideoSwarm and shared source browsing live under Tools so the Library view stays focused.
SETTINGS
Engine, controller, and GPU pool controls are available without leaving the same performance shell as the rest of the instrument.
MOTION — Animation Sequencer
Timeline, playhead, and story-generation status live under MOTION so spatial performance and sequencer edits stay in one tab.
STREAM: output routing, encoder destinations, and recording controls.
## ✨ Feature Highlights
### 🎹 Live Performance Interface
Control your AI video generation in real-time with multiple interface options:
Web UI — Browser-based Performance Dashboard
Neon-styled web interface with macro sliders, motion presets, and live parameter control
TUI — Terminal-based Control Center
Full-featured ncurses interface for terminal enthusiasts with ASCII preview and parameter control
### 🎨 Advanced Prompt Management
Morph between multiple prompts and control narrative flow:
Web UI: Visual prompt morphing with slots and blend control
TUI: Text-based prompt management with morph ranges
### 🎥 Camera Motion Control
Gamepad-style camera controls and curve editing:
Web UI: Visual motion curves with preset system
TUI: Multi-lane curve editor in your terminal
### 🎨 LoRA Management
Control and blend multiple LoRA models with crossfader:
Web UI: Dedicated LoRA browser tab with strength controls and A/B grouping
TUI: LoRA controls integrated within the prompts tab (F2)
### 🎵 Audio-Reactive Modulation
Sync your visuals to audio with beat-driven macros and LFOs:
Web UI: Audio waveform + upload spectrogram + live spectrum bars, LFO modulators, beat macros, and parameter mapping
TUI: Beat-synced parameter automation
### 🎛️ ControlNet Integration & Settings
Fine-tune your generation with ControlNet and render settings:
Web UI: ControlNet slots and engine configuration
TUI: Complete control from your terminal
### 🎼 MIDI Controller Support
Map any MIDI controller to live parameters:
- **Web UI**: Browser-based Web MIDI support
- **TUI**: CC mapping with learn mode
- Full parameter control via hardware knobs and faders
## Requires
- Moderately fast GPU (4070ti / 5060ti tested) with at least 12G VRAM
- Models that have either Lightning support or accompanying LORa that need 1-2 steps for a frame. SDXL Lightning is proven to be working fine
- 32GB ram is minimum, 64gb ram recommended
- Stable Diffusion Forge + Deforum extension from **https://github.com/Tok/sd-forge-deforum** and **https://github.com/lllyasviel/stable-diffusion-webui-forge**
- There is docker stack including these in the package but it is recommended to run on external node
## 📦 What's Inside
### Core Tools
- **`forge_cli`** — Model-aware txt2img/Deforum CLI with preset support and sensible defaults
- **`defora_tui`** — Full multi-tab ncurses interface (LIVE, PROMPTS, LORA, MOTION, AUDIO, CONTROLNET, SETTINGS)
- **`deforumation_cli_panel`** — Lightweight control panel with rebindable hotkeys (strength/CFG/noise/pan/zoom/rot/FOV)
- **`deforumation_dashboard`** — Curses dashboard mirroring the Deforumation GUI tabs
### Audio & Modulation
- **`audio_reactive_modulator`** — Map audio bands to mediator parameters; output schedules or stream live
- **Beat-synced macros** — Drive parameters with tempo-aligned modulation
### Run Management
- **`deforumation_runs_cli`** — Browse run manifests, set overrides, rerun/continue generations
- **`deforumation_request_dispatcher`** — Merge manifests/presets/overrides and execute runs
- **`monitor_cli`** — Tail latest frames and show live mediator values
### Streaming & Output
- **`stream_helper`** — Push rendered frames to RTMP/SRT/WHIP via ffmpeg
- **Web streaming stack** — Docker compose with ffmpeg encoder, Nginx/Node (HLS), RabbitMQ, and mediator bridge
- **Web UI** — Browser-based performance interface with WebSocket controls and HLS video
### Documentation
- **`docs/`** — Architecture, workflows, server targeting, schemas, troubleshooting guides
## 🚀 Quick Start
### 1. Clone and Install
```bash
git clone https://github.com/janiluuk/defora.git
cd defora
git submodule update --init --recursive
# or: ./scripts/clone_deforumation.sh
pip install -r requirements.txt
```
### 2. Start Stable Diffusion Forge
Start Forge with Deforum API enabled:
```bash
# Example: assuming Forge is installed in ~/stable-diffusion-webui-forge
cd ~/stable-diffusion-webui-forge
./webui.sh --deforum-api
```
### 3. Choose Your Interface
#### 🎹 Quick Generation (Command Line)
Generate images or animations with smart defaults:
```bash
# Single image
./forge_cli "a synthwave city at night"
# Animation (240 frames)
./forge_cli deforum -f 240 "surreal biomechanical cathedral"
# img2img (requires running Forge)
./forge_cli img2img --init-image ./ref.png "enhance details, cinematic" --denoising-strength 0.55
# Inpainting (mask image, same size as init recommended)
./forge_cli img2img --init-image ./scene.png --mask-image ./mask.png \
"object removal, clean background" --denoising-strength 0.72
```
#### 🎮 Live Performance (Web UI)
Full browser-based performance interface:
```bash
docker-compose up --build
# Open http://localhost:8080
```
**Features**: Live macro sliders, prompt morphing, motion curves, audio sync, MIDI control
#### 💻 Terminal Control (TUI)
Full-featured ncurses interface for terminal users:
```bash
./defora_tui
```
**Navigation**: F1–F7 to switch tabs (incl. **LORA** on F3), ←/→ to adjust parameters, Q to quit
#### 🎛️ Lightweight Panel (CLI)
Minimal control panel with hotkey bindings:
```bash
./deforumation_cli_panel --host 127.0.0.1 --port 8766
```
### 4. Advanced Tools
**Browse & Rerun Saved Generations:**
```bash
./deforumation_runs_cli
```
**Audio-Reactive Modulation:**
```bash
# Generate audio schedule
./audio_reactive_modulator --audio song.wav --fps 24 --output audio_mod.json
# Stream live to mediator
./audio_reactive_modulator --audio song.wav --fps 24 --live \
--mediator-host 127.0.0.1 --mediator-port 8766
# Optional post-pass on the generated schedule (module:function)
./audio_reactive_modulator --audio song.wav --fps 24 --output out.json \
--post-plugin defora_cli.plugins.audio_post:process
```
**Monitor Live Generation:**
```bash
./monitor_cli --frames runs//frames
```
**Stream to RTMP/SRT:**
```bash
./stream_helper start --source runs//frames \
--target rtmp://example/live/key --fps 24
```
## Key concepts
- **Audio-visual instrument**: Treat prompts, camera, and ControlNet as live parameters; drive them via CLI/TUI/Web or controllers.
- **Presets & manifests**: Deforum JSON presets and run manifests can be merged with CLI overrides. The runs TUI writes `*_request.json` that the dispatcher consumes.
- **Mediator control**: The panel, dashboard, and audio modulator talk to the mediator websocket so you can steer generation without the full UI.
- **Model-aware defaults**: `forge_cli` picks steps/CFG/sampler based on the active model (Flux/SDXL/SD1.5) and can auto-switch to Flux-schnell.
## Mediator (DeforumationQT)
- The DeforumationQT mediator is vendored under `deforumation/`. Run it to bridge SD-Forge/Deforum to the Deforumation UI and our CLI tools (panel/dashboard/audio modulator).
- See `docs/mediator_setup.md` for mediator startup steps and for installing the sd-forge Deforum bridge from `deforumation/Deforum_Version/sd-forge/`.
- Docker users: `docker-compose up --build mediator sd-forge` will start the mediator (ports 8765/8766) and a Forge container with the Deforumation-patched Deforum extension pre-installed (UI on port 7860).
## Environment knobs
- `FORGE_API_BASE` — Python CLI target Forge server (e.g., `http://192.168.2.101:7860`).
- `SD_FORGE_HOST` / `SD_FORGE_PORT` — **web stack** Forge hostname and port (Docker Compose defaults to `sd-forge:7860`).
- `FORGE_OUT_DIR` — output directory for `forge_cli`.
- `DEFORUMATION_FORGE_CLI` — path override for Forge CLI when auto-dispatching.
- `DEFORUMATION_AUTO_DISPATCH=1` — have `deforumation_runs_cli` immediately dispatch requests.
- `DEFORUMATION_FRAMES_DIR` — default frames path for `monitor_cli`.
- `DEFORUMATION_ASCII_PREVIEW=1` — enable ASCII thumbnails in monitor/runs TUI (needs Pillow).
- `DEFORUMATION_MEDIATOR_HOST`/`DEFORUMATION_MEDIATOR_PORT` — defaults for mediator-backed UIs (panel, dashboard, monitor).
- `CONTROL_TOKEN` — WebSocket control token for the web UI (set when running docker-compose).
- `MEDIATOR_HOST` (compose bridge) — set this if `host.docker.internal` is not available on your host (common on Linux) so the control bridge can reach the mediator.
- `SD_FORGE_POLL_MS` — if set to a positive value (milliseconds) on the **web** stack, the API periodically probes SD-Forge so `/api/status` and the Web UI Forge indicator stay current without opening model list endpoints.
- `SEQUENCER_DIR` — directory for saved animation sequencer timelines (Web UI MOTION tab); default is `docker/web/sequencers` next to the Node server.
- `DEFORA_TUI_LORA_STATE` — optional path for `defora_tui` LoRA tab save/load (JSON); default `~/.config/defora/tui_lora.json`.
- `PLUGINS_DIR` — optional directory for `docker/web/plugins/manifest.json` (Web `GET /api/plugins`). CLI post-plugins use a separate registry: `defora_cli/plugins/manifest.json` (see `defora_cli.plugins`).
- Web MIDI: enable in your browser and map controls in the **Settings** tab (Controllers / WebMIDI) to live parameters.
## Project Status & Roadmap
See [ROADMAP.md](ROADMAP.md) for:
- Current feature status and completeness
- Unfinished/in-progress features
- Planned features and timeline
- Long-term vision and goals
## Deploy (production)
Same pattern as Sparkki: rsync + remote Docker Compose via jumphost `pi@sparkki.dudeisland.eu:4322` → `root@192.168.2.100:/srv/defora`.
```bash
./scripts/production-deploy.sh
```
GitHub Actions: [`.github/workflows/deploy-production.yml`](.github/workflows/deploy-production.yml) (after green CI on `main`, or manual dispatch). Requires secret `DEPLOY_SSH_PRIVATE_KEY`. See [`docs/deploy.md`](docs/deploy.md).
## Layout
- CLI/package code: `defora_cli/`
- Executable wrappers: `./forge_cli`, `./deforumation_request_dispatcher`, `./deforumation_runs_cli`, `./deforumation_cli_panel`, `./deforumation_dashboard`, `./defora_tui`, `./monitor_cli`, `./stream_helper`, `./audio_reactive_modulator`
- Web UI & streaming stack: `docker-compose.yml`, `docker/web/` (Nginx+Node+Vue front-end, HLS, controls), `docker/bridge/` (mediator bridge)
- Docs: `docs/` (workflows, server targeting, schema, streaming)
- Tests: `tests/` (Python) and `docker/web/test` (web UI smoke tests)
- Logo: `assets/defora_logo.svg` (dark-mode friendly, neon gradient)
## Testing
Run the suite (requires pytest installed):
```bash
./scripts/run_tests.sh # or: python -m pytest
```