https://github.com/omerbt/tokenflow
Official Pytorch Implementation for "TokenFlow: Consistent Diffusion Features for Consistent Video Editing" presenting "TokenFlow" (ICLR 2024)
https://github.com/omerbt/tokenflow
iclr2024 stable-diffusion text-to-image text-to-video tokenflow video-editing
Last synced: 5 months ago
JSON representation
Official Pytorch Implementation for "TokenFlow: Consistent Diffusion Features for Consistent Video Editing" presenting "TokenFlow" (ICLR 2024)
- Host: GitHub
- URL: https://github.com/omerbt/tokenflow
- Owner: omerbt
- License: mit
- Created: 2023-07-20T16:40:00.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2025-02-03T15:34:18.000Z (8 months ago)
- Last Synced: 2025-04-13T16:50:24.232Z (6 months ago)
- Topics: iclr2024, stable-diffusion, text-to-image, text-to-video, tokenflow, video-editing
- Language: Python
- Homepage: https://diffusion-tokenflow.github.io
- Size: 27.4 MB
- Stars: 1,649
- Watchers: 74
- Forks: 139
- Open Issues: 41
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024)
## [Project Page]
[](https://arxiv.org/abs/2307.10373) [](https://huggingface.co/spaces/weizmannscience/tokenflow)
[//]: # ([](https://replicate.com/cjwbw/multidiffusion))
[//]: # ([](https://huggingface.co/spaces/weizmannscience/text2live))
https://github.com/omerbt/TokenFlow/assets/52277000/93dccd63-7e9a-4540-a941-31962361b0bb
**TokenFlow** is a framework that enables consistent video editing, using a pre-trained text-to-image diffusion model, without any further training or finetuning.
[//]: # (as described in (link to paper).)
[//]: # (. It can be used for localized and global edits that change the texture of existing objects or augment the scene with semi-transparent effects (e.g. smoke, fire, snow).)
[//]: # (### Abstract)
>The generative AI revolution has been recently expanded to videos. Nevertheless, current state-of-the-art video mod- els are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial lay- out and dynamics of the input video. Our method is based on our key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in con- junction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.
For more see the [project webpage](https://diffusion-tokenflow.github.io).
## Sample results
## Environment
```
conda create -n tokenflow python=3.9
conda activate tokenflow
pip install -r requirements.txt
```
## Preprocess
Preprocess you video by running using the following command:
```
python preprocess.py --data_path \
--inversion_prompt <'' or a string describing the video content>
```
Additional arguments:
```
--save_dir
--H
--W
--sd_version
--steps
--save_steps
--n_frames
```
more information on the arguments can be found here.
### Note:
The video reconstruction will be saved as inverted.mp4. A good reconstruction is required for successfull editing with our method.
## Editing
- TokenFlow is designed for structure-preserving edits.
- Our method is built on top of an image editing technique (e.g., Plug-and-Play, ControlNet, etc.) - therefore, it is important to ensure that the edit works with the chosen base technique.
- The LDM decoder may introduce some jitterness, depending on the original video.
To edit your video, first create a yaml config as in ``configs/config_pnp.yaml``.
Then run
```
python run_tokenflow_pnp.py
```
Similarly, if you want to use ControlNet or SDEedit, create a yaml config as in ``config/config_controlnet.yaml`` or ```configs/config_SDEdit.yaml``` and run ```python run_tokenflow_controlnet.py``` or ``python run_tokenflow_SDEdit.py`` respectivly.
## Citation
```
@article{tokenflow2023,
title = {TokenFlow: Consistent Diffusion Features for Consistent Video Editing},
author = {Geyer, Michal and Bar-Tal, Omer and Bagon, Shai and Dekel, Tali},
journal={arXiv preprint arxiv:2307.10373},
year={2023}
}
```