https://github.com/wangjiangshan0725/RF-Solver-Edit
Taming FLUX for Image Inversion & Editing; OpenSora for Video Inversion & Editing! (Official implementation for Taming Rectified Flow for Inversion and Editing.)
https://github.com/wangjiangshan0725/RF-Solver-Edit
diffusion-transformer flux image-editing image-inversion opensora rectified-flow video-editing video-inversion
Last synced: 16 days ago
JSON representation
Taming FLUX for Image Inversion & Editing; OpenSora for Video Inversion & Editing! (Official implementation for Taming Rectified Flow for Inversion and Editing.)
- Host: GitHub
- URL: https://github.com/wangjiangshan0725/RF-Solver-Edit
- Owner: wangjiangshan0725
- License: apache-2.0
- Created: 2024-11-05T07:48:09.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-12-05T08:17:43.000Z (10 months ago)
- Last Synced: 2024-12-05T09:26:36.038Z (10 months ago)
- Topics: diffusion-transformer, flux, image-editing, image-inversion, opensora, rectified-flow, video-editing, video-inversion
- Language: Python
- Homepage: https://rf-solver-edit.github.io/
- Size: 36 MB
- Stars: 282
- Watchers: 10
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
# Taming Rectified Flow for Inversion and Editing
[Jiangshan Wang](https://scholar.google.com/citations?user=HoKoCv0AAAAJ&hl=en)1,2, [Junfu Pu](https://pujunfu.github.io/)2, [Zhongang Qi](https://scholar.google.com/citations?hl=en&user=zJvrrusAAAAJ&view_op=list_works&sortby=pubdate)2, [Jiayi Guo](https://www.jiayiguo.net)1, [Yue Ma](https://mayuelala.github.io/)3,
[Nisha Huang](https://scholar.google.com/citations?user=wTmPkSsAAAAJ&hl=en)1, [Yuxin Chen](https://scholar.google.com/citations?hl=en&user=dEm4OKAAAAAJ)2, [Xiu Li](https://scholar.google.com/citations?user=Xrh1OIUAAAAJ&hl=en&oi=ao)1, [Ying Shan](https://scholar.google.com/citations?hl=en&user=4oXBp9UAAAAJ&view_op=list_works&sortby=pubdate)2
1 Tsinghua University, 2 Tencent ARC Lab, 3 HKUST
[](https://arxiv.org/abs/2411.04746)
[](https://huggingface.co/spaces/wjs0725/RF-Solver-Edit)
[](https://github.com/logtd/ComfyUI-Fluxtapoz)
We propose RF-Solver to solve the rectified flow ODE with less error, thus enhancing both sampling quality and inversion-reconstruction accuracy for rectified-flow-based generative models. Furthermore, we propose RF-Edit to leverage the RF-Solver for image and video editing tasks. Our methods achieve impressive performance on various tasks, including text-to-image generation, image/video inversion, and image/video editing.
# 🔥 News
- [2025.5.1] 🎉 Our paper is accepted by ICML 2025.
- [2025.3.24] We have re-organized our code, and releasing the code for video editing!
- [2024.11.30] Our demo is available on 🤗 [Huggingface Space](https://huggingface.co/spaces/wjs0725/RF-Solver-Edit)!
- [2024.11.24] Thanks @[logtd](https://github.com/logtd) for implementing RF-Solver in [LTX-Video](https://github.com/logtd/ComfyUI-LTXTricks)!
- [2024.11.18] More examples for style transfer are available!
- [2024.11.18] Gradio Demo for image editing is available!
- [2024.11.16] Thanks @[logtd](https://github.com/logtd) for integrating RF-Solver into [ComfyUI](https://github.com/logtd/ComfyUI-Fluxtapoz)!
- [2024.11.11] The [homepage](https://rf-solver-edit.github.io/) of the project is available!
- [2024.11.08] Code for image editing is released!
- [2024.11.08] Paper released!
# 👨💻 ToDo
- ☑️ Release the gradio demo
- ☑️ Release scripts for more image editing cases
- ☑️ Release the code for video editing
# 🖼️ Code for Image Editing
For image editing, RF-Edit employs FLUX as the backbone, which comprises several double blocks and single blocks. Double blocks independently modulate text and image features, while single blocks concatenate these features for unified modulation. In this architecture, RF-Edit shares features within the single blocks, as they capture information from both the source image and the source prompt, enhancing the ability of the model to preserve the structural information of the source image.
We have provided the code and demo for image editing using FLUX as the backbone, which can be found Here.
# 🎥 Code for Video Editing
For video editing, in our paper, we employ OpenSora as the backbone. The DiT blocks in OpenSora include spatial attention, temporal attention, and text cross-attention. Within this architecture, the structural information of the source video is captured in the spatial attention module, where we implement feature sharing.
Note that the more powerful video generation model HunyuanVideo is released recently, which is also a RF-based method. The code for video editing in this Repo is implemented based on HunyuanVideo.
We have provided the code and demo for video editing using HunyuanVideo as the backbone, which can be found Here.
# 🎨 Gallery
## Image Stylization
## Image Editing
## Video Editing
## Inversion and Reconstruction
# 📖 Method
## RF-Solver
We derive the exact formulation of the solution for Rectified Flow ODE. The non-linear part in this solution is processed by Taylor Expansion. Through higher order expansion, the approximation error in the solution is significantly reduced, thus achieving impressive performance on both text-to-image sampling and image/video inversion.
## RF-Edit
Based on RF-Solver, we further propose the RF-Edit for image and video editing. RF-Edit framework leverages the features from inversion in the denoising process, which enables high-quality editing while preserving the structural information of source image/video. RF-Edit contains two sub-modules, especially for image editing and video editing.
# 🖋️ Citation
If you find our work helpful, please **star 🌟** this repo and **cite 📑** our paper. Thanks for your support!
```
@article{wang2024taming,
title={Taming Rectified Flow for Inversion and Editing},
author={Wang, Jiangshan and Pu, Junfu and Qi, Zhongang and Guo, Jiayi and Ma, Yue and Huang, Nisha and Chen, Yuxin and Li, Xiu and Shan, Ying},
journal={arXiv preprint arXiv:2411.04746},
year={2024}
}
```
# Acknowledgements
We thank [FLUX](https://github.com/black-forest-labs/flux/tree/main) and [HunyuanVideo](https://github.com/Tencent/HunyuanVideo) for their clean codebase.
# Contact
The code in this repository is still being reorganized. Errors that may arise during the organizing process could lead to code malfunctions or discrepancies from the original research results. If you have any questions or concerns, please send emails to wjs23@mails.tsinghua.edu.cn.