https://github.com/hansam95/nmg

https://github.com/hansam95/nmg

Last synced: 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/hansam95/nmg
• Owner: hansam95
• Created: 2024-02-01T10:50:51.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-10-21T04:04:23.000Z (3 months ago)
• Last Synced: 2024-10-21T07:07:09.702Z (3 months ago)
• Language: Jupyter Notebook
• Size: 8.02 MB
• Stars: 19
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-diffusion-categorized - [Code

README

        
# [ICLR 2024] Nois Map Guidance: Inversion with Spatial Context for Real Image Editing

[![arXiv](https://img.shields.io/badge/arXiv-2402.04625-red)](https://arxiv.org/abs/2402.04625)

> **Nois Map Guidance: Inversion with Spatial Context for Real Image Editing**


> Hansam Cho, Jonghyun Lee, Seoung Bum Kim, Tae-Hyun Oh, Yonghyun Jeong


> 

>**Abstract**: 


Text-guided diffusion models have become a popular tool in image synthesis, known for producing high-quality and diverse images. However, their application to editing real images often encounters hurdles primarily due to the text condition deteriorating the reconstruction quality and subsequently affecting editing fidelity. Null-text Inversion (NTI) has made strides in this area, but it fails to capture spatial context and requires computationally intensive per-timestep optimization. Addressing these challenges, we present NOISE MAP GUIDANCE (NMG), an inversion method rich in a spatial context, tailored for real-image editing. Significantly, NMG achieves this without necessitating optimization, yet preserves the editing quality. Our empirical investigations highlight NMG’s adaptability across various editing techniques and its robustness to variants of DDIM inversions.

## Description

Official implementation of Noise Map Guidance: Inversion with Spatial Context for Real Image Editing 

![image](images/teaser.jpg)

## Setup

```

conda env create -f environment.yaml

conda activate nmg

```

## NMG + Editing Methods

### Prompt-to-Prompt

The [nmg_ptp.ipynb](nmg_ptp.ipynb) is Notebook for NMG with [Prompt-to-Prompt](https://arxiv.org/abs/2208.01626) editing, capable of performing tasks such as  **object swap**, **contextual alterations**, **face attribute editing**, **color change**, and **global editing**. To efficiently process these tasks, it's recommended to use a GPU equipped with a minimum of 15GB of VRAM.

### MasaCtrl

The [nmg_masactrl.ipynb](nmg_masactrl.ipynb) is Notebook for NMG with [MasaCtrl](https://arxiv.org/abs/2304.08465) editing, capable of performing tasks such as  **viewpoint alternation**, and **pose modification**. To efficiently process these tasks, it's recommended to use a GPU equipped with a minimum of 23GB of VRAM.

### pix2pix-zero

The [nmg_pix2pix.ipynb](nmg_pix2pix.ipynb) is Notebook for NMG with [pix2pix-zero](https://arxiv.org/abs/2302.03027) editing, capable of performing tasks such as  **dog → cat**, and **cat → dog**. To efficiently process these tasks, it's recommended to use a GPU equipped with a minimum of 11GB of VRAM.

## Acknowledgements

This repository is built upon [diffusers](https://huggingface.co/docs/diffusers/index), unofficial implementation of [prompt-to-prompt](https://github.com/Weifeng-Chen/prompt2prompt/tree/main), [pix2pix-zero pipeline](https://github.com/huggingface/diffusers/blob/v0.16.0/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_pix2pix_zero.py),and [MasaCtrl](https://github.com/TencentARC/MasaCtrl).

## Citation

```bibtex

@inproceedings{cho2023noise,

  title={Noise Map Guidance: Inversion with Spatial Context for Real Image Editing},

  author={Cho, Hansam and Lee, Jonghyun and Kim, Seoung Bum and Oh, Tae-Hyun and Jeong, Yonghyun},

  booktitle={The Twelfth International Conference on Learning Representations},

  year={2023}

}

```