Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://ezioby.github.io/padinv/
[ECCV 2022] PadInv: High-fidelity GAN Inversion with Padding Space
https://ezioby.github.io/padinv/
gan-inversion image-editing image-synthesis
Last synced: 3 months ago
JSON representation
[ECCV 2022] PadInv: High-fidelity GAN Inversion with Padding Space
- Host: GitHub
- URL: https://ezioby.github.io/padinv/
- Owner: EzioBy
- Created: 2022-03-17T12:24:01.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-12-17T00:45:49.000Z (over 1 year ago)
- Last Synced: 2024-01-17T06:48:15.084Z (5 months ago)
- Topics: gan-inversion, image-editing, image-synthesis
- Language: Python
- Homepage: https://ezioby.github.io/padinv/
- Size: 11.1 MB
- Stars: 86
- Watchers: 11
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Lists
- GAN-Inversion - Qingyan Bai
README
# PadInv - High-fidelity GAN Inversion with Padding Space
> **High-fidelity GAN Inversion with Padding Space**
> Qingyan Bai*, Yinghao Xu*, Jiapeng Zhu, Weihao Xia, Yujiu Yang, Yujun Shen
> *European Conference on Computer Vision (ECCV) 2022*
**Figure:** Our encoder produces instance-aware coefficients to replace the fixed padding used in the generator. Such a design improves GAN inversion with better spatial details.
[[Paper](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136750036.pdf)]
[[Project Page](https://ezioby.github.io/padinv/)]
[[ArXiv Paper with Supp](https://arxiv.org/abs/2203.11105)]
[[ECVA Link](https://www.ecva.net/papers/eccv_2022/papers_ECCV/html/877_ECCV_2022_paper.php)]
In this work, we propose to involve the **padding space** of the generator to complement the native latent space, facilitating high-fidelity GAN inversion. Concretely, we replace the constant padding (*e.g.*, usually zeros) used in convolution layers with some instance-aware coefficients. In this way, the inductive bias assumed in the pre-trained model can be appropriately adapted to fit each individual image. We demonstrate that such a space extension allows a more flexible image manipulation, such as the **separate control** of face contour and facial details, and enables a **novel editing manner** where users can *customize* their own manipulations highly efficiently.
## Qualitative Results
From top to bottom: (a) high-fidelity GAN inversion with spatial details, (b) face blending with contour from one image and details from another, and (c) customized manipulations *with one image pair*.
Additional inversion results.
Additional face blending results.
Additional customized editing results.
## Preparation
To train or test PadInv, preparing the data and pre-trained GAN checkpoints is needed at first.
For data, please download FFHQ and CelebA-HQ-testset for face domain,
and LSUN Church and Bedroom for indoor and outdoor scene, respectively.
For pre-trained GAN checkpoints, you can download them here:
[StyleGAN2-FFHQ](https://drive.google.com/file/d/1cF7oKKVOgMuXpXifSeD8PyaVYTBaOBsf/view?usp=sharing),
[StyleGAN2-Church](https://drive.google.com/file/d/1Ww9YXlAVuRt0Ign0ulYtsttmX7Mh35wH/view?usp=sharing),
[StyleGAN2-Bedroom](https://drive.google.com/file/d/1LTEU2Dt2pyJTHHUTjwFWAkmNco3EXmfL/view?usp=sharing).
## Training
#### Training Scripts
Please use the following scripts to train PadInv corresponding to various domains.
```
# Face
bash scripts/encoder_scipts/encoder_stylegan2_ffhq_train.sh 8 your_training_set_path your_test_set_path your_gan_ckp_path --job_name=your_job_name
# Church
bash scripts/encoder_scipts/encoder_stylegan2_church_train.sh 8 your_training_set_path your_test_set_path your_gan_ckp_path --job_name=your_job_name
# Bedroom
bash scripts/encoder_scipts/encoder_stylegan2_bedroom_train.sh 8 your_training_set_path your_test_set_path your_gan_ckp_path --job_name=your_job_name
```
In scripts above, '8' indicates the gpu amount for training.
'your_training_set_path' and 'your_test_set_path' indicate the dataset paths
(e.g. ```data/ffhq.zip, data/CelebA-HQ-test```, or ```data/bedroom_train_lmdb```).
For training and testing on LSUN, we support reading the LMDB directory thanks to
[Hammer](https://github.com/bytedance/Hammer).
'your_gan_ckp_path' indicates the path of the pre-trained GAN checkpoint to be inverted.
'your_job_name' indicates the name of this training job and the name of the job working directory.
#### Results
1. Testing metric results and visualization results of inversion can be found in ```work_dir/your_job_name/results/```.
2. The training log is saved at ```work_dir/your_job_name/log.txt```.
3. Checkpoints can be found in ```work_dir/your_job_name/checkpoints/```.
Note that we save the checkpoints corresponding to the best metrics and the latest ones.
## BibTeX
If you find our work or code helpful for your research, please consider to cite:
```bibtex
@inproceedings{bai2022high,
title={High-fidelity GAN inversion with padding space},
author={Bai, Qingyan and Xu, Yinghao and Zhu, Jiapeng and Xia, Weihao and Yang, Yujiu and Shen, Yujun},
booktitle={European Conference on Computer Vision},
pages={36--53},
year={2022},
organization={Springer}
}
```
## Acknowledgement
Thanks to
[Hammer](https://github.com/bytedance/Hammer),
[StyleGAN2](https://github.com/NVlabs/stylegan2), and
[Pixel2Style2Pixel](https://github.com/eladrich/pixel2style2pixel)
for sharing the code.