{"id":13738065,"url":"https://github.com/yuval-alaluf/restyle-encoder","last_synced_at":"2025-05-16T15:08:45.785Z","repository":{"id":37806817,"uuid":"355244720","full_name":"yuval-alaluf/restyle-encoder","owner":"yuval-alaluf","description":"Official Implementation for \"ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement\" (ICCV 2021) https://arxiv.org/abs/2104.02699","archived":false,"fork":false,"pushed_at":"2022-10-01T11:24:16.000Z","size":28799,"stargazers_count":1041,"open_issues_count":1,"forks_count":154,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-26T00:33:15.625Z","etag":null,"topics":["generative-adversarial-networks","iccv2021","iterative-refinement","stylegan","stylegan-encoder"],"latest_commit_sha":null,"homepage":"https://yuval-alaluf.github.io/restyle-encoder/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuval-alaluf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-06T15:45:57.000Z","updated_at":"2025-04-24T07:33:10.000Z","dependencies_parsed_at":"2023-01-19T01:15:56.283Z","dependency_job_id":null,"html_url":"https://github.com/yuval-alaluf/restyle-encoder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Frestyle-encoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Frestyle-encoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Frestyle-encoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Frestyle-encoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuval-alaluf","download_url":"https://codeload.github.com/yuval-alaluf/restyle-encoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254553958,"owners_count":22090417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generative-adversarial-networks","iccv2021","iterative-refinement","stylegan","stylegan-encoder"],"created_at":"2024-08-03T03:02:10.270Z","updated_at":"2025-05-16T15:08:40.775Z","avatar_url":"https://github.com/yuval-alaluf.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (ICCV 2021)\n\n\u003e Recently, the power of unconditional image synthesis has significantly advanced through the use of Generative Adversarial Networks (GANs). The task of inverting an image into its corresponding latent code of the trained GAN is of utmost importance as it allows for the manipulation of real images, leveraging the rich semantics learned by the network. Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders.\n\n\u003ca href=\"https://arxiv.org/abs/2104.02699\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2104.02699-b31b1b.svg\" height=22.5\u003e\u003c/a\u003e\n\u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" height=22.5\u003e\u003c/a\u003e  \n\n\u003ca href=\"https://www.youtube.com/watch?v=9RzCZZBjlxM\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Two Minute Papers\u0026message=ReStyle Video\u0026color=red\" height=22.5\u003e\u003c/a\u003e  \n\u003ca href=\"https://youtu.be/6pGzLECSIWM\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=ICCV 2021 \u0026message=5 Minute Video\u0026color=red\" height=22.5\u003e\u003c/a\u003e  \n\u003ca href=\"https://replicate.ai/yuval-alaluf/restyle_encoder\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Replicate\u0026message=Demo and Docker Image\u0026color=darkgreen\" height=22.5\u003e\u003c/a\u003e\n\nInference Notebook: \u003ca href=\"http://colab.research.google.com/github/yuval-alaluf/restyle-encoder/blob/master/notebooks/inference_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=20\u003e\u003c/a\u003e  \nAnimation Notebook: \u003ca href=\"http://colab.research.google.com/github/yuval-alaluf/restyle-encoder/blob/master/notebooks/animations_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=20\u003e\u003c/a\u003e  \n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/teaser.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nDifferent from conventional encoder-based inversion techniques, our residual-based ReStyle scheme incorporates an iterative refinement mechanism to progressively converge to an accurate inversion of real images. For each domain, we show the input image on the left followed by intermediate inversion outputs.\n\u003c/p\u003e\n\n## Description   \nOfficial Implementation of our ReStyle paper for both training and evaluation. ReStyle introduces an iterative\nrefinement mechanism which can be applied over different StyleGAN encoders for solving the StyleGAN inversion task.\n\n## Table of Contents\n  * [Description](#description)\n  * [Getting Started](#getting-started)\n    + [Prerequisites](#prerequisites)\n    + [Installation](#installation)\n  * [Pretrained Models](#pretrained-models)\n    + [ReStyle-pSp](#restyle-psp)\n    + [ReStyle-e4e](#restyle-e4e)\n    + [Auxiliary Models](#auxiliary-models)\n  * [Training](#training)\n    + [Preparing your Data](#preparing-your-data)\n    + [Preparing your Generator](#preparing-your-generator)\n    + [Training ReStyle](#training-restyle)\n      - [Additional Notes](#additional-notes)\n  * [Inference Notebooks](#inference-notebooks)\n  * [Testing](#testing)\n    + [Inference](#inference)\n    + [Step-by-Step Inference](#step-by-step-inference)\n    + [Computing Metrics](#computing-metrics)\n  * [Editing](#editing)\n  * [Encoder Bootstrapping](#encoder-bootstrapping)\n  * [Repository structure](#repository-structure)\n  * [Credits](#credits)\n  * [Acknowledgments](#acknowledgments)\n  * [Citation](#citation)\n\n## Getting Started\n### Prerequisites\n- Linux or macOS\n- NVIDIA GPU + CUDA CuDNN (CPU may be possible with some modifications, but is not inherently supported)\n- Python 3\n\n### Installation\n- Dependencies:  \nWe recommend running this repository using [Anaconda](https://docs.anaconda.com/anaconda/install/). \nAll dependencies for defining the environment are provided in `environment/restyle_env.yaml`.\n\n## Pretrained Models\nIn this repository, we provide pretrained ReStyle encoders applied over the \n[pSp](https://github.com/eladrich/pixel2style2pixel) and [e4e](https://github.com/omertov/encoder4editing) encoders \nacross various domains.\n\nPlease download the pretrained models from the following links.\n\n### ReStyle-pSp\n| Path | Description\n| :--- | :----------\n|[FFHQ - ReStyle + pSp](https://drive.google.com/file/d/1sw6I2lRIB0MpuJkpc8F5BJiSZrc0hjfE/view?usp=sharing)  | ReStyle applied over pSp trained on the [FFHQ](https://github.com/NVlabs/ffhq-dataset) dataset.\n|[Stanford Cars - ReStyle + pSp](https://drive.google.com/file/d/1zJHqHRQ8NOnVohVVCGbeYMMr6PDhRpPR/view?usp=sharing)  | ReStyle applied over pSp trained on the [Stanford Cars](https://ai.stanford.edu/~jkrause/cars/car_dataset.html) dataset.\n|[LSUN Church - ReStyle + pSp](https://drive.google.com/file/d/1bcxx7mw-1z7dzbJI_z7oGpWG1oQAvMaD/view?usp=sharing)  | ReStyle applied over pSp trained on the [LSUN](https://www.yf.io/p/lsun) Church dataset.\n|[AFHQ Wild - ReStyle + pSp](https://drive.google.com/file/d/1GyFXVTNDUw3IIGHmGS71ChhJ1Rmslhk7/view?usp=sharing)  | ReStyle applied over pSp trained on the [AFHQ Wild](https://github.com/clovaai/stargan-v2) dataset.\n\n### ReStyle-e4e\n| Path | Description\n| :--- | :----------\n|[FFHQ - ReStyle + e4e](https://drive.google.com/file/d/1e2oXVeBPXMQoUoC_4TNwAWpOPpSEhE_e/view?usp=sharing)  | ReStyle applied over e4e trained on the [FFHQ](https://github.com/NVlabs/ffhq-dataset) dataset.\n|[Stanford Cars - ReStyle + e4e](https://drive.google.com/file/d/1v6Yi-Yne6JX0Pyas6XPS83IzKKUvCe8r/view?usp=sharing)  | ReStyle applied over e4e trained on the [Stanford Cars](https://ai.stanford.edu/~jkrause/cars/car_dataset.html) dataset.\n|[LSUN Church - ReStyle + e4e](https://drive.google.com/file/d/1sCRzbM1fGTgTbWO87jP_rxApw-tS0CH_/view?usp=sharing)  | ReStyle applied over e4e trained on the [LSUN](https://www.yf.io/p/lsun) Church dataset.\n|[AFHQ Wild - ReStyle + e4e](https://drive.google.com/file/d/1HQlUPKirShHyL3kGnwVRjhfcb-j9s4V3/view?usp=sharing)  | ReStyle applied over e4e trained on the [AFHQ Wild](https://github.com/clovaai/stargan-v2) dataset.\n|[LSUN Horse - ReStyle + e4e](https://drive.google.com/file/d/19_sUpTYtJmhSAolKLm3VgI-ptYqd-hgY/view?usp=sharing)  | ReStyle applied over e4e trained on the [LSUN](https://www.yf.io/p/lsun) Horse dataset.\n\n### Auxiliary Models\nIn addition, we provide various auxiliary models needed for training your own ReStyle models from scratch.  \nThis includes the StyleGAN generators and pre-trained models used for loss computation.\n\n| Path | Description\n| :--- | :----------\n|[FFHQ StyleGAN](https://drive.google.com/file/d/1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT/view?usp=sharing) | StyleGAN2 model trained on FFHQ with 1024x1024 output resolution.\n|[LSUN Car StyleGAN](https://drive.google.com/file/d/1UmMHHB3DU1trTB8_9Fjkck5ZwArnD81B/view?usp=sharing) | StyleGAN2 model trained on LSUN Car with 512x384 output resolution.\n|[LSUN Church StyleGAN](https://drive.google.com/file/d/1ZSZOFtw5wZfPhWEzHFB9LI__bikaHsn6/view?usp=sharing) | StyleGAN2 model trained on LSUN Church with 256x256 output resolution.\n|[LSUN Horse StyleGAN](https://drive.google.com/file/d/1bM7ms1tWCucnySjiS6-NNrPJGDkqQQWi/view?usp=sharing) | StyleGAN2 model trained on LSUN Horse with 256x256 output resolution.\n|[AFHQ Wild StyleGAN](https://drive.google.com/file/d/1z6IVVaCJuFTksKwp1CM3emWOVHbrBip-/view?usp=sharing) | StyleGAN-ADA model trained on AFHQ Wild with 512x512 output resolution.\n|[IR-SE50 Model](https://drive.google.com/file/d/1KW7bjndL3QG3sxBbZxreGHigcCCpsDgn/view?usp=sharing) | Pretrained IR-SE50 model taken from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch) for use in our ID loss and encoder backbone on human facial domain.\n|[ResNet-34 Model](https://download.pytorch.org/models/resnet34-333f7ec4.pth) | ResNet-34 model trained on ImageNet taken from [torchvision](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) for initializing our encoder backbone.\n|[MoCov2 Model](https://drive.google.com/file/d/18rLcNGdteX5LwT7sv_F7HWr12HpVEzVe/view) | Pretrained ResNet-50 model trained using MOCOv2 for computing MoCo-based loss on non-facial domains. The model is taken from the [official implementation](https://github.com/facebookresearch/moco).\n|[CurricularFace Backbone](https://drive.google.com/file/d/1f4IwVa2-Bn9vWLwB-bUwm53U_MlvinAj/view?usp=sharing) | Pretrained CurricularFace model taken from [HuangYG123](https://github.com/HuangYG123/CurricularFace) for use in ID similarity metric computation.\n|[MTCNN](https://drive.google.com/file/d/1tJ7ih-wbCO6zc3JhI_1ZGjmwXKKaPlja/view?usp=sharing) | Weights for MTCNN model taken from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch) for use in ID similarity metric computation. (Unpack the tar.gz to extract the 3 model weights.)\n\nNote: all StyleGAN models are converted from the official TensorFlow models to PyTorch using the conversion script from [rosinality](https://github.com/rosinality/stylegan2-pytorch).\n\nBy default, we assume that all auxiliary models are downloaded and saved to the directory `pretrained_models`. \nHowever, you may use your own paths by changing the necessary values in `configs/path_configs.py`. \n\n## Training\n### Preparing your Data\nIn order to train ReStyle on your own data, you should perform the following steps: \n1. Update `configs/paths_config.py` with the necessary data paths and model paths for training and inference.\n```\ndataset_paths = {\n    'train_data': '/path/to/train/data'\n    'test_data': '/path/to/test/data',\n}\n```\n2. Configure a new dataset under the `DATASETS` variable defined in `configs/data_configs.py`. There, you should define \nthe source/target data paths for the train and test sets as well as the transforms to be used for training and inference.\n```\nDATASETS = {\n\t'my_data_encode': {\n\t\t'transforms': transforms_config.EncodeTransforms,   # can define a custom transform, if desired\n\t\t'train_source_root': dataset_paths['train_data'],\n\t\t'train_target_root': dataset_paths['train_data'],\n\t\t'test_source_root': dataset_paths['test_data'],\n\t\t'test_target_root': dataset_paths['test_data'],\n\t}\n}\n```\n3. To train with your newly defined dataset, simply use the flag `--dataset_type my_data_encode`.\n\n### Preparing your Generator\nIn this work, we use rosinality's [StyleGAN2 implementation](https://github.com/rosinality/stylegan2-pytorch). \nIf you wish to use your own generator trained using NVIDIA's implementation there are a few options we recommend:\n1. Using NVIDIA's StyleGAN2 / StyleGAN-ADA TensorFlow implementation.  \n   You can then convert the TensorFlow `.pkl` checkpoints to the supported format using the conversion script found in [rosinality's implementation](https://github.com/rosinality/stylegan2-pytorch#convert-weight-from-official-checkpoints).\n2. Using NVIDIA's StyleGAN-ADA PyTorch implementation.  \n   You can then convert the PyTorch `.pkl` checkpoints to the supported format using the conversion script created by [Justin Pinkney](https://github.com/justinpinkney) found in [dvschultz's fork](https://github.com/dvschultz/stylegan2-ada-pytorch/blob/main/SG2_ADA_PT_to_Rosinality.ipynb).  \n\n\nOnce you have the converted `.pt` files, you should be ready to use them in this repository.  \n\n\n### Training ReStyle\nThe main training scripts can be found in `scripts/train_restyle_psp.py` and `scripts/train_restyle_e4e.py`. Each of the \ntwo scripts will run ReStyle applied over the corresponding base inversion method.  \nIntermediate training results are saved to `opts.exp_dir`. This includes checkpoints, train outputs, and test outputs.  \nAdditionally, if you have tensorboard installed, you can visualize tensorboard logs in `opts.exp_dir/logs`.\n\nWe currently support applying ReStyle on the pSp encoder from [Richardson et al. [2020]](https://arxiv.org/abs/2008.00951) \nand the e4e encoder from [Tov et al. [2021]](https://arxiv.org/abs/2102.02766). \n\nTraining ReStyle with the settings used in the paper can be done by running the following commands.\n\n- ReStyle applied over pSp:\n```\npython scripts/train_restyle_psp.py \\\n--dataset_type=ffhq_encode \\\n--encoder_type=BackboneEncoder \\\n--exp_dir=experiment/restyle_psp_ffhq_encode \\\n--workers=8 \\\n--batch_size=8 \\\n--test_batch_size=8 \\\n--test_workers=8 \\\n--val_interval=5000 \\\n--save_interval=10000 \\\n--start_from_latent_avg \\\n--lpips_lambda=0.8 \\\n--l2_lambda=1 \\\n--w_norm_lambda=0 \\\n--id_lambda=0.1 \\\n--input_nc=6 \\\n--n_iters_per_batch=5 \\\n--output_size=1024 \\\n--stylegan_weights=pretrained_models/stylegan2-ffhq-config-f.pt\n```\n  \n- ReStyle applied over e4e:\n```\npython scripts/train_restyle_e4e.py \\\n--dataset_type ffhq_encode \\\n--encoder_type ProgressiveBackboneEncoder \\\n--exp_dir=experiment/restyle_e4e_ffhq_encode \\\n--workers=8 \\\n--batch_size=8 \\\n--test_batch_size=8 \\\n--test_workers=8 \\\n--start_from_latent_avg \\\n--lpips_lambda=0.8 \\\n--l2_lambda=1 \\\n--delta_norm_lambda 0.0002 \\\n--id_lambda 0.1 \\\n--use_w_pool \\\n--w_discriminator_lambda 0.1 \\\n--progressive_start 20000 \\\n--progressive_step_every 2000 \\\n--input_nc 6 \\\n--n_iters_per_batch=5 \\\n--output_size 1024 \\\n--stylegan_weights=pretrained_models/stylegan2-ffhq-config-f.pt\n```\n\n#### Additional Notes\n- Encoder backbones:\n    - For the human facial domain (`ffhq_encode`), we use an IRSE-50 backbone using the flags:\n        - `--encoder_type=BackboneEncoder` for pSp\n        - `--encoder_type=ProgressiveBackboneEncoder` for e4e\n    - For all other domains, we use a ResNet34 encoder backbone using the flags:\n        - `--encoder_type=ResNetBackboneEncoder` for pSp\n        - `--encoder_type=ResNetProgressiveBackboneEncoder` for e4e\n- ID/similarity losses: \n    - For the human facial domain we also use a specialized ID loss which is set using the flag `--id_lambda=0.1`.\n    - For all other domains, please set `--id_lambda=0` and `--moco_lambda=0.5` to use the MoCo-based similarity loss from Tov et al. \n        - Note, you __cannot__ set both `id_lambda` and `moco_lambda` to be active simultaneously.\n- You should also adjust the `--output_size` and `--stylegan_weights` flags according to your StyleGAN generator. \n- See `options/train_options.py` and `options/e4e_train_options.py` for all training-specific flags. \n\n## Inference Notebooks\nTo help visualize the results of ReStyle we provide a Jupyter notebook found in `notebooks/inference_playground.ipynb`.   \nThe notebook will download the pretrained models and run inference on the images found in `notebooks/images` or \non images of your choosing. It is recommended to run this in [Google Colab](http://colab.research.google.com/github/yuval-alaluf/restyle-encoder/blob/master/notebooks/inference_playground.ipynb).\n\nWe have also provided a notebook for generating interpolation videos such as those found in the project page. This \nnotebook can be run using Google Colab [here](http://colab.research.google.com/github/yuval-alaluf/restyle-encoder/blob/master/notebooks/animations_playground.ipynb).\n\n## Testing\n### Inference\nYou can use `scripts/inference_iterative.py` to apply a trained model on a set of images:\n```\npython scripts/inference_iterative.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=5\n```\nThis script will save each step's outputs in a separate sub-directory (e.g., the outputs of step `i` will\nbe saved in `/path/to/experiment/inference_results/i`). \n\nNotes: \n- By default, the images will be saved at their original output resolutions (e.g., `1024x1024` for faces, `512x384` for cars).\nIf you wish to save outputs resized to resolutions of `256x256` (or `256x192` for cars), you can do so by adding the flag `--resize_outputs`.\n- This script will also save all the latents as an `.npy` file in a dictionary format as follows: \n```\n{\n    \"0.jpg\": [latent_step_1, latent_step_2, ..., latent_step_N],\n    \"1.jpg\": [latent_step_1, latent_step_2, ..., latent_step_N],\n    ...\n}\n```\nThat is, the keys of the dictionary are the image file names and the values are lists of length `N` containing the output latent of each \nstep where `N` is the number of inference steps. Each element in the list is of shape `(Kx512)` where `K` is the number\nof style inputs of the generator. \n\nYou can use the saved latents to perform latent space manipulations, for example.\n\n\n### Step-by-Step Inference \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/2441.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/02530.jpg\" width=\"800px\"/\u003e\n\u003cbr\u003e\nVisualizing the intermediate outputs. Here, the intermediate outputs are saved from left to right with the input image shown on the right-hand side.\n\u003c/p\u003e\n\nSometimes, you may wish to save each step's outputs side-by-side instead of in separate sub-folders. This would allow one \nto easily see the progression in the reconstruction with each step. To save the step-by-step outputs as a single image, \nyou can run the following: \n```\npython scripts/inference_iterative_save_coupled.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=5\n```\n\n### Computing Metrics\nGiven a trained model and generated outputs, we can compute the loss metrics on a given dataset.  \nThese scripts receive the inference output directory and ground truth directory.\n- Calculating LPIPS loss:\n```\npython scripts/calc_losses_on_images.py \\\n--mode lpips \\\n--output_path=/path/to/experiment/inference_results \\\n--gt_path=/path/to/test_images\n```\n- Calculating L2 loss:\n```\npython scripts/calc_losses_on_images.py \\\n--mode l2 \\\n--output_path=/path/to/experiment/inference_results \\\n--gt_path=/path/to/test_images\n```\n- Calculating the identity loss for the human facial domain: \n```\npython scripts/calc_id_loss_parallel.py \\\n--output_path=/path/to/experiment/inference_results \\\n--gt_path=/path/to/test_images\n```\n\nThese scripts will traverse through each sub-directory of `output_path` to compute the metrics on each step's output images.\n\n## Editing\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/ardern.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/macron.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/merkel.jpg\" width=\"800px\"/\u003e\n\u003cbr\u003e\nEditing results using InterFaceGAN on inversions obtained using ReStyle-e4e.\n\u003c/p\u003e\nFor performing edits using ReStyle-e4e, you can run the script found in `editing/inference_editing.py`, as follows:  \n\n```\npython editing/inference_editing.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=/path/to/e4e_ffhq_encoder.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=5 \\\n--edit_directions=age,pose,smile \\\n--factor_ranges=5,5,5\n```\n\nThis script will perform the inversion immediately followed by the latent space edit.  \nThe results for each edit will be saved to different sub-directories in the specified experiment directory. For each image, \nwe save the original image followed by the inversion and the resulting edits.  \nWe support running inference using ReStyle-e4e models on the faces domain using edit several directions \nobtained via InterFaceGAN (`age`, `pose`, and `smile`). \n\n\n## Encoder Bootstrapping\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/346.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/2598.jpg\" width=\"800px\"/\u003e\n\u003cbr\u003e\nImage toonification results using our proposed encoder bootstrapping technique.\n\u003c/p\u003e\n\nIn the paper, we introduce an encoder bootstrapping technique that can be used to solve the image toonification task by \npairing an FFHQ-based encoder with a Toon-based encoder.  \nBelow we provide the models used to generate the results in the paper: \n| Path | Description\n| :--- | :----------\n|[FFHQ - ReStyle + pSp](https://drive.google.com/file/d/1sw6I2lRIB0MpuJkpc8F5BJiSZrc0hjfE/view?usp=sharing)  | Same FFHQ encoder as linked above.\n|[Toonify - ReStyle + pSp](https://drive.google.com/file/d/1GtudVDig59d4HJ_8bGEniz5huaTSGO_0/view?usp=sharing)  | ReStyle applied over pSp trained for the image toonification task.\n|[Toonify Generator](https://drive.google.com/file/d/1r3XVCt_WYUKFZFxhNH-xO2dTtF6B5szu/view?usp=sharing)  | Toonify generator from [Doron Adler](https://github.com/Norod) and [Justin Pinkney](https://github.com/justinpinkney) converted to Pytorch using rosinality's conversion script.\n\nNote that the ReStyle toonify model is trained using only real images with __no__ paired data. \nMore details regarding the training parameters and settings of the toonify encoder can be found [here](https://github.com/eladrich/pixel2style2pixel#additional-applications).\n\nIf you wish to run inference using these two models and the bootstrapping technique you may run the following: \n```\npython scripts/encoder_bootstrapping_inference.py \\\n--exp_dir=/path/to/experiment \\\n--model_1_checkpoint_path=/path/to/restyle_psp_ffhq_encode.pt \\\n--model_2_checkpoint_path=/path/to/restyle_psp_toonify.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=1  # one step for each encoder is typically good\n```\nHere, we output the per-step outputs side-by-side with the inverted initialization real-image on the left and the original \ninput image on the right.  \n\n\n\n## Repository structure\n| Path | Description \u003cimg width=200\u003e\n| :--- | :---\n| restyle-encoder | Repository root folder\n| \u0026boxvr;\u0026nbsp; configs | Folder containing configs defining model/data paths and data transforms\n| \u0026boxvr;\u0026nbsp; criteria | Folder containing various loss criterias for training\n| \u0026boxvr;\u0026nbsp; datasets | Folder with various dataset objects\n| \u0026boxvr;\u0026nbsp; docs | Folder containing images displayed in the README\n| \u0026boxvr;\u0026nbsp; environment | Folder containing Anaconda environment used in our experiments\n| \u0026boxvr;\u0026nbsp; licenses | Folder containing licenses of the open source projects used in this repository\n| \u0026boxvr; models | Folder containing all the models and training objects\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; e4e_modules | Folder containing the latent discriminator implementation from [encoder4editing](https://github.com/omertov/encoder4editing)\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; encoders | Folder containing various architecture implementations including our simplified encoder architectures\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; mtcnn | MTCNN implementation from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch)\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; stylegan2 | StyleGAN2 model from [rosinality](https://github.com/rosinality/stylegan2-pytorch)\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; psp.py | Implementation of pSp encoder extended to work with ReStyle\n| \u0026boxv;\u0026nbsp; \u0026boxur;\u0026nbsp; e4e.py | Implementation of e4e encoder extended to work with ReStyle\n| \u0026boxvr;\u0026nbsp; notebooks | Folder with jupyter notebook containing ReStyle inference playground\n| \u0026boxvr;\u0026nbsp; options | Folder with training and test command-line options\n| \u0026boxvr;\u0026nbsp; scripts | Folder with running scripts for training, inference, and metric computations\n| \u0026boxvr;\u0026nbsp; training | Folder with main training logic and Ranger implementation from [lessw2020](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)\n| \u0026boxvr;\u0026nbsp; utils | Folder with various utility functions\n| \u003cimg width=300\u003e | \u003cimg\u003e\n\n\n## Credits\n**StyleGAN2 model and implementation:**  \nhttps://github.com/rosinality/stylegan2-pytorch  \nCopyright (c) 2019 Kim Seonghyeon  \nLicense (MIT) https://github.com/rosinality/stylegan2-pytorch/blob/master/LICENSE  \n\n**IR-SE50 model and implementations:**  \nhttps://github.com/TreB1eN/InsightFace_Pytorch  \nCopyright (c) 2018 TreB1eN  \nLicense (MIT) https://github.com/TreB1eN/InsightFace_Pytorch/blob/master/LICENSE  \n\n**Ranger optimizer implementation:**  \nhttps://github.com/lessw2020/Ranger-Deep-Learning-Optimizer   \nLicense (Apache License 2.0) https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/master/LICENSE  \n\n**LPIPS model and implementation:**  \nhttps://github.com/S-aiueo32/lpips-pytorch  \nCopyright (c) 2020, Sou Uchida  \nLicense (BSD 2-Clause) https://github.com/S-aiueo32/lpips-pytorch/blob/master/LICENSE  \n\n**pSp model and implementation:**   \nhttps://github.com/eladrich/pixel2style2pixel  \nCopyright (c) 2020 Elad Richardson, Yuval Alaluf  \nLicense (MIT) https://github.com/eladrich/pixel2style2pixel/blob/master/LICENSE\n\n**e4e model and implementation:**   \nhttps://github.com/omertov/encoder4editing\nCopyright (c) 2021 omertov  \nLicense (MIT) https://github.com/omertov/encoder4editing/blob/main/LICENSE\n\n**Please Note**: The CUDA files under the [StyleGAN2 ops directory](https://github.com/eladrich/pixel2style2pixel/tree/master/models/stylegan2/op) are made available under the [Nvidia Source Code License-NC](https://nvlabs.github.io/stylegan2/license.html)\n\n## Acknowledgments\nThis code borrows heavily from [pixel2style2pixel](https://github.com/eladrich/pixel2style2pixel) and \n[encoder4editing](https://github.com/omertov/encoder4editing).\n\n## Citation\nIf you use this code for your research, please cite the following works:\n```\n@InProceedings{alaluf2021restyle,\n      author = {Alaluf, Yuval and Patashnik, Or and Cohen-Or, Daniel},\n      title = {ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement}, \n      month = {October},\n      booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},  \n      year = {2021}\n}\n```\n```\n@InProceedings{richardson2021encoding,\n      author = {Richardson, Elad and Alaluf, Yuval and Patashnik, Or and Nitzan, Yotam and Azar, Yaniv and Shapiro, Stav and Cohen-Or, Daniel},\n      title = {Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation},\n      booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n      month = {June},\n      year = {2021}\n}\n@article{tov2021designing,\n      title={Designing an Encoder for StyleGAN Image Manipulation},\n      author={Tov, Omer and Alaluf, Yuval and Nitzan, Yotam and Patashnik, Or and Cohen-Or, Daniel},\n      journal={arXiv preprint arXiv:2102.02766},\n      year={2021}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuval-alaluf%2Frestyle-encoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuval-alaluf%2Frestyle-encoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuval-alaluf%2Frestyle-encoder/lists"}