{"id":13730497,"url":"https://github.com/yuval-alaluf/hyperstyle","last_synced_at":"2025-04-12T21:25:55.769Z","repository":{"id":37450541,"uuid":"433516569","full_name":"yuval-alaluf/hyperstyle","owner":"yuval-alaluf","description":"Official Implementation for \"HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing\" (CVPR 2022) https://arxiv.org/abs/2111.15666","archived":false,"fork":false,"pushed_at":"2022-09-17T16:06:21.000Z","size":53956,"stargazers_count":1019,"open_issues_count":5,"forks_count":115,"subscribers_count":28,"default_branch":"main","last_synced_at":"2025-04-04T00:52:44.979Z","etag":null,"topics":["cvpr2022","generative-adversarial-network","hypernetworks","stylegan","stylegan-encoder"],"latest_commit_sha":null,"homepage":"https://yuval-alaluf.github.io/hyperstyle/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuval-alaluf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-30T17:07:00.000Z","updated_at":"2025-04-04T00:04:40.000Z","dependencies_parsed_at":"2022-07-14T08:49:32.429Z","dependency_job_id":null,"html_url":"https://github.com/yuval-alaluf/hyperstyle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Fhyperstyle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Fhyperstyle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Fhyperstyle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuval-alaluf%2Fhyperstyle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuval-alaluf","download_url":"https://codeload.github.com/yuval-alaluf/hyperstyle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248633344,"owners_count":21136857,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2022","generative-adversarial-network","hypernetworks","stylegan","stylegan-encoder"],"created_at":"2024-08-03T02:01:15.697Z","updated_at":"2025-04-12T21:25:55.746Z","avatar_url":"https://github.com/yuval-alaluf.png","language":"Python","funding_links":[],"categories":["Image Synthesis"],"sub_categories":["AI Tools for Searching"],"readme":"# HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (CVPR 2022)\n\n\u003e Yuval Alaluf*, Omer Tov*, Ron Mokady, Rinon Gal, Amit H. Bermano  \n\u003e *Denotes equal contribution  \n\u003e \n\u003e The inversion of real images into StyleGAN's latent space is a well-studied problem. Nevertheless, applying existing approaches to real-world scenarios remains an open challenge, due to an inherent trade-off between reconstruction and editability: latent space regions which can accurately represent real images typically suffer from degraded semantic control. Recent work proposes to mitigate this trade-off by fine-tuning the generator to add the target image to well-behaved, editable regions of the latent space. While promising, this fine-tuning scheme is impractical for prevalent use as it requires a lengthy training phase for each new image. In this work, we introduce this approach into the realm of encoder-based inversion. We propose HyperStyle, a hypernetwork that learns to modulate StyleGAN's weights to faithfully express a given image in editable regions of the latent space. A naive modulation approach would require training a hypernetwork with over three billion parameters. Through careful network design, we reduce this to be in line with existing encoders. HyperStyle yields reconstructions comparable to those of optimization techniques with the near real-time inference capabilities of encoders. Lastly, we demonstrate HyperStyle's effectiveness on several applications beyond the inversion task, including the editing of out-of-domain images which were never seen during training.\n\n\u003ca href=\"https://arxiv.org/abs/2111.15666\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2111.15666-b31b1b.svg\" height=22.5\u003e\u003c/a\u003e\n\u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" height=22.5\u003e\u003c/a\u003e  \n\n\u003ca href=\"https://youtu.be/_sbXmLY2jMw\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=CVPR 2022\u0026message=5 Minute Video\u0026color=red\" height=22.5\u003e\u003c/a\u003e  \n\nInference Notebook: \u003ca href=\"http://colab.research.google.com/github/yuval-alaluf/hyperstyle/blob/master/notebooks/inference_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=20\u003e\u003c/a\u003e  \nAnimation Notebook: \u003ca href=\"http://colab.research.google.com/github/yuval-alaluf/hyperstyle/blob/master/notebooks/animations_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=20\u003e\u003c/a\u003e  \nDomain Adaptation Notebook: \u003ca href=\"http://colab.research.google.com/github/yuval-alaluf/hyperstyle/blob/master/notebooks/domain_adaptation_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=20\u003e\u003c/a\u003e  \n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/teaser.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nGiven a desired input image, our hypernetworks learn to modulate a pre-trained StyleGAN network to achieve accurate image reconstructions in editable regions of the latent space. \nDoing so enables one to effectively apply techniques such as StyleCLIP and InterFaceGAN for editing real images.\n\u003c/p\u003e\n\n# Description   \nOfficial Implementation of our HyperStyle paper for both training and evaluation. HyperStyle introduces a new approach for learning to efficiently modify a pretrained StyleGAN generator based on a given target image through the use of hypernetworks.\n\n# Table of Contents\n- [Getting Started](#getting-started)\n  * [Prerequisites](#prerequisites)\n  * [Installation](#installation)\n- [Pretrained HyperStyle Models](#pretrained-hyperstyle-models)\n  * [Auxiliary Models](#auxiliary-models)\n    + [Pretrained W-Encoders](#pretrained-w-encoders)\n    + [StyleGAN2 Generators](#stylegan2-generators)\n    + [Other Utility Models](#other-utility-models)\n- [Training](#training)\n  * [Preparing your Data](#preparing-your-data)\n  * [Preparing your Generator](#preparing-your-generator)\n  * [Training HyperStyle](#training-hyperstyle)\n    + [Additional Notes](#additional-notes)\n    + [Pre-Extracting Initial Inversions](#pre-extracting-initial-inversions)\n- [Inference](#inference)\n  * [Inference Notebooks](#inference-notebooks)\n  * [Inference Script](#inference-script)\n  * [Computing Metrics](#computing-metrics)\n  * [Editing](#editing)\n    + [Editing Faces with InterFaceGAN](#editing-faces-with-interfacegan)\n    + [Editing Cars with GanSpace](#editing-cars-with-ganspace)\n    + [Editing Faces with StyleCLIP](#editing-faces-with-styleclip)\n- [Domain Adaptation](#domain-adaptation)\n- [Repository structure](#repository-structure)\n- [Related Works](#related-works)\n- [Credits](#credits)\n- [Acknowledgments](#acknowledgments)\n- [Citation](#citation)\n\n# Getting Started\n## Prerequisites\n- Linux or macOS\n- NVIDIA GPU + CUDA CuDNN (CPU may be possible with some modifications, but is not inherently supported)\n- Python 3\n\n## Installation\n- Dependencies:  We recommend running this repository using [Anaconda](https://docs.anaconda.com/anaconda/install/).  \nAll dependencies for defining the environment are provided in `environment/hyperstyle_env.yaml`.\n\n\u003cbr\u003e\n\n# Pretrained HyperStyle Models\nIn this repository, we provide pretrained HyperStyle models for various domains.  \nAll models make use of a modified, pretrained [e4e encoder](https://github.com/omertov/encoder4editing) for obtaining an initial inversion into the W latent space.\n\nPlease download the pretrained models from the following links.\n\n| Path | Description\n| :--- | :----------\n|[Human Faces](https://drive.google.com/file/d/1C3dEIIH1y8w1-zQMCyx7rDF0ndswSXh4/view?usp=sharing)  | HyperStyle trained on the [FFHQ](https://github.com/NVlabs/ffhq-dataset) dataset.\n|[Cars](https://drive.google.com/file/d/1WZ7iNv5ENmxXFn6dzPeue1jQGNp6Nr9d/view?usp=sharing)  | HyperStyle trained on the [Stanford Cars](https://ai.stanford.edu/~jkrause/cars/car_dataset.html) dataset.\n|[Wild](https://drive.google.com/file/d/1OMAKYRp3T6wzGr0s3887rQK-5XHlJ2gp/view?usp=sharing)  | HyperStyle trained on the [AFHQ Wild](https://github.com/clovaai/stargan-v2) dataset.\n\n\n## Auxiliary Models\nIn addition, we provide various auxiliary models needed for training your own HyperStyle models from scratch.  \nThese include the pretrained e4e encoders into W, pretrained StyleGAN2 generators, and models used for loss computation.\n\n\u003cbr\u003e\n\n### Pretrained W-Encoders\n| Path | Description\n| :--- | :----------\n|[Faces W-Encoder](https://drive.google.com/file/d/1M-hsL3W_cJKs77xM1mwq2e9-J0_m7rHP/view?usp=sharing) | Pretrained e4e encoder trained on FFHQ into the W latent space.\n|[Cars W-Encoder](https://drive.google.com/file/d/1GZke8pfXMSZM9mfT-AbP1Csyddf5fas7/view?usp=sharing) | Pretrained e4e encoder trained on Stanford Cars into the W latent space.\n|[Wild W-Encoder](https://drive.google.com/file/d/1MhEHGgkTpnTanIwuHYv46i6MJeet2Nlr/view?usp=sharing) | Pretrained e4e encoder trained on AFHQ Wild into the W latent space.\n\n\u003cbr\u003e\n\n### StyleGAN2 Generators\n| Path | Description\n| :--- | :----------\n|[FFHQ StyleGAN](https://drive.google.com/file/d/1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT/view?usp=sharing) | StyleGAN2 model trained on FFHQ with 1024x1024 output resolution.\n|[LSUN Car StyleGAN](https://drive.google.com/file/d/1UmMHHB3DU1trTB8_9Fjkck5ZwArnD81B/view?usp=sharing) | StyleGAN2 model trained on LSUN Car with 512x384 output resolution.\n|[AFHQ Wild StyleGAN](https://drive.google.com/file/d/1z6IVVaCJuFTksKwp1CM3emWOVHbrBip-/view?usp=sharing) | StyleGAN-ADA model trained on AFHQ Wild with 512x512 output resolution.\n|[Toonify](https://drive.google.com/file/d/1r3XVCt_WYUKFZFxhNH-xO2dTtF6B5szu/view?usp=sharing) | Toonify generator from Doron Adler and Justin Pinkney converted to Pytorch using rosinality's conversion script, used in domain adaptation.\n|[Pixar](https://drive.google.com/file/d/1trPW-To9L63x5gaXrbAIPkOU0q9f_h05/view?usp=sharing) | Pixar generator from StyleGAN-NADA used in domain adaptation.\n\nNote: all StyleGAN models are converted from the official TensorFlow models to PyTorch using the conversion script from [rosinality](https://github.com/rosinality/stylegan2-pytorch).\n\n\u003cbr\u003e\n\n### Other Utility Models\n| Path | Description\n| :--- | :----------\n|[IR-SE50 Model](https://drive.google.com/file/d/1KW7bjndL3QG3sxBbZxreGHigcCCpsDgn/view?usp=sharing) | Pretrained IR-SE50 model taken from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch) for use in our ID loss and encoder backbone on human facial domain.\n|[ResNet-34 Model](https://download.pytorch.org/models/resnet34-333f7ec4.pth) | ResNet-34 model trained on ImageNet taken from [torchvision](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) for initializing our encoder backbone.\n|[MoCov2 Model](https://drive.google.com/file/d/18rLcNGdteX5LwT7sv_F7HWr12HpVEzVe/view) | Pretrained ResNet-50 model trained using MOCOv2 for computing MoCo-based loss on non-facial domains. The model is taken from the [official implementation](https://github.com/facebookresearch/moco).\n|[CurricularFace Backbone](https://drive.google.com/file/d/1f4IwVa2-Bn9vWLwB-bUwm53U_MlvinAj/view?usp=sharing) | Pretrained CurricularFace model taken from [HuangYG123](https://github.com/HuangYG123/CurricularFace) for use in ID similarity metric computation.\n|[MTCNN](https://drive.google.com/file/d/1tJ7ih-wbCO6zc3JhI_1ZGjmwXKKaPlja/view?usp=sharing) | Weights for MTCNN model taken from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch) for use in ID similarity metric computation. (Unpack the tar.gz to extract the 3 model weights.)\n\nBy default, we assume that all auxiliary models are downloaded and saved to the directory `pretrained_models`. \nHowever, you may use your own paths by changing the necessary values in `configs/path_configs.py`. \n\n\u003cbr\u003e\n\u003cbr\u003e\n\n# Training\n\n## Preparing your Data\nIn order to train HyperStyle on your own data, you should perform the following steps: \n1. Update `configs/paths_config.py` with the necessary data paths and model paths for training and inference.\n```\ndataset_paths = {\n    'train_data': '/path/to/train/data'\n    'test_data': '/path/to/test/data',\n}\n```\n2. Configure a new dataset under the `DATASETS` variable defined in `configs/data_configs.py`. There, you should define the source/target data paths for the train and test sets as well as the transforms to be used for training and inference.\n```\nDATASETS = {\n\t'my_hypernet': {\n\t\t'transforms': transforms_config.EncodeTransforms,   # can define a custom transform, if desired\n\t\t'train_source_root': dataset_paths['train_data'],\n\t\t'train_target_root': dataset_paths['train_data'],\n\t\t'test_source_root': dataset_paths['test_data'],\n\t\t'test_target_root': dataset_paths['test_data'],\n\t}\n}\n```\n3. To train with your newly defined dataset, simply use the flag `--dataset_type my_hypernet`.\n\n\u003cbr\u003e\n\n## Preparing your Generator\nIn this work, we use rosinality's [StyleGAN2 implementation](https://github.com/rosinality/stylegan2-pytorch). \nIf you wish to use your own generator trained using NVIDIA's implementation there are a few options we recommend:\n1. Using NVIDIA's StyleGAN2 / StyleGAN-ADA TensorFlow implementation.  \n   You can then convert the TensorFlow `.pkl` checkpoints to the supported format using the conversion script found in [rosinality's implementation](https://github.com/rosinality/stylegan2-pytorch#convert-weight-from-official-checkpoints).\n2. Using NVIDIA's StyleGAN-ADA PyTorch implementation.  \n   You can then convert the PyTorch `.pkl` checkpoints to the supported format using the conversion script created by [Justin Pinkney](https://github.com/justinpinkney) found in [dvschultz's fork](https://github.com/dvschultz/stylegan2-ada-pytorch/blob/main/SG2_ADA_PT_to_Rosinality.ipynb).  \n\nOnce you have the converted `.pt` files, you should be ready to use them in this repository.  \n\n\u003cbr\u003e\n\n## Training HyperStyle\nThe main training script can be found in `scripts/train.py`.  \nIntermediate training results are saved to `opts.exp_dir`. This includes checkpoints, train outputs, and test outputs.  \nAdditionally, if you have tensorboard installed, you can visualize tensorboard logs in `opts.exp_dir/logs`. \n\nTraining HyperStyle with the settings used in the paper can be done by running the following command. Here, we provide an example for training on the human faces domain:\n\n```\npython scripts/train.py \\\n--dataset_type=ffhq_hypernet \\\n--encoder_type=SharedWeightsHyperNetResNet \\\n--exp_dir=experiments/hyperstyle \\\n--workers=8 \\\n--batch_size=8 \\\n--test_batch_size=8 \\\n--test_workers=8 \\\n--val_interval=5000 \\\n--save_interval=10000 \\\n--lpips_lambda=0.8 \\\n--l2_lambda=1 \\\n--id_lambda=0.1 \\\n--n_iters_per_batch=5 \\\n--max_val_batches=150 \\\n--output_size=1024 \\\n--load_w_encoder \\\n--w_encoder_checkpoint_path pretrained_models/faces_w_encoder \\ \n--layers_to_tune=0,2,3,5,6,8,9,11,12,14,15,17,18,20,21,23,24\n```\n  \n\n### Additional Notes\n- To select which generator layers to tune with the hypernetwork, you can use the `--layers_to_tune` flag.\n    - By default, we will alter all non-toRGB convolutional layers. \n- ID/similarity losses: \n    - For the human facial domain we use a specialized ID loss based on a pretrained ArcFace network. This is set using the flag `--id_lambda=0.1`.\n    - For all other domains, please set `--id_lambda=0` and `--moco_lambda=0.5` to use the MoCo-based similarity loss from Tov et al. \n        - Note, you __cannot__ set both `id_lambda` and `moco_lambda` to be active simultaneously.\n- You should also adjust the `--output_size` and `--stylegan_weights` flags according to your StyleGAN generator. \n- To use the HyperStyle with Refinement Blocks based on separable convolutions (see the ablation study), you can set the `encoder_type` to `SharedWeightsHyperNetResNetSeparable`.\n- See `options/train_options.py` for all training-specific flags. \n\n\u003cbr\u003e \n\n### Pre-Extracting Initial Inversions\nTo provide a small speed-up and slightly reduce memory consumption, we could pre-extract all the latents and inversions from our W-encoder rather than inverting on the fly during training.   \nWe provide an example for how to do this in `configs/data_configs.py` under the `ffhq_hypernet_pre_extract` dataset.  \nHere, we must define: \n- `train_source_root`: the directory holding all the initial inversions\n- `train_target_root`: the directory holding all target images (i.e., original images)\n- `train_latents_path`: the `.npy` file holding the latents for the inversions \n    of the form  \n    ```latents = { \"0.jpg\": latent, \"1.jpg\": latent, ... }```.\n\nAnd similarly for the test dataset.  \n\nPerforming the above and pre-extracting the latents and inversions could also allow you to train HyperStyle using latents from various encoders such as pSp, e4e, and ReStyle into W+ rather than using our pretrained encoder into W.\n\nDuring training, we will use the `LatentsImagesDataset` for loading the inversion, latent code, and target image. \n\n\u003cbr\u003e\n\n# Inference\n\n## Inference Notebooks\nTo help visualize the results of ReStyle we provide a Jupyter notebook found in `notebooks/inference_playground.ipynb`.   \nThe notebook will download the pretrained models and run inference on the images found in `notebooks/images` or on images of your choosing. It is recommended to run this in [Google Colab](http://colab.research.google.com/github/yuval-alaluf/hyperstyle/blob/master/notebooks/inference_playground.ipynb).\n\nWe have also provided a notebook for generating interpolation videos such as those found in the project page. This \nnotebook can be run using Google Colab [here](http://colab.research.google.com/github/yuval-alaluf/hyperstyle/blob/master/notebooks/animations_playground.ipynb).\n\n\u003cbr\u003e\n\n## Inference Script\nYou can use `scripts/inference.py` to apply a trained HyperStyle model on a set of images:\n```\npython scripts/inference.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=5 \\\n--load_w_encoder \\\n--w_encoder_checkpoint_path /path/to/w_encoder.pt\n```\nThis script will save each step's outputs in a separate sub-directory (e.g., the outputs of step `i` will be saved in `/path/to/experiment/inference_results/i`). In addition, \nside-by-side reconstruction results will be saved to `/path/to/experiment/inference_coupled`.\n\nNotes: \n- By default, the images will be saved at their original output resolutions (e.g., `1024x1024` for faces, `512x384` for cars).\n    - If you wish to save outputs resized to resolutions of `256x256` (or `256x192` for cars), you can do so by adding the flag `--resize_outputs`.\n- This script will also save all the latents as an `.npy` file in a dictionary format as follows: \n    - ```latents = { \"0.jpg\": latent, \"1.jpg\": latent, ... }```\n- In addition, by setting the flag `--save_weight_deltas`, we will save the final predicted weight deltas for each image.  \n    - These will be saved as `.npy` files in the sub-directory `weight_deltas`.\n    - Setting this flag is important if you would like to apply them for some down-stream task. For example, if you would like apply them for editing using StyleCLIP (see below).\n\n\u003cbr\u003e\n\n## Computing Metrics\nGiven a trained model and generated outputs, we can compute the loss metrics on a given dataset.  \nThese scripts receive the inference output directory and ground truth directory.\n```\npython scripts/calc_losses_on_images.py \\\n--metrics lpips,l2,msssim \\\n--output_path=/path/to/experiment/inference_results \\\n--gt_path=/path/to/test_images\n```\nHere, we can compute multiple metrics using a comma-separated list with the flag `--metrics`.\n\nSimilarly, to compute the ID similarity: \n```\npython scripts/calc_losses_on_images.py \\\n--output_path=/path/to/experiment/inference_results \\\n--gt_path=/path/to/test_images\n```\n\nThese scripts will traverse through each sub-directory of `output_path` to compute the metrics on each step's output images.\n\n\u003cbr\u003e\n\n## Editing\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/dicaprio_styleclip.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/blunt_interfacegan.jpg\" width=\"800px\"/\u003e\n\u003cimg src=\"docs/cars_ganspace.jpg\" width=\"800px\"/\u003e\n\u003cbr\u003e\nEditing results obtained via HyperStyle using StyleCLIP, InterFaceGAN, and GanSpace, respectively.\n\u003c/p\u003e\n\nFor performing inference and editing using InterFaceGAN (for faces) and GANSpace (for cars), you can run `editing/inference_face_editing.py` and `editing/inference_cars_editing.py`.\n\n\u003cbr\u003e\n\n### Editing Faces with InterFaceGAN\n```\npython editing/inference_face_editing.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=3 \\\n--edit_directions=age,pose,smile \\\n--factor_range=5 \\\n----load_w_encoder\n```\nFor InterFaceGAN we currently support edits of age, pose, and smile. \n\n\u003cbr\u003e\n\n### Editing Cars with GanSpace\n```\npython editing/inference_cars_editing.py \\\n--exp_dir=/path/to/experiment \\\n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--n_iters_per_batch=3\n```\nFor GANSpace we currently support edits of pose, cube, color, and grass.\n\nThese scripts will perform the inversion immediately followed by the latent space edit.  \nFor each image, we save the original image followed by the inversion and the resulting edits.  \n\n\u003cbr\u003e\n\n### Editing Faces with StyleCLIP \nIn addition, we support editing with StyleCLIP's global directions approach on the human faces domain. Editing can be performed by running `editing/styleclip/edit.py`. For example,\n```\npython editing/styleclip/edit.py \\\n--exp_dir /path/to/experiment \\   \n--weight_deltas_path /path/to/experiment/weight_deltas \\\n--neutral_text \"a face\" \\\n--target_tex \"a face with a beard\" \\\n```\n**Note:** before running the above script, you need to install the official CLIP package:\n```\npip install git+https://github.com/openai/CLIP.git\n``` \n\n**Note:** we assume that `latents.npy` and the directory `weight_deltas`, obtained by running \n`inference.py` are both saved in the the given `exp_dir`.  \nFor each input image we save a grid of results with different values of `alpha` and `beta` as defined in StyleCLIP.  \n\n\u003cbr\u003e\n\n# Domain Adaptation\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/domain_adaptation.jpg\" width=\"800px\"/\u003e\n\u003cbr\u003e\nDomain adaptation results obtained via HyperStyle by applying the learned weight offsets to various fine-tuned generators.\n\u003c/p\u003e\n\nIn `scripts/run_domain_adaptation.py`, we provide a script for performing domain adaptation from the FFHQ domain to another (e.g., toons or sketches).\nSpecifically, using a HyperStyle network trained on FFHQ, we can predict the weight offsets for a given input image. \nWe can then apply the predicted weight offsets to a fine-tuned generator to obtain a translated image that better preserves the input image.\n\nA example command is provided below: \n```\npython scripts/run_domain_adaptation.py \\\n--exp_dir /path/to/experiment \\   \n--checkpoint_path=experiment/checkpoints/best_model.pt \\\n--data_path=/path/to/test_data \\\n--test_batch_size=4 \\\n--test_workers=4 \\\n--load_w_encoder \\\n--w_encoder_checkpoint_path=pretrained_models/faces_w_encoder.pt \\\n--restyle_checkpoint_path=pretrained_models/restyle_e4e_ffhq_encode.pt \\\n--finetuned_generator_checkpoint_path=pretrained_models/pixar.pt \\\n--n_iters_per_batch=2 \\\n--restyle_n_iterations=2\n```\n\nHere, since we are performing a translation to a new domain, we recommend setting the number of iterations to a small number (e.g., 2-3).  \n\nBelow we provide links the pre-trained ReStyle-e4e network and various fine-tuned generators. \n\n| Path | Description\n| :--- | :----------\n|[FFHQ ReStyle e4e](https://drive.google.com/file/d/1e2oXVeBPXMQoUoC_4TNwAWpOPpSEhE_e/view?usp=sharing) | ReStyle e4e trained on FFHQ with 1024x1024 output resolution.\n|[Toonify](https://drive.google.com/file/d/1r3XVCt_WYUKFZFxhNH-xO2dTtF6B5szu/view?usp=sharing) | Toonify generator from Doron Adler and Justin Pinkney converted to Pytorch using rosinality's conversion script, used in domain adaptation.\n|[Pixar](https://drive.google.com/file/d/1trPW-To9L63x5gaXrbAIPkOU0q9f_h05/view?usp=sharing) | Pixar generator from StyleGAN-NADA used in domain adaptation.\n|[Sketch](https://drive.google.com/file/d/1aHhzmxT7eD90txAN93zCl8o9CUVbMFnD/view?usp=sharing) | Sketch generator from StyleGAN-NADA used in domain adaptation.\n|[Disney Princess](https://drive.google.com/file/d/1rXHZu4Vd0l_KCiCxGbwL9Xtka7n3S2NB/view?usp=sharing) | Disney princess generator from StyleGAN-NADA used in domain adaptation.\n\n\n\u003cbr\u003e\n\n# Repository structure\n| Path | Description \u003cimg width=200\u003e\n| :--- | :---\n| hyperstyle | Repository root folder\n| \u0026boxvr;\u0026nbsp; configs | Folder containing configs defining model/data paths and data transforms\n| \u0026boxvr;\u0026nbsp; criteria | Folder containing various loss criterias for training\n| \u0026boxvr;\u0026nbsp; datasets | Folder with various dataset objects\n| \u0026boxvr;\u0026nbsp; docs | Folder containing images displayed in the README\n| \u0026boxvr;\u0026nbsp; environment | Folder containing Anaconda environment used in our experiments\n| \u0026boxvr;\u0026nbsp; editing | Folder containing scripts for applying various editing techniques\n| \u0026boxvr;\u0026nbsp; licenses | Folder containing licenses of the open source projects used in this repository\n| \u0026boxvr; models | Folder containing all the models and training objects\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; encoders | Folder containing various encoder architecture implementations such as the W-encoder, pSp, and e4e\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; hypernetworks | Implementations of our hypernetworks and Refinement Blocks\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; mtcnn | MTCNN implementation from [TreB1eN](https://github.com/TreB1eN/InsightFace_Pytorch)\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; stylegan2 | StyleGAN2 model from [rosinality](https://github.com/rosinality/stylegan2-pytorch)\n| \u0026boxv;\u0026nbsp; \u0026boxvr;\u0026nbsp; hyperstyle.py | Main class for our HyperStyle network\n| \u0026boxvr;\u0026nbsp; notebooks | Folder with jupyter notebooks containing HyperStyle inference playgrounds\n| \u0026boxvr;\u0026nbsp; options | Folder with training and test command-line options\n| \u0026boxvr;\u0026nbsp; scripts | Folder with running scripts for training, inference, and metric computations\n| \u0026boxvr;\u0026nbsp; training | Folder with main training logic and Ranger implementation from [lessw2020](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)\n| \u0026boxvr;\u0026nbsp; utils | Folder with various utility functions\n| \u003cimg width=300\u003e | \u003cimg\u003e\n\n# Related Works \nMany GAN inversion techniques focus on finding a latent code that most accurately reconstructs a given image using a fixed, pre-trained generator. These works include encoder-based approaches such as [pSp](https://github.com/eladrich/pixel2style2pixel), [e4e](https://github.com/omertov/encoder4editing) and [ReStyle](https://github.com/yuval-alaluf/restyle-encoder), and optimization techniques such as those from [Abdal et al.](https://arxiv.org/abs/1904.03189) and [Zhu et al.](https://arxiv.org/abs/2012.09036), among many others.\n\nIn contrast, HyperStyle learns to modulate the **weights** of a pre-trained StyleGAN using a hypernetwork to achieve more accurate reconstructions. Previous generator tuning approaches performed a per-image optimization for fine-tuning the generator weights ([Roich et. al](https://github.com/danielroich/PTI)) or feature activations ([Bau et al.](https://arxiv.org/abs/2005.07727)).\n\nGiven our inversions we can apply off-the-shelf editing techniques such as [StyleCLIP](https://github.com/orpatashnik/StyleCLIP), [InterFaceGAN](https://github.com/genforce/interfacegan), and [GANSpace](https://github.com/harskish/ganspace), even on the modified generator.  \n\nFinally, we can apply weight offsets learned on HyperStyle trained on FFHQ to fine-tuned generators such as those obtained from [StyleGAN-NADA](https://github.com/rinongal/StyleGAN-nada), resulting in more faithful translations.\n\n# Credits\n**StyleGAN2 model and implementation:**  \nhttps://github.com/rosinality/stylegan2-pytorch  \nCopyright (c) 2019 Kim Seonghyeon  \nLicense (MIT) https://github.com/rosinality/stylegan2-pytorch/blob/master/LICENSE  \n\n**IR-SE50 model and implementations:**  \nhttps://github.com/TreB1eN/InsightFace_Pytorch  \nCopyright (c) 2018 TreB1eN  \nLicense (MIT) https://github.com/TreB1eN/InsightFace_Pytorch/blob/master/LICENSE  \n\n**Ranger optimizer implementation:**  \nhttps://github.com/lessw2020/Ranger-Deep-Learning-Optimizer   \nLicense (Apache License 2.0) https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/master/LICENSE  \n\n**LPIPS model and implementation:**  \nhttps://github.com/S-aiueo32/lpips-pytorch  \nCopyright (c) 2020, Sou Uchida  \nLicense (BSD 2-Clause) https://github.com/S-aiueo32/lpips-pytorch/blob/master/LICENSE  \n\n**pSp model and implementation:**   \nhttps://github.com/eladrich/pixel2style2pixel  \nCopyright (c) 2020 Elad Richardson, Yuval Alaluf  \nLicense (MIT) https://github.com/eladrich/pixel2style2pixel/blob/master/LICENSE\n\n**e4e model and implementation:**   \nhttps://github.com/omertov/encoder4editing  \nCopyright (c) 2021 omertov   \nLicense (MIT) https://github.com/omertov/encoder4editing/blob/main/LICENSE\n\n**ReStyle model and implementation:**  \nhttps://github.com/yuval-alaluf/restyle-encoder  \nCopyright (c) 2021 Yuval Alaluf    \nLicense (MIT) https://github.com/yuval-alaluf/restyle-encoder/blob/main/LICENSE\n\n**StyleCLIP implementation:**  \nhttps://github.com/orpatashnik/StyleCLIP   \nCopyright (c) 2021 Or Patashnik, Zongze Wu  \nhttps://github.com/orpatashnik/StyleCLIP/blob/main/LICENSE  \n\n**StyleGAN-NADA models:**  \nhttps://github.com/rinongal/StyleGAN-nada   \nCopyright (c) 2021 rinongal  \nhttps://github.com/rinongal/StyleGAN-nada/blob/main/LICENSE  \n\n\n**Please Note**: The CUDA files under the [StyleGAN2 ops directory](https://github.com/yuval-alaluf/hyperstyle/tree/master/models/stylegan2/op) are made available under the [Nvidia Source Code License-NC](https://nvlabs.github.io/stylegan2/license.html)\n\n# Acknowledgments\nThis code borrows from [pixel2style2pixel](https://github.com/eladrich/pixel2style2pixel), \n[encoder4editing](https://github.com/omertov/encoder4editing), and [ReStyle](https://github.com/yuval-alaluf/restyle-encoder).\n\n\n# Citation\nIf you use this code for your research, please cite the following work:\n```\n@misc{alaluf2021hyperstyle,\n      title={HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing}, \n      author={Yuval Alaluf and Omer Tov and Ron Mokady and Rinon Gal and Amit H. Bermano},\n      year={2021},\n      eprint={2111.15666},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuval-alaluf%2Fhyperstyle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuval-alaluf%2Fhyperstyle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuval-alaluf%2Fhyperstyle/lists"}