{"id":17000107,"url":"https://github.com/giannisdaras/ilo","last_synced_at":"2026-04-02T03:50:52.606Z","repository":{"id":50357616,"uuid":"312111308","full_name":"giannisdaras/ilo","owner":"giannisdaras","description":"[ICML 2021] Official implementation: Intermediate Layer Optimization for Inverse Problems using Deep Generative Models","archived":false,"fork":false,"pushed_at":"2022-07-28T12:11:44.000Z","size":51036,"stargazers_count":119,"open_issues_count":2,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-27T21:50:25.985Z","etag":null,"topics":["compressed-sensing","denoising","ilo","inpainting","stylegan","stylegan-pytorch","stylegan2","super-resolution"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/giannisdaras.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-11T23:01:41.000Z","updated_at":"2025-01-09T23:54:50.000Z","dependencies_parsed_at":"2022-09-23T15:00:38.558Z","dependency_job_id":null,"html_url":"https://github.com/giannisdaras/ilo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisdaras%2Filo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisdaras%2Filo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisdaras%2Filo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/giannisdaras%2Filo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/giannisdaras","download_url":"https://codeload.github.com/giannisdaras/ilo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243858794,"owners_count":20359257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compressed-sensing","denoising","ilo","inpainting","stylegan","stylegan-pytorch","stylegan2","super-resolution"],"created_at":"2024-10-14T04:11:04.040Z","updated_at":"2026-04-02T03:50:52.565Z","avatar_url":"https://github.com/giannisdaras.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Official Implementation: Intermediate Layer Optimization for Inverse Problems using Deep Generative Models\n![](https://img.shields.io/badge/pytorch-green) ![](https://img.shields.io/badge/ilo-ilo-green)\n\nThis repository hosts the official PyTorch implementation of the paper: \"Intermediate Layer Optimization for Inverse Problems using Deep Generative Models\".\n\n**Important Note**: The results of this paper have been improved by our ICML 22 work: [Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problems](https://arxiv.org/abs/2206.09104)([code](https://github.com/giannisdaras/sgilo)). This new work achieves greater results, especially in the extremely sparse measurements regime.\n\nPaper: https://arxiv.org/abs/2102.07364\n\n\nAuthored by: Giannis Daras, Joseph Dean (equal contribution), Ajil Jalal, Alexandros G. Dimakis\n\n\nColab demo: \n\n[![Colab](https://camo.githubusercontent.com/52feade06f2fecbf006889a904d221e6a730c194/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667)](https://rb.gy/tj1ypl)\n\n## Abstract\n\u003e We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small l1 ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.\n\n## Results\n\n### Inpainting\n![](results/inpainting.png)\n\n\n### Super-resolution\n![](results/super_res.png)\n\n### Denoising\n![](results/denoising.png)\n\n### Morphing using a robust classifier\n![](results/morphing.png)\n\n\n##  Installation\nFirst install the python dependencies by running `pip install -r requirements.txt`.\n\nNext, download dependency files:\n\n`gdown --id 1c1qtz3MVTAvJpYvsMIR5MoSvdiwN2DGb` (shape predictor)\n\n`gdown --id 1JCBiKY_yUixTa6F1eflABL88T4cii2GR` (stylegan pre-trained checkpoint)\n\nIf you don't have gdown installed, run: `pip install gdown` first.\n\n\n\n## Examples\n\n### Image Preprocessing\nOur `prepare_image.py` script offers some basic image preprocessing utilities.\nA basic config for the script is given in `configs/preprocess.yaml`.\nTo change the image preprocessing task or to stack various preprocessing operations together, you need to adjust the arguments of the script.\nYou can do that by either adjusting the file directly or by passing the corresponding CLI arguments as shown in the following examples.\n\n#### Interactive masking\nYou can use this tool to create (interactively) masks to your images.\n\nExample command:\n\n`python prepare_image.py preprocessing=\\[interactive_mask\\] input_files=\\[files/original/turing.png,files/original/mona_lisa.png\\]`\n\n#### Hard coded mask\nYou can hard code the location of the mask.\n\nExample command:\n\n`python prepare_image.py preprocessing=\\[mask\\] input_files=\\[files/original/mona_lisa.png\\] bounding_box.horizontal=\\[100,1000\\] bounding_box.vertical=\\[100,1000\\]`\n\n#### Random mask\nDo random inpainting.\n\nExample command:\n\n`python prepare_image.py preprocessing=\\[remove_pixels\\] input_files=\\[files/original/mona_lisa.png\\] observed_percentage=\\[10,20,30\\] per_input=3`\n\n\n#### Automatically align images\n\nExample command:\n\n`python prepare_image.py preprocessing=\\[align\\] input_files=\\[files/original/mona_lisa.png\\]`\n\n#### Chain operations\n\nExample command:\n\n`python prepare_image.py preprocessing=\\[interactive_mask,remove_pixels\\] input_files=\\[files/original/mona_lisa.png\\]`\n\n### Solving inverse problems\nThe script `main.py` can be used to do inpainting, super-resolution, denoising, out-of-distribution generation, etc.\nThe script in default mode is configured by the file `configs/config.yaml`.\n\nA config file should have the following form:\n```\n# configures the working direction of hydra.\n# If using relative paths, the working directory should be the root folder of ILO.\nhydra:\n  run:\n    dir: '.'\n\n# only StyleGAN is supported for now.\nmodel_type: 'stylegan'\n\nstylegan:\n  seed: 42\n  device: cuda\n\n  # The pre-trained StyleGAN checkpoint\n  ckpt: stylegan2-ffhq-config-f.pt\n\n  ## weights of different losses\n  geocross: 0.01\n  mse: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n  pe: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n  dead_zone_linear: [1.0, 1.0, 1.0, 1.0]\n  # tolerance of dead zone linear function\n  dead_zone_linear_alpha: 0.1\n  # Additional loss term to a reference image. \\\n  # Leave it to 0, unless you know what you are doing.\n  reference_loss: 0.0\n\n  # hack for making lpips work\n  lpips_method: 'fill'\n  # classifier used in LPIPS loss\n  cls_name: vgg16\n\n\n  ## Task specific\n\n  # circulant matrices\n  fast_compress: false\n  observed_percentage: 80\n\n  # can be decreased for super-resolution\n  image_size:\n    - 1024\n    - 1024\n\n  # controls whether we want to match black pixels. Leave to true for inpainting.\n  mask_black_pixels: true\n\n  ## Optimization\n\n  # how many steps to run per layer\n  steps: '1000, 1000, 1000, 1000, 1000, 1000'\n  # whether to project latents to unit ball\n  project: true\n  # learning rate per layer\n  lr: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]\n  # whether to schedule per layer or in total\n  lr_same_pace: false\n  # Which layer to start optimizing from. Use 0, unless there are saved noises.\n  # If you want to skip optimization in some layers, just use 0 to the corresponding indices of steps.\n  start_layer: 0\n  end_layer: 8\n  # Whether to restore opt. variables from previous run.\n  restore: false\n  # paths to previous opt. variables\n  saved_noises:\n    - files/noises.pt\n    - files/latent.pt\n    - files/gen_outs.pt\n  # projections\n  do_project_gen_out: false\n  do_project_noises: false\n  do_project_latent: false\n  max_radius_gen_out: [1000, 1000, 6000]\n  max_radius_noises: [1000, 1000, 6000]\n  max_radius_latent: [100, 1000, 6000]\n\n  # specific to video. Leave it as is, unless you know what you are doing.\n  is_video: false\n  max_frame_radius_gen_out: [200]\n  max_frame_radius_noises: [5]\n  max_frame_radius_latent: [200]\n  video_freq: 30\n  per_frame_steps: '100'\n\n  ## files\n  # If is_sequence=True, then the input_files and the output_files should be directories.\n  is_sequence: false\n  input_files:\n    - files/original/turing.png\n  output_files:\n    - files/inpainting/turing.png\n\n  ## specific to datasets\n  dataset_type: CelebaHQ\n  # if is_dataset=true, then we are sampling from a dataset instead of having fixed files.\n  is_dataset: false\n  # how many samples to get from dataset. Is activated only when is_dataset=true\n  num_dataset: 1\n  # extension of dataset files (for glob)\n  files_ext: '.png'\n\n  # logging\n  # if save_latent=True, then the optimization variables will be saved and can be used in later run.\n  save_latent: false\n  # if true, intermediate frames from optimization are saved.\n  save_gif: false\n  # determines how often we save intermediate frames. Activated only if save_gif=True.\n  save_every: 50\n  # determined whether we our final generated image is the one with the lowest MSE to a reference image.\n  save_on_ref: false\n```\n\n\nThe role of each parameter is explained in the comments.\nBelow, we give some example commands for different inverse problems.\nFor all the examples we are using the default value for `input_files`, but feel free to change it by passing the appropriate CLI argument (or by changing the config file) to run it with your own images.\n\n\nFor any of these tasks, you can either enable l1 projections or no.\nEnabling l1 projections helps controlling how close the generated images are to the range of StyleGAN.\nThe caveat is that tuning the radii of these projections can be particularly toilsome.\nFor most settings, you will be able to get particularly good results with disabled projections.\nProjections are particularly useful when considering out-of-distribution generation.\nSee there for an example on how to use them.\n\n#### Inpainting\nFor inpainting, we want to disable perceptual loss and use only MSE loss.\nIf we have enough measurements, we enable LPIPS loss as well.\n\nExample command:\n\n`python main.py stylegan.mse=\\[1.0,1.0,1.0,1.0\\] stylegan.pe=\\[0.0,0.0,0.0,0.0\\] stylegan.steps=\\[25,25,25,25\\]`\n\n\n#### Super-resolution\nFor super-resolution, you only need to specify image-size.\nIf the original image is higher res, BicubicDownSample will be used for\ndownscaling prior to inversion.\n\nExample command:\n\n`python main.py stylegan.image_size=\\[256,256\\] stylegan.mse=\\[1.0,1.0,1.0,1.0\\] stylegan.pe=\\[1.0,1.0,1.0,1.0\\] stylegan.steps=\\[25,25,25,25\\]`\n\nIf using LPIPS, make use that your image is at least 256x256.\n\n#### Compressed sensing with circulant matrices\nFor compressed sensing with partial circulant matrices, we need to enable the `--fast-compress` argument and specify\nthe `--observed percentage`.\n\n\nExample command:\n`python main.py stylegan.mse=\\[1.0,1.0,1.0,1.0\\] stylegan.pe=\\[0.0,0.0,0.0,0.0\\] stylegan.steps=\\[300,200,100\\] stylegan.fast_compress=1 stylegan.observed_percentage=50`\n\n\n## Acknowledgments and License\n\nWe use the StyleGAN-2 PyTorch implementation of the following repository: https://github.com/rosinality/stylegan2-pytorch.\nWe wholeheartedly thank the author for open-sourcing this implementation.\n\nThe PyTorch implementation is based on the official Tensorflow implementation: https://github.com/NVlabs/stylegan2.\nWe are grateful to the authors of StyleGAN-2 for their work and their open-sourced code and models.\n\nPlease refer to the license files listed in the repositories [1](https://github.com/rosinality/stylegan2-pytorch), [2](https://github.com/NVlabs/stylegan2).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiannisdaras%2Filo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgiannisdaras%2Filo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgiannisdaras%2Filo/lists"}