{"id":16836289,"url":"https://github.com/aleju/gan-error-avoidance","last_synced_at":"2025-04-11T05:10:43.494Z","repository":{"id":66066014,"uuid":"96132631","full_name":"aleju/gan-error-avoidance","owner":"aleju","description":"Learning to Avoid Errors in GANs by Input Space Manipulation (Code for paper)","archived":false,"fork":false,"pushed_at":"2017-07-07T09:40:06.000Z","size":10219,"stargazers_count":23,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-25T03:12:19.113Z","etag":null,"topics":["artifacts","celeba","deep-learning","gan","generative-adversarial-network","pytorch","weight-normalization"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1707.00768","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aleju.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-03T16:55:45.000Z","updated_at":"2021-10-23T12:34:55.000Z","dependencies_parsed_at":"2023-04-22T01:35:32.231Z","dependency_job_id":null,"html_url":"https://github.com/aleju/gan-error-avoidance","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleju%2Fgan-error-avoidance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleju%2Fgan-error-avoidance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleju%2Fgan-error-avoidance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleju%2Fgan-error-avoidance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aleju","download_url":"https://codeload.github.com/aleju/gan-error-avoidance/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345266,"owners_count":21088244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artifacts","celeba","deep-learning","gan","generative-adversarial-network","pytorch","weight-normalization"],"created_at":"2024-10-13T12:12:58.363Z","updated_at":"2025-04-11T05:10:43.483Z","avatar_url":"https://github.com/aleju.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\nThis repository contains code for the paper [(1707.00768) Learning to Avoid Errors in GANs by Manipulating Input Spaces](https://arxiv.org/abs/1707.00768).\nThe basic idea of the paper is to distort the input space (i.e. the noise vectors) in a way that the image quality is improved.\nMore intuitively, the generator is trained to understand its own weaknesses and allowed to simply evade input vectors that would result in bad quality -- though it is punished whenever it does that.\nThe paper introduces LIS modules, which consist of residual fully connected layers and are placed at the beginning of the generator.\nThese modules manipulate the input vectors and thereby reduce the frequency of image artifacts (especially the frequency of completely broken images).\nThey are almost free with regards to computation/memory.\nImage diversity is mostly unaffected by them.\n\n![Idea](images/idea.jpg?raw=true \"idea\")\n\nThe code in this repository is based on the one from the paper \"On the Effects of Batch and Weight Normalization in Generative Adversarial Networks\" (see [repository](https://github.com/stormraiser/GAN-weight-norm)).\nThe command line arguments are largely similar or the same.\n\n\n# Images\n\nG-LIS with 1 LIS module, trained on CelebA (300k batches):\n\n![CelebA, 80x80, 1 LIS module](images/glis-exp04-riter1-r1-small.jpg?raw=true \"CelebA, 80x80, 1 LIS module\")\n\n\nG-LIS with 3 LIS modules, trained on CelebA (300k batches).\nFirst image per group shows the image generated by G when using the original noise vector (i.e. input space).\nAll following images show the images generated by G when using the noise vectors (i.e. input spaces) generated by the first, second or third LIS module.\n\n![CelebA, 80x80, 3 LIS modules, chains](images/glis-exp01-riter3-chains-large.jpg?raw=true \"CelebA, 80x80, 3 LIS modules, chains\")\n\n\nG-LIS with 1 LIS module at 160x160, trained on CelebA (570k batches). Note that while some faces have visible visual artifacts,\nthere are no completely broken ones (to the point where the face is hardly recognizable).\nWhen training a similar model without LIS modules, it will generate at least some broken faces.\nLIS modules prevent such gross errors.\n\n![CelebA, 160x160, 1 LIS module](images/glis-exp05-riter1-160x160-large.jpg?raw=true \"CelebA, 160x160, 1 LIS module\")\n\n\nG-LIS with 3 LIS modules, trained on 120flowers (600k batches):\n\n![120flowers, 80x80, 3 LIS modules](images/glis-exp10-riter3-120flowers-large.jpg?raw=true \"120flowers, 80x80, 3 LIS module\")\n\n\nG-LIS with 1 LIS module, trained on 10kcats (600k batches, 3 layers instead of 4 in G and D, NN-upsampling, dropout before last layer in D):\n\n![10kcats, 64x64, 1 LIS module](images/glis-exp16-riter1-large.jpg?raw=true \"10kcats, 64x64, 1 LIS module\")\n\n\nExample interpolation, same model as before:\n\n![10kcats, 64x64, 1 LIS module, interpolation](images/glis-exp16-riter1-interpolation.jpg?raw=true \"10kcats, 64x64, 1 LIS module, interpolation\")\n\n\nG-LIS with 1 LIS module, trained on LSUN churches (500k batches):\n\n![LSUN churches, 80x80, 1 LIS module](images/glis-exp13-riter1-churches-large.jpg?raw=true \"LSUN churches, 80x80, 1 LIS module\")\n\n\nG-LIS with 3 LIS modules (with 2 of 3 modules executed for this image), trained on CIFAR-10 (1M batches):\n\n![CIFAR-10, 32x32, 3 LIS modules](images/glis-exp11-riter2of3-cifar10-large.jpg?raw=true \"CIFAR-10, 32x32, 3 LIS modules\")\n\n\nR-iterative, images showing progressive improvements per iteration (trained on CelebA, 130k batches):\n\n![R-iterative, CelebA, 80x80, 3 iterations](images/riterative-exp01-riter3-chains-small.jpg?raw=true \"R-iterative, 80x80, 3 iterations\")\n\n\nR-separate correcting artifacts, before/after applying the model (2.5k batches for R, spatial dropout at 10%):\n\n![R-separate, CelebA, 80x80](images/glis-exp02-riter0-rsep2500-g50k-sd010-both-small.jpg?raw=true \"R-separate, CelebA, 80x80\")\n\n\n# Training Videos\n\nTraining on CelebA at 160x160 with one LIS module. This seems to produce more errors than 80x80 with one module.\nThe video shows faces generated from the original noise vectors (each left) and the same noise vectors after they were manipulated by the LIS module.\n\n[![Training on CelebA 160x160 before after](images/videos/glis_celeba_160x160_before_after.jpg)](http://www.youtube.com/watch?v=SkC76XoYOrs)\n\n\nSame training as above, but this time only the second image of each pair is shown (i.e. only with applied LIS module).\n\n[![Training on CelebA 160x160 after](images/videos/glis_celeba_160x160_only_after.jpg)](http://www.youtube.com/watch?v=gT71geKc-cY)\n\n\nSimilar to the previous video, but training was done on the 10kcats datasets at 64x64 with 3 LIS modules (showing the results of the last module).\nNote that this is not the exactly same model as for the images above. The eyes seem to have less artifacts here.\nThis model used one more layer in G and D as well as two more LIS modules.\nOne of these factors likely resulted in less structural artifacts.\nDue to aggressive cropping the training images ended up looking a bit blurry.\n\n[![Training on 10kcats 160x160 after](images/videos/glis_10kcats_160x160_only_after.jpg)](http://www.youtube.com/watch?v=16PP68-onwQ)\n\n\n# LIS module\n\nBasic architecture of LIS modules:\n\n![LIS modules](images/lis-modules.jpg?raw=true \"LIS module\")\n\n\n# Dependencies\n\n* python2.7 (some uses of xrange in the code)\n* pytorch at commit 065c59860a8bb5a01634bb0ce9eff6647ae1f342. At some later commit (don't know exactly which one), the weight normalization modules start to raise errors due to some internal pytorch change.\n* numpy, scipy\n* imgaug (can be commented out if no augmentation is used)\n* matplotlib\n* seaborn (only required for plotting scripts)\n* scikit-learn (only required for plotting scripts)\n\n# Usage\n\nThe below commands are optimized for CelebA. Download that dataset and extract it to a directory, e.g. `/path/to/datasets/celeba`.\nThen split it into train and test:\n```bash\npython common/split_data.py --dataset folder --dataroot /path/to/datasets/celeba --running 1 --final 1\n```\n\n## R-iterative\n\nIn order to train R-iterative, saving checkpoints to e.g. `/path/to/checkpoints/exp01` (where `exp01` is here the name of the experiment).\n```bash\npython r_iterative/main.py --dataset folder --dataroot /path/to/datasets/celeba --crop_size 160 --image_size 80 --code_size 256 --norm weight --lr 0.00005 --niter 300000 --save_path /path/to/checkpoints/exp01\n```\nThat command trains at `80x80` resolution (`--image_size`) and learning rate `0.00005` (`--lr`) for `300,000` batches (`--niter`).\nAfter the initial run of G, it executes `3` iterations of R+G (`--r_iterations`) with lambda\\_R (strength of similarity constraint) of `0.9` (changeable via `--lambda_r`).\nIf you have to stop the experiment for some time, you can continue it by simply appending `--load_path /path/to/checkpoints/exp01` at the end of the command (do not delete `--save_path`).\nBy default, a checkpoint is saved every 2000 batches (`--save_interval`).\n\nIn order to sample images from G+R use\n```bash\npython r_iterative/sample_images.py --image_size 80 --code_size 256 --norm weight --r_iterations 3 --load_path_g /path/to/checkpoints/exp01/net_archive/last_gen.pt --load_path_r /path/to/checkpoints/exp01/net_archive/last_r.pt --save_path /path/to/outputs/exp01/final_sampled_images\n```\nThis loads the last G and R from experiment `exp01` and saves images to directory `/path/to/outputs/exp01/`.\nNote that the various arguments (e.g. `--image_size`, `--r_iterations`, ...) must match the ones that were used during the training.\n\n## G-LIS (G with LIS modules)\n\nThen run the training, saving checkpoints to e.g. `/path/to/checkpoints/exp01` (where `exp01` is here the name of the experiment).\n```bash\npython g_lis/main.py --dataset folder --dataroot /path/to/datasets/celeba --crop_size 160 --image_size 80 --code_size 256 --norm weight --lr 0.00002 --r_iterations 1 --niter 300000 --save_path /path/to/checkpoints/exp01\n```\nThat command trains at `80x80` resolution (`--image_size`) and learning rate `0.00002` (`--lr`) for `300,000` batches (`--niter`).\nIt uses `1` LIS module (`--r_iterations`) with lambda\\_R (strength of similarity constraint) of `0.9` (changeable via `--lambda_r`).\nIf you have to stop the experiment for some time, you can continue it by simply appending `--load_path /path/to/checkpoints/exp01` at the end of the command (do not delete `--save_path`).\nBy default, a checkpoint is saved every 5000 batches (`--save_interval`).\n\nIn order to sample images from the trained generator use\n```bash\npython g_lis/sample_images.py --image_size 80 --code_size 256 --norm weight --r_iterations 1 --load_path_g /path/to/checkpoints/exp01/net_archive/last_gen.pt --save_path /path/to/outputs/exp01/sampled_images/\n```\nThis loads the last generator from experiment `exp01` and saves images to directory `/path/to/outputs/exp01/sampled_images/`.\nNote that the various arguments (e.g. `--image_size`, `--r_iterations`, ...) must match the ones that were used during the training.\nThe main subdirectories in `sampled_images/` that you will care about are likely `sampled_images_rX`, where `X` starts at 0 (original noise vectors) and goes up to the value of `--r_iterations` (noise vectors of the last LIS module).\n\nIn order to create t-SNE plots of the noise vectors before/after LIS modules use\n```bash\npython g_lis/sample_tsne.py --image_size 80 --code_size 256 --norm weight --r_iterations 1 --load_path_g /path/to/checkpoints/exp01/net_archive/last_gen.pt --save_path /path/to/outputs/exp01/tsne/\n```\nThe plots will be saved to `/path/to/outputs/exp01/tsne/`. Note that this also caches the embedded vectors. Rerunning the command with the same `--save_path` will load the cached points instead of re-embedding them.\n\nIn order to create probability distribution plots for the noise vector components before/after LIS modules use\n```bash\npython g_lis/sample_density.py --image_size 80 --code_size 256 --norm weight --r_iterations 1 --load_path_g /path/to/checkpoints/exp01/net_archive/last_gen.pt --save_path /path/to/checkpoints/exp01/density/\n```\nThe plots will be saved to `/path/to/outputs/exp01/density/`.\n\nIn order to calculate Inception Scores (requires `tensorflow`) use\n```bash\npython g_lis/calculate_inception_scores.py --dataset folder --dataroot /path/to/datasets/celeba --crop_size 160 --image_size 80 --code_size 256 --norm weight --r_iterations 3 --inception_images 50000 --load_path /path/to/checkpoints/exp01\n```\nThis will produce a lot of output, run for ages and requires significant amounts of RAM. Files that contain inception scores will be written to `/path/to/checkpoints/exp01`.\n\nThe below commands are roughly the ones that were used to train on CIFAR-10, 102flowers and LSUN churches:\n```bash\npython g_lis/main.py --dataset cifar10 --dataroot /path/to/datasets/cifar-10 --crop_size 28 --image_size 32 --augment cifar10 --code_size 256 --norm weight --lr 0.00002 --r_iterations 3 --niter 1000000 --save_interval 25000 --vis_interval 1000 --nlayer 4 --nfeature 128 --lambda_r 0.9 --nb_cache_total 300000 --nb_cache_lists 5 --cache_p_drop 0.05 --save_path /path/to/checkpoints/exp-cifar #--load_path /path/to/checkpoints/exp-cifar\npython g_lis/main.py --dataset folder --dataroot /path/to/datasets/102flowers --crop_size 450 --image_size 80 --augment flowers102 --code_size 256 --norm weight --lr 0.00002 --r_iterations 3 --niter 500000 --save_interval 20000 --vis_interval 5000 --nb_cache_total 30000 --nb_cache_lists 3 --cache_p_drop 0.1 --save_path /path/to/checkpoints/exp-120flowers #--load_path /path/to/checkpoints/exp-120flowers\npython g_lis/main.py --dataset folder --dataroot /path/to/datasets/church_outdoor_train_80x80_square --crop_size 76 --image_size 80 --augment lsun_churches --code_size 256 --norm weight --lr 0.00002 --r_iterations 1 --niter 500000 --save_interval 20000 --vis_interval 5000 --save_path /save/to/checkpoints/exp-churches #--load_path /save/to/checkpoints/exp-churches\n```\nThe directory `church_outdoor_train_80x80_square` contained the LSUN churches images, which were zero-padded to be squared and then resized to 80x80.\n\n## R-separate\n\nR-separate is integrated into G-LIS. First, train a G-LIS module according to the commands listed above.\nThen train R-separate using\n```bash\npython g_lis/train_r.py --dataset folder --dataroot /path/to/datasets/celeba --crop_size 160 --image_size 80 --code_size 256 --norm weight --r_iterations 1 --lr 0.00005 --niter 2500 --spatial_dropout_r 0.1 --save_path_r /path/to/checkpoints/r-exp01 --load_path /path/to/checkpoints/exp01\n```\nThis trains R-separate for 2500 batches with spatial dropout of 10%.\nIt is trained based on a G-LIS model which's checkpoints are saved to `/path/to/checkpoints/exp01`.\nThe checkpoints of R-separate are saved to `/path/to/checkpoints/r-exp01`.\n\nThen sample images using\n```bash\npython g_lis/sample_images.py --image_size 80 --code_size 256 --norm weight --r_iterations 1 --spatial_dropout_r 0.1 --load_path_g /path/to/checkpoints/exp01/net_archive/last_gen.pt --load_path_r /path/to/checkpoints/r-exp01/net_archive/2500_r.pt --save_path /path/to/outputs/exp01/sampled_images_r/\n```\nThis saves images to `/path/to/outputs/exp01/sampled_images_r/`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleju%2Fgan-error-avoidance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleju%2Fgan-error-avoidance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleju%2Fgan-error-avoidance/lists"}