{"id":13563835,"url":"https://github.com/mit-han-lab/data-efficient-gans","last_synced_at":"2025-05-15T19:09:40.542Z","repository":{"id":37403302,"uuid":"272484725","full_name":"mit-han-lab/data-efficient-gans","owner":"mit-han-lab","description":"[NeurIPS 2020] Differentiable Augmentation for Data-Efficient GAN Training","archived":false,"fork":false,"pushed_at":"2024-09-24T17:46:07.000Z","size":35532,"stargazers_count":1305,"open_issues_count":28,"forks_count":176,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-12T20:45:06.348Z","etag":null,"topics":["data-efficient","gans","generative-adversarial-network","image-generation","neurips-2020","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2006.10738","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-han-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-15T16:07:24.000Z","updated_at":"2025-04-07T10:04:10.000Z","dependencies_parsed_at":"2024-08-01T13:19:46.656Z","dependency_job_id":"801c766c-c82a-4c50-8735-123609c842e7","html_url":"https://github.com/mit-han-lab/data-efficient-gans","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fdata-efficient-gans","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fdata-efficient-gans/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fdata-efficient-gans/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fdata-efficient-gans/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-han-lab","download_url":"https://codeload.github.com/mit-han-lab/data-efficient-gans/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254404357,"owners_count":22065641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-efficient","gans","generative-adversarial-network","image-generation","neurips-2020","pytorch","tensorflow"],"created_at":"2024-08-01T13:01:23.777Z","updated_at":"2025-05-15T19:09:40.521Z","avatar_url":"https://github.com/mit-han-lab.png","language":"Python","funding_links":[],"categories":["Python","Related Codebase"],"sub_categories":["Vision"],"readme":"# Data-Efficient GANs with DiffAugment\r\n\r\n### [project](https://data-efficient-gans.mit.edu/) | [paper](https://arxiv.org/pdf/2006.10738) | [datasets](https://www.dropbox.com/scl/fi/f641djlqsyen5emxnh22r/data-efficient-gans.zip?rlkey=xlmkf445fzwfno91i7954sp60\u0026st=0itvqzdc\u0026dl=0) | [video](https://www.youtube.com/watch?v=SsqcjS6SVM0) | [slides](https://data-efficient-gans.mit.edu/slides.pdf)\r\n\r\n\u003cimg src=\"imgs/interp.gif\"/\u003e\r\n\r\n*Generated using only 100 images of Obama, grumpy cats, pandas, the Bridge of Sighs, the Medici Fountain, the Temple of Heaven, without pre-training.*\r\n\r\n**[NEW!]** PyTorch training with [DiffAugment-stylegan2-pytorch](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2-pytorch) is now available!\r\n\r\n**[NEW!]** Our [Colab tutorial](https://colab.research.google.com/gist/zsyzzsoft/5fbb71b9bf9a3217576bebae5de46fc2/data-efficient-gans.ipynb) is released! [![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/zsyzzsoft/5fbb71b9bf9a3217576bebae5de46fc2/data-efficient-gans.ipynb)\r\n\r\n**[NEW!]** FFHQ training is supported! See the [DiffAugment-stylegan2](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2#FFHQ) README.\r\n\r\n**[NEW!]** Time to generate 100-shot interpolation videos with [generate_gif.py](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2/generate_gif.py)!\r\n\r\n**[NEW!]** Our [DiffAugment-biggan-imagenet](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-imagenet) repo (for TPU training) is released!\r\n\r\n**[NEW!]** Our [DiffAugment-biggan-cifar](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-cifar) PyTorch repo is released!\r\n\r\nThis repository contains our implementation of Differentiable Augmentation (DiffAugment) in both PyTorch and TensorFlow. It can be used to significantly improve the data efficiency for GAN training. We have provided [DiffAugment-stylegan2](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2) (TensorFlow) and [DiffAugment-stylegan2-pytorch](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2-pytorch), [DiffAugment-biggan-cifar](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-cifar) (PyTorch) for GPU training, and [DiffAugment-biggan-imagenet](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-imagenet) (TensorFlow) for TPU training.\r\n\r\n\u003cimg src=\"imgs/low-shot-comparison.jpg\" width=\"1000px\"/\u003e\r\n\r\n*Low-shot generation without pre-training. With DiffAugment, our model can generate high-fidelity images using only 100 Obama portraits, grumpy cats, or pandas from our collected 100-shot datasets, 160 cats or 389 dogs from the AnimalFace dataset at 256×256 resolution.*\r\n\r\n\u003cimg src=\"imgs/cifar10-results.jpg\" width=\"1000px\"/\u003e\r\n\r\n*Unconditional generation results on CIFAR-10. StyleGAN2’s performance drastically degrades given less training data. With DiffAugment, we are able to roughly match its FID and outperform its Inception Score (IS) using only **20%** training data.*\r\n\r\nDifferentiable Augmentation for Data-Efficient GAN Training\u003cbr\u003e\r\n[Shengyu Zhao](https://scholar.google.com/citations?user=gLCdw70AAAAJ), [Zhijian Liu](http://zhijianliu.com/), [Ji Lin](http://linji.me/), [Jun-Yan Zhu](https://www.cs.cmu.edu/~junyanz/), and [Song Han](https://songhan.mit.edu/)\u003cbr\u003e\r\nMIT, Tsinghua University, Adobe Research, CMU\u003cbr\u003e\r\n[arXiv](https://arxiv.org/pdf/2006.10738.pdf)\r\n\r\n\r\n## Overview\r\n\u003cimg src=\"imgs/method.jpg\" width=\"1000px\"/\u003e\r\n\r\n*Overview of DiffAugment for updating D (left) and G (right). DiffAugment applies the augmentation T to both the real sample x and the generated output G(z). When we update G, gradients need to be back-propagated through T (iii), which requires T to be differentiable w.r.t. the input.*\r\n\r\n## Training and Generation with 100 Images [![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/zsyzzsoft/5fbb71b9bf9a3217576bebae5de46fc2/data-efficient-gans.ipynb)\r\n\r\nTo generate an interpolation video using our pre-trained models:\r\n```bash\r\ncd DiffAugment-stylegan2\r\npython generate_gif.py -r mit-han-lab:DiffAugment-stylegan2-100-shot-obama.pkl -o obama.gif\r\n```\r\nor to train a new model:\r\n```bash\r\npython run_low_shot.py --dataset=100-shot-obama --num-gpus=4\r\n```\r\nYou may also try out `100-shot-grumpy_cat`, `100-shot-panda`, `100-shot-bridge_of_sighs`, `100-shot-medici_fountain`, `100-shot-temple_of_heaven`, `100-shot-wuzhen`, or the folder containing your own training images. Please refer to the [DiffAugment-stylegan2](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2#100-shot-generation) README for the dependencies and details.\r\n\r\n**[NEW!]** PyTorch training is now available:\r\n```bash\r\ncd DiffAugment-stylegan2-pytorch\r\npython train.py --outdir=training-runs --data=https://data-efficient-gans.mit.edu/datasets/100-shot-obama.zip --gpus=1\r\n```\r\n\r\n## DiffAugment for StyleGAN2\r\n\r\nTo run *StyleGAN2 + DiffAugment* for unconditional generation on the 100-shot datasets, CIFAR, FFHQ, or LSUN, please refer to the [DiffAugment-stylegan2](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2) README or [DiffAugment-stylegan2-pytorch](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-stylegan2-pytorch) for the PyTorch version.\r\n\r\n## DiffAugment for BigGAN\r\n\r\nPlease refer to the [DiffAugment-biggan-cifar](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-cifar) README to run *BigGAN + DiffAugment* for conditional generation on CIFAR (using GPUs), and the [DiffAugment-biggan-imagenet](https://github.com/mit-han-lab/data-efficient-gans/tree/master/DiffAugment-biggan-imagenet) README to run on ImageNet (using TPUs).\r\n\r\n## Using DiffAugment for Your Own Training\r\n\r\nTo help you use DiffAugment in your own codebase, we provide portable DiffAugment operations of both TensorFlow and PyTorch versions in [DiffAugment_tf.py](https://github.com/mit-han-lab/data-efficient-gans/blob/master/DiffAugment_tf.py) and [DiffAugment_pytorch.py](https://github.com/mit-han-lab/data-efficient-gans/blob/master/DiffAugment_pytorch.py). Generally, DiffAugment can be easily adopted in any model by substituting every *D(x)* with *D(T(x))*, where *x* can be real images or fake images, *D* is the discriminator, and *T* is the DiffAugment operation. For example,\r\n\r\n```python\r\nfrom DiffAugment_pytorch import DiffAugment\r\n# from DiffAugment_tf import DiffAugment\r\npolicy = 'color,translation,cutout' # If your dataset is as small as ours (e.g.,\r\n# hundreds of images), we recommend using the strongest Color + Translation + Cutout.\r\n# For large datasets, try using a subset of transformations in ['color', 'translation', 'cutout'].\r\n# Welcome to discover more DiffAugment transformations!\r\n\r\n...\r\n# Training loop: update D\r\nreals = sample_real_images() # a batch of real images\r\nz = sample_latent_vectors()\r\nfakes = Generator(z) # a batch of fake images\r\nreal_scores = Discriminator(DiffAugment(reals, policy=policy))\r\nfake_scores = Discriminator(DiffAugment(fakes, policy=policy))\r\n# Calculating D's loss based on real_scores and fake_scores...\r\n...\r\n\r\n...\r\n# Training loop: update G\r\nz = sample_latent_vectors()\r\nfakes = Generator(z) # a batch of fake images\r\nfake_scores = Discriminator(DiffAugment(fakes, policy=policy))\r\n# Calculating G's loss based on fake_scores...\r\n...\r\n```\r\n\r\nWe have implemented Color, Translation, and Cutout DiffAugment as visualized below:\r\n\r\n\u003cimg src=\"imgs/augmentations.jpg\" width=\"800px\"/\u003e\r\n\r\n## Citation\r\n\r\nIf you find this code helpful, please cite our paper:\r\n```\r\n@inproceedings{zhao2020diffaugment,\r\n  title={Differentiable Augmentation for Data-Efficient GAN Training},\r\n  author={Zhao, Shengyu and Liu, Zhijian and Lin, Ji and Zhu, Jun-Yan and Han, Song},\r\n  booktitle={Conference on Neural Information Processing Systems (NeurIPS)},\r\n  year={2020}\r\n}\r\n```\r\n\r\n## Acknowledgements\r\n\r\nWe thank NSF Career Award #1943349, MIT-IBM Watson AI Lab, Google, Adobe, and Sony for supporting this research.  Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC). We thank William S. Peebles and Yijun Li for helpful comments.\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fdata-efficient-gans","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-han-lab%2Fdata-efficient-gans","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fdata-efficient-gans/lists"}