{"id":2040479,"url":"https://github.com/autonomousvision/stylegan-t","last_synced_at":"2025-04-12T18:44:43.797Z","repository":{"id":65455314,"uuid":"591370247","full_name":"autonomousvision/stylegan-t","owner":"autonomousvision","description":"[ICML'23] StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis","archived":false,"fork":false,"pushed_at":"2023-04-07T08:58:50.000Z","size":1291,"stargazers_count":1174,"open_issues_count":13,"forks_count":56,"subscribers_count":112,"default_branch":"main","last_synced_at":"2025-04-03T21:11:34.779Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-20T15:37:36.000Z","updated_at":"2025-04-03T19:53:47.000Z","dependencies_parsed_at":"2024-12-02T17:32:07.415Z","dependency_job_id":null,"html_url":"https://github.com/autonomousvision/stylegan-t","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fstylegan-t","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fstylegan-t/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fstylegan-t/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fstylegan-t/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/stylegan-t/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248617200,"owners_count":21134190,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-01-21T04:00:01.340Z","updated_at":"2025-04-12T18:44:43.758Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cimg src=\"docs/banner.png\"\u003e\n\n#### [[Project]](https://sites.google.com/view/stylegan-t/)    [[PDF]](https://arxiv.org/abs/2301.09515)    [[Video]](https://www.youtube.com/watch?v=MMj8OTOUIok) \nThis repository contains the **training code** for our paper \"StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis\". **We do not provide pretrained checkpoints.**\n\nby [Axel Sauer](https://axelsauer.com/), [Tero Karras](https://research.nvidia.com/person/tero-karras), [Samuli Laine](https://research.nvidia.com/person/samuli-laine), [Andreas Geiger](https://www.cvlibs.net/), [Timo Aila](https://research.nvidia.com/person/timo-aila)\n\n## Requirements ##\n- Use the following commands with Miniconda3 to create and activate your environment:\n  ```\n  conda create --name sgt python=3.9\n  conda activate sgt\n  conda install pytorch=1.9.1 torchvision==0.10.1 pytorch-cuda=11.6 -c pytorch -c nvidia\n  pip install -r requirements.txt\n  ```\n- GCC 7 or later compilers. The recommended GCC version depends on your CUDA version; see for example, CUDA 11.4 system requirements.\n- If you run into problems when setting up the custom CUDA kernels, we refer to the [Troubleshooting docs](https://github.com/NVlabs/stylegan3/blob/main/docs/troubleshooting.md#why-is-cuda-toolkit-installation-necessary) of the StyleGAN3 repo.\n\n\n## Data Preparation ##\n\nStyleGAN-T can be trained on unconditional and conditional datasets. For small-scale experiments, we recommend zip datasets. When training on datasets with more than 1 million images, we recommend using webdatasets.\n\n### Zip Dataset\nZip-Datasets are stored in the same format as in the previous iterations of [StyleGAN](https://github.com/NVlabs/stylegan3): uncompressed ZIP archives containing uncompressed PNG files and a metadata file `dataset.json` for labels. Custom datasets can be created from a folder containing images.\n\n\n**CIFAR-10:** Download the [CIFAR-10 python version](https://www.cs.toronto.edu/~kriz/cifar.html) and convert to ZIP archive:\n\n```.bash\npython dataset_tool.py --source downloads/cifar10/cifar-10-python.tar.gz \\\n    --dest data/cifar10-32x32.zip\n```\n\n**FFHQ:** Download the [Flickr-Faces-HQ dataset](https://github.com/NVlabs/ffhq-dataset) as 1024x1024 images and convert to ZIP archive at the same resolution:\n\n```.bash\npython dataset_tool.py --source downloads/ffhq/images1024x1024 \\\n    --dest data/ffhq1024.zip --resolution 1024x1024\n```\n\n**COCO validation set:** The COCO validation set is used for tracking zero-shot FID and CLIP score. First, download the [COCO meta data](https://drive.google.com/file/d/1Xbg36mTJGG68RI_YgSPwhAE2msxgGfaC/view?usp=sharing). Then, run\n```.bash\npython dataset_tool.py --source downloads/captions_val2014.json \\\n  --dest data/coco_val256.zip --resolution 256x256 --transform center-crop\n```\n\nIt is recommend to prepare the dataset zip at the highest possible resolution, e.g. for FFHQ, the zip should contain images with 1024x1024 pixels. When training lower-resolution models, the training script can downsample the images on the fly. \n\n### WebDataset\nFor preparing webdatasets, we used the excellent [img2dataset](https://github.com/rom1504/img2dataset) tool. Documentation for downloading different datasets can be found [here](https://github.com/rom1504/img2dataset/tree/main/dataset_examples). For our experiments, we used data from the following sources: [CC3M](https://ai.google.com/research/ConceptualCaptions/download), [CC12M](https://github.com/google-research-datasets/conceptual-12m), [YFFC100m](https://huggingface.co/datasets/dalle-mini/YFCC100M_OpenAI_subset), [Redcaps](https://huggingface.co/datasets/red_caps), [LAION-aesthetic-6plus](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6plus).\n\nThe joint dataset should have the following structure\n```joint_dataset/\njoint_dataset/cc3m/0000.tar\njoint_dataset/cc3m/0001.tar\n...\njoint_dataset/laion_6plus/0000.tar\njoint_dataset/laion_6plus/0001.tar\n...\n```\n## Training ##\n\nTraining StyleGAN-T with full capacity on MYDATASET.zip at a resolution of 64x64 pixels:\n\n```\npython -m torch.distributed.run --standalone --nproc_per_node 1 train.py \\\n  --outdir ./training-runs/ --cfg full --data ./data/MYDATASET.zip \\\n  --img-resolution 64 --batch 128 --batch-gpu 8 --kimg 25000 --metrics fid50k_full\n```\n\n- The above commands can be parallelized across multiple GPUs by adjusting ```--nproc_per_node```.\n- ```--batch``` specifies the overall batch size, ```--batch-gpu``` specifies the batch size per GPU. Be aware that ```--batch-gpu``` is also a hyperparameter as the discriminator uses (local) BatchNorm; We generally recommend ```--batch-gpu``` of 4 or 8.\nThe training loop will automatically accumulate gradients if you use fewer GPUs until the overall batch size is reached. \n- Samples and metrics are saved in ```outdir```. You can inspect ```METRIC_NAME.json``` or run tensorboard in ```training-runs/``` to monitor the training progress.\n- The generator will be conditional if the dataset contains text labels; otherwise, it will be unconditional.\n- For a webdataset comprised of different subsets, the data path should point to the joint parent directory: ```--data path/to/joint_dataset/```\n\nTo use the same configuration we used for our ablation study, use ```--cfg lite```. If you want direct control over network parameters, use a custom config. E.g., a smaller models which has 1 residual block, a capacity multiplier of 16384 and a maximum channel count of 256, run\n\n```\npython -m torch.distributed.run --standalone --nproc_per_node 1 train.py \\\n  --outdir ./training-runs/ --data ./data/MYDATASET.zip \\\n  --img-resolution 64 --batch 128 --batch-gpu 8 --kimg 25000 --metrics fid50k_full \\\n  --cfg custom --cbase 16384 --cmax 256 --res-blocks 1\n```\n\nFor a description of all input arguments, run ```python train.py --help```\n\n###  Starting from pretrained checkpoints\n\nIf you want to use a previously trained model, you can start from a checkpoint by specifying its path adding ```--resume PATH_TO_NETWORK_PKL```. If you want to continue training from where you left off in a previous run, you can also specify the number of images processed in that run using ```--resume-kimg XXX```, where XXX is that number.\n\n### Training modes\n\nBy default, all layers of the generator are trained and the CLIP text encoder is frozen. If you want to train only the text encoder, provide ```--train-mode text-encoder```. \n\nIf you want to do progressive growing, first train a model at 64x64 pixels. Then provide the path to this pretrained network via ```-resume```, the new target resolution via ```--img-resolution```  and use ```--train-mode freeze64``` to freeze the blocks of the 64x64 model and only train the high resolution layers. For example:\n\n```\npython -m torch.distributed.run --standalone --nproc_per_node 1 train.py \\\n  --outdir ./training-runs/ --data ./data/MYDATASET.zip \\\n  --img-resolution 512 --batch 128 --batch-gpu 4 --kimg 25000 \\\n  --cfg lite --resume PATH_TO_NETWORK_64 --train-mode freeze64\n```\n\n## Generating Samples ##\nTo generate samples with a given network, run\n```\npython gen_images.py --network PATH_TO_NETWORK_PKL \\\n --prompt 'A painting of a fox in the style of starry night.' \\\n --truncation 1 --outdir out --seeds 0-29\n```\n\nFor a description of all input arguments, run ```python gen_images.py --help```\n\n## Quality Metrics ##\nTo calculate metrics for a specific network snapshot, run\n\n```\npython calc_metrics.py --metrics METRIC_NAME --network PATH_TO_NETWORK_PKL\n```\n\nMetric computation is only supported on zip datasets, not webdatasets. The zero-shot COCO metrics expect a ```coco_val256.zip``` to be present in the same folder as the training dataset. Alternatively, one can explicitely set an environment variable as follows: ```export COCOPATH=path/to/coco_val256.zip```.\n\nTo see the available metrics, run ```python calc_metrics.py --help```\n\n\n## License ##\n\nCopyright \u0026copy; 2023, NVIDIA Corporation. All rights reserved.\n\nThis work is made available under the [Nvidia Source Code License](https://nvlabs.github.io/stylegan2-ada-pytorch/license.html).\n\nExcempt are the files ```training/diffaug.py``` and ```networks/vit_utils.py``` which are partially or fully based on third party github repositories. These two files are copyright their respective authors and under their respective licenses; we include the original license and link to the source at the beginning of the files.\n\n## Development ##\n\nThis is a research reference implementation and is treated as a one-time code drop. As such, we do not accept outside code contributions in the form of pull requests.\n\n## Citation ##\n\n```bibtex\n@InProceedings{Sauer2023ARXIV,\n  author    = {Axel Sauer and Tero Karras and Samuli Laine and Andreas Geiger and Timo Aila},\n  title     = {{StyleGAN-T}: Unlocking the Power of {GANs} for Fast Large-Scale Text-to-Image Synthesis},\n  journal   = {{arXiv.org}},\n  volume    = {abs/2301.09515},\n  year      = {2023},\n  url       = {https://arxiv.org/abs/2301.09515},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fstylegan-t","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fstylegan-t","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fstylegan-t/lists"}