{"id":13563327,"url":"https://github.com/VinAIResearch/CREPS","last_synced_at":"2025-04-03T19:32:49.366Z","repository":{"id":165910093,"uuid":"617928586","full_name":"VinAIResearch/CREPS","owner":"VinAIResearch","description":"Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis (CVPR 2023)","archived":false,"fork":false,"pushed_at":"2025-01-04T06:43:27.000Z","size":56252,"stargazers_count":31,"open_issues_count":0,"forks_count":1,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-01-04T07:32:49.407Z","etag":null,"topics":["generative-adversarial-network","implicit-neural-representation","scale-invariance","stylegan3"],"latest_commit_sha":null,"homepage":"https://vinairesearch.github.io/CREPS/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VinAIResearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-23T12:05:55.000Z","updated_at":"2024-12-15T12:52:19.000Z","dependencies_parsed_at":"2024-01-14T03:48:43.100Z","dependency_job_id":"3f1b9a92-9641-4359-84ab-986121b21c55","html_url":"https://github.com/VinAIResearch/CREPS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VinAIResearch%2FCREPS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VinAIResearch%2FCREPS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VinAIResearch%2FCREPS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VinAIResearch%2FCREPS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VinAIResearch","download_url":"https://codeload.github.com/VinAIResearch/CREPS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247065471,"owners_count":20877785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generative-adversarial-network","implicit-neural-representation","scale-invariance","stylegan3"],"created_at":"2024-08-01T13:01:17.898Z","updated_at":"2025-04-03T19:32:49.359Z","avatar_url":"https://github.com/VinAIResearch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"##### Table of contents\n1. [Requirements](#requirements)\n2. [Getting Started](#getting-started)\n3. [Using networks from Python](#using-networks-from-python)\n4. [Preparing datasets](#preparing-datasets)\n5. [Training](#training)\n6. [Quality Metrics](#quality-metrics)\n7. [Contacts](#Contacts)\n\n# Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis\n\u003ca href=\"https://thuanz123.github.io/creps\"\u003e\u003cimg src=\"https://img.shields.io/badge/WEBSITE-Visit%20project%20page-blue?style=for-the-badge\"\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2303.14157\"\u003e\u003cimg src=\"https://img.shields.io/badge/arxiv-2303.14157-red?style=for-the-badge\"\u003e\u003c/a\u003e\n\n[Thuan Hoang Nguyen](https://thuanz123.github.io/),\n[Thanh Van Le](https://luvata.github.io/),\n[Anh Tran](https://scholar.google.com/citations?user=FYZ5ODQAAAAJ)\u003cbr\u003e\nVinAI Research, Vietnam\n\n\u003e **Abstract:** \nAny-scale image synthesis offers an efficient and scalable solution to synthesize photo-realistic images at any scale, even going beyond 2K resolution. However, existing GAN-based solutions depend excessively on convolutions and a hierarchical architecture, which introduce inconsistency and the \"texture sticking\" issue when scaling the output resolution. From another perspective, INR-based generators are scale-equivariant by design, but their huge memory footprint and slow inference hinder these networks from being adopted in large-scale or real-time systems. In this work, we propose **C**olumn-**R**ow **E**ntangled **P**ixel **S**ynthesis (**CREPS**), a new generative model that is both efficient and scale-equivariant without using any spatial convolutions or coarse-to-fine design. To save memory footprint and make the system scalable, we employ a novel bi-line representation that decomposes layer-wise feature maps into separate \"thick\" column and row encodings. Experiments on various datasets, including FFHQ, LSUN-Church, MetFaces, and Flickr-Scenery, confirm CREPS' ability to synthesize scale-consistent and alias-free images at any arbitrary resolution with proper training and inference speed.\n\n![teaser.png](./assets/teaser.png)\n\nDetails of the model architecture and experimental results can be found in [our following paper](https://arxiv.org/abs/2303.14157).\n```bibtex\n@inproceedings{thuan2023creps,\n  title={Efficient Scale-Invariant Generator with Column-Row Entangled Pixel Synthesis},\n  author={Thuan Hoang Nguyen, Thanh Van Le, Anh Tran},\n  year={2023},\n  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n}\n```\n**Please CITE** our paper whenever our model implementation is used to help produce published results or incorporated into other software.\n\n## Additional material\n\n- [CREPS pre-trained models](https://drive.google.com/drive/folders/19ntlAXD7MwYqN7OPVuBcZgaC_yFCwpk7?usp=share_link)\n- [StyleGAN3 pre-trained models](https://ngc.nvidia.com/catalog/models/nvidia:research:stylegan3) compatible with this codebase\n  \u003e \u003csub\u003eAccess individual networks via `https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/\u003cMODEL\u003e`, where `\u003cMODEL\u003e` is one of:\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-t-ffhq-1024x1024.pkl`, `stylegan3-t-ffhqu-1024x1024.pkl`, `stylegan3-t-ffhqu-256x256.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-r-ffhq-1024x1024.pkl`, `stylegan3-r-ffhqu-1024x1024.pkl`, `stylegan3-r-ffhqu-256x256.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-t-metfaces-1024x1024.pkl`, `stylegan3-t-metfacesu-1024x1024.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-r-metfaces-1024x1024.pkl`, `stylegan3-r-metfacesu-1024x1024.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-t-afhqv2-512x512.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan3-r-afhqv2-512x512.pkl`\u003c/sub\u003e\u003cbr\u003e\n- [StyleGAN2 pre-trained models](https://ngc.nvidia.com/catalog/models/nvidia:research:stylegan2) compatible with this codebase\n  \u003e \u003csub\u003eAccess individual networks via `https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan2/versions/1/files/\u003cMODEL\u003e`, where `\u003cMODEL\u003e` is one of:\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-ffhq-1024x1024.pkl`, `stylegan2-ffhq-512x512.pkl`, `stylegan2-ffhq-256x256.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-ffhqu-1024x1024.pkl`, `stylegan2-ffhqu-256x256.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-metfaces-1024x1024.pkl`, `stylegan2-metfacesu-1024x1024.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-afhqv2-512x512.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-afhqcat-512x512.pkl`, `stylegan2-afhqdog-512x512.pkl`, `stylegan2-afhqwild-512x512.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-brecahad-512x512.pkl`, `stylegan2-cifar10-32x32.pkl`\u003c/sub\u003e\u003cbr\u003e\n  \u003e \u003csub\u003e`stylegan2-celebahq-256x256.pkl`, `stylegan2-lsundog-256x256.pkl`\u003c/sub\u003e\u003cbr\u003e\n\n## Requirements\n\n* Linux and Windows are supported, but we recommend Linux for performance and compatibility reasons.\n* 1\u0026ndash;8 high-end NVIDIA GPUs with at least 12 GB of memory. We have done all testing and development using Tesla V100 and A100 GPUs.\n* 64-bit Python 3.8 and PyTorch 1.9.0 (or later). See https://pytorch.org for PyTorch install instructions.\n* CUDA toolkit 11.1 or later.  (Why is a separate CUDA toolkit installation required?  See [Troubleshooting](./assets/troubleshooting.md#why-is-cuda-toolkit-installation-necessary)).\n* GCC 7 or later (Linux) or Visual Studio (Windows) compilers.  Recommended GCC version depends on CUDA version, see for example [CUDA 11.4 system requirements](https://docs.nvidia.com/cuda/archive/11.4.1/cuda-installation-guide-linux/index.html#system-requirements).\n* Python libraries: see [environment.yml](./environment.yml) for exact library dependencies.  You can use the following commands with Miniconda3 to create and activate your CREPS Python environment:\n  - `conda env create -f environment.yml`\n  - `conda activate creps`\n* Docker users:\n  - Ensure you have correctly installed the [NVIDIA container runtime](https://docs.docker.com/config/containers/resource_constraints/#gpu).\n  - Use the [provided Dockerfile](./Dockerfile) to build an image with the required library dependencies.\n\nThe code relies heavily on custom PyTorch extensions that are compiled on the fly using NVCC. On Windows, the compilation requires Microsoft Visual Studio. We recommend installing [Visual Studio Community Edition](https://visualstudio.microsoft.com/vs/) and adding it into `PATH` using `\"C:\\Program Files (x86)\\Microsoft Visual Studio\\\u003cVERSION\u003e\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat\"`.\n\nSee [Troubleshooting](./assets/troubleshooting.md) for help on common installation and run-time problems.\n\n## Getting started\n\nPre-trained networks are stored as `*.pkl` files that can be referenced using local filenames or URLs:\n\n```.bash\n# Generate an image using pre-trained FFHQ model.\npython gen_images.py --outdir=out --trunc=1 --seeds=2 \\\n    --network=creps-ffhq-512x512.pkl --scale=2\n\n# Render a 4x2 grid of interpolations for seeds 0 through 31.\npython gen_video.py --output=lerp.mp4 --trunc=1 --seeds=0-31 --grid=4x2 \\\n    --network=creps-ffhq-512x512.pkl\n```\n\nOutputs from the above commands are placed under `out/*.png`, controlled by `--outdir`. Downloaded network pickles are cached under `$HOME/.cache/dnnlib`, which can be overridden by setting the `DNNLIB_CACHE_DIR` environment variable. The default PyTorch extension build directory is `$HOME/.cache/torch_extensions`, which can be overridden by setting `TORCH_EXTENSIONS_DIR`.\n\n**Docker**: You can run the above curated image example using Docker as follows:\n\n```.bash\n# Build the creps:latest image\ndocker build --tag creps .\n\n# Run the gen_images.py script using Docker:\ndocker run --gpus all -it --rm --user $(id -u):$(id -g) \\\n    -v `pwd`:/scratch --workdir /scratch -e HOME=/scratch \\\n    creps \\\n    python gen_images.py --outdir=out --trunc=1 --seeds=2 \\\n         --network=creps-ffhq-512x512.pkl\n```\n\nNote: The Docker image requires NVIDIA driver release `r470` or later.\n\nThe `docker run` invocation may look daunting, so let's unpack its contents here:\n\n- `--gpus all -it --rm --user $(id -u):$(id -g)`: with all GPUs enabled, run an interactive session with current user's UID/GID to avoid Docker writing files as root.\n- ``-v `pwd`:/scratch --workdir /scratch``: mount current running dir (e.g., the top of this git repo on your host machine) to `/scratch` in the container and use that as the current working dir.\n- `-e HOME=/scratch`: let PyTorch and CREPS code know where to cache temporary files such as pre-trained models and custom PyTorch extension build results. Note: if you want more fine-grained control, you can instead set `TORCH_EXTENSIONS_DIR` (for custom extensions build dir) and `DNNLIB_CACHE_DIR` (for pre-trained model download cache). You want these cache dirs to reside on persistent volumes so that their contents are retained across multiple `docker run` invocations.\n\n## Using networks from Python\n\nYou can use pre-trained networks in your own Python code as follows:\n\n```python\nwith open('ffhq.pkl', 'rb') as f:\n    G = pickle.load(f)['G_ema'].cuda()  # torch.nn.Module\nz = torch.randn([1, G.z_dim]).cuda()    # latent codes\nc = None                                # class labels (not used in this example)\nimg = G(z, c)                           # NCHW, float32, dynamic range [-1, +1], no truncation\n```\n\nThe above code requires `torch_utils` and `dnnlib` to be accessible via `PYTHONPATH`. It does not need source code for the networks themselves \u0026mdash; their class definitions are loaded from the pickle via `torch_utils.persistence`.\n\nThe pickle contains three networks. `'G'` and `'D'` are instantaneous snapshots taken during training, and `'G_ema'` represents a moving average of the generator weights over several training steps. The networks are regular instances of `torch.nn.Module`, with all of their parameters and buffers placed on the CPU at import and gradient computation disabled by default.\n\nThe generator consists of two submodules, `G.mapping` and `G.synthesis`, that can be executed separately. They also support various additional options:\n\n```python\nw = G.mapping(z, c, truncation_psi=0.5, truncation_cutoff=8)\nimg = G.synthesis(w, noise_mode='const', force_fp32=True)\n```\n\nPlease refer to [`gen_images.py`](./gen_images.py) for complete code example.\n\n## Preparing datasets\n\nDatasets are stored as uncompressed ZIP archives containing uncompressed PNG files and a metadata file `dataset.json` for labels. Custom datasets can be created from a folder containing images; see [`python dataset_tool.py --help`](./assets/dataset-tool-help.txt) for more information. Alternatively, the folder can also be used directly as a dataset, without running it through `dataset_tool.py` first, but doing so may lead to suboptimal performance.\n\n**FFHQ**: Download the [Flickr-Faces-HQ dataset](https://github.com/NVlabs/ffhq-dataset) as 1024x1024 images and create a zip archive using `dataset_tool.py`:\n\n```.bash\n# Original 1024x1024 resolution.\npython dataset_tool.py --source=/tmp/images1024x1024 --dest=~/datasets/ffhq-1024x1024.zip\n\n# Scaled down 256x256 resolution.\npython dataset_tool.py --source=/tmp/images1024x1024 --dest=~/datasets/ffhq-256x256.zip \\\n    --resolution=256x256\n```\n\nSee the [FFHQ README](https://github.com/NVlabs/ffhq-dataset) for information on how to obtain the unaligned FFHQ dataset images. Use the same steps as above to create a ZIP archive for training and validation.\n\n**MetFaces**: Download the [MetFaces dataset](https://github.com/NVlabs/metfaces-dataset) and create a ZIP archive:\n\n```.bash\npython dataset_tool.py --source=~/downloads/metfaces/images --dest=~/datasets/metfaces-1024x1024.zip\n```\n\nSee the [MetFaces README](https://github.com/NVlabs/metfaces-dataset) for information on how to obtain the unaligned MetFaces dataset images. Use the same steps as above to create a ZIP archive for training and validation.\n\n**AFHQv2**: Download the [AFHQv2 dataset](https://github.com/clovaai/stargan-v2/blob/master/README.md#animal-faces-hq-dataset-afhq) and create a ZIP archive:\n\n```.bash\npython dataset_tool.py --source=~/downloads/afhqv2 --dest=~/datasets/afhqv2-512x512.zip\n```\n\nNote that the above command creates a single combined dataset using all images of all three classes (cats, dogs, and wild animals), matching the setup used in the CREPS paper. Alternatively, you can also create a separate dataset for each class:\n\n```.bash\npython dataset_tool.py --source=~/downloads/afhqv2/train/cat --dest=~/datasets/afhqv2cat-512x512.zip\npython dataset_tool.py --source=~/downloads/afhqv2/train/dog --dest=~/datasets/afhqv2dog-512x512.zip\npython dataset_tool.py --source=~/downloads/afhqv2/train/wild --dest=~/datasets/afhqv2wild-512x512.zip\n```\n\n## Training\n\nYou can train new networks using `train.py`. For example:\n\n```.bash\n# Train CREPS for FFHQ using 8 GPUs.\npython train.py --outdir=~/training-runs --cfg=creps --data=~/datasets/ffhq-512x512.zip \\\n    --gpus=8 --batch=32 --gamma=8.2 --mirror=1\n\n# Fine-tune CREPS for MetFaces using 1 GPU, starting from the pre-trained FFHQ pickle.\npython train.py --outdir=~/training-runs --cfg=creps --data=~/datasets/metfaces-1024x1024.zip \\\n    --gpus=8 --batch=32 --gamma=6.6 --mirror=1 --kimg=5000 --snap=5 \\\n    --resume=creps-ffhq-512x512.pkl\n```\n\nBesides CREPS, our code also supports StyleGAN3 and StyleGAN2. Note that the result quality and training time depend heavily on the exact set of options. The most important ones (`--gpus`, `--batch`, and `--gamma`) must be specified explicitly, and they should be selected with care. See [`python train.py --help`](./assets/train-help.txt) for the full list of options and [Training configurations](./assets/configs.md) for general guidelines \u0026amp; recommendations, along with the expected training speed \u0026amp; memory usage in different scenarios.\n\nThe results of each training run are saved to a newly created directory, for example `~/training-runs/00000-creps-afhqv2-512x512-gpus8-batch32-gamma8.2`. The training loop exports network pickles (`network-snapshot-\u003cKIMG\u003e.pkl`) and random image grids (`fakes\u003cKIMG\u003e.png`) at regular intervals (controlled by `--snap`). For each exported pickle, it evaluates FID (controlled by `--metrics`) and logs the result in `metric-fid50k_full.jsonl`. It also records various statistics in `training_stats.jsonl`, as well as `*.tfevents` if TensorBoard is installed.\n\n## Quality metrics\n\nBy default, `train.py` automatically computes FID for each network pickle exported during training. We recommend inspecting `metric-fid50k_full.jsonl` (or TensorBoard) at regular intervals to monitor the training progress. When desired, the automatic computation can be disabled with `--metrics=none` to speed up the training slightly.\n\nAdditional quality metrics can also be computed after the training:\n\n```.bash\n# Previous training run: look up options automatically, save result to JSONL file.\npython calc_metrics.py --metrics=fid50k_full \\\n    --network=~/training-runs/creps-mydataset/network-snapshot-000000.pkl\n\n# Pre-trained network pickle: specify dataset explicitly, print result to stdout.\npython calc_metrics.py --metrics=fid50k_full --data=~/datasets/ffhq-1024x1024.zip --mirror=1 \\\n    --network=creps-ffhq-512x512.pkl --scale=2\n```\n\nThe first example looks up the training configuration and performs the same operation as if `--metrics=fid50k_full` had been specified during training. The second example downloads a pre-trained network pickle, in which case the values of `--data` and `--mirror` must be specified explicitly.\n\nNote that the metrics can be quite expensive to compute (up to 1h), and many of them have an additional one-off cost for each new dataset (up to 30min). Also note that the evaluation is done using a different random seed each time, so the results will vary if the same metric is computed multiple times.\n\nRecommended metrics:\n* `fid50k_full`: Fr\u0026eacute;chet inception distance\u003csup\u003e[1]\u003c/sup\u003e against the full dataset.\n* `kid50k_full`: Kernel inception distance\u003csup\u003e[2]\u003c/sup\u003e against the full dataset.\n* `pr50k3_full`: Precision and recall\u003csup\u003e[3]\u003c/sup\u003e againt the full dataset.\n* `ppl2_wend`: Perceptual path length\u003csup\u003e[4]\u003c/sup\u003e in W, endpoints, full image.\n* `eqt50k_int`: Equivariance\u003csup\u003e[5]\u003c/sup\u003e w.r.t. integer translation (EQ-T).\n* `eqt50k_frac`: Equivariance w.r.t. fractional translation (EQ-T\u003csub\u003efrac\u003c/sub\u003e).\n* `eqr50k`: Equivariance w.r.t. rotation (EQ-R).\n\nLegacy metrics:\n* `fid50k`: Fr\u0026eacute;chet inception distance against 50k real images.\n* `kid50k`: Kernel inception distance against 50k real images.\n* `pr50k3`: Precision and recall against 50k real images.\n* `is50k`: Inception score\u003csup\u003e[6]\u003c/sup\u003e for CIFAR-10.\n\nReferences:\n1. [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://arxiv.org/abs/1706.08500), Heusel et al. 2017\n2. [Demystifying MMD GANs](https://arxiv.org/abs/1801.01401), Bi\u0026nacute;kowski et al. 2018\n3. [Improved Precision and Recall Metric for Assessing Generative Models](https://arxiv.org/abs/1904.06991), Kynk\u0026auml;\u0026auml;nniemi et al. 2019\n4. [A Style-Based Generator Architecture for Generative Adversarial Networks](https://arxiv.org/abs/1812.04948), Karras et al. 2018\n5. [Alias-Free Generative Adversarial Networks](https://nvlabs.github.io/stylegan3), Karras et al. 2021\n6. [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498), Salimans et al. 2016\n\n## Acknowledgments\nThis repo is built upon [StyleGAN3](https://github.com/NVlabs/stylegan3) and we thank the authors for their great works and efforts to release source code. Furthermore, a special \"thank you\" to [CIPS](https://github.com/advimman/CIPS) and [AnyresGAN](https://github.com/chail/anyres-gan), which heavily inspire our work.\n\n## Contacts\nIf you have any questions, please drop an email to _v.thuannh5@vinai.io_ or open an issue in this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVinAIResearch%2FCREPS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVinAIResearch%2FCREPS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVinAIResearch%2FCREPS/lists"}