{"id":28276232,"url":"https://github.com/lmxyy/sige","last_synced_at":"2025-06-16T16:31:22.809Z","repository":{"id":62607764,"uuid":"559403072","full_name":"lmxyy/sige","owner":"lmxyy","description":"[NeurIPS 2022, T-PAMI 2023] Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models","archived":false,"fork":false,"pushed_at":"2024-03-18T03:39:31.000Z","size":45081,"stargazers_count":265,"open_issues_count":2,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-21T04:19:00.320Z","etag":null,"topics":["acceleration","conditional-gan","conditional-gans","ddim","ddpm","diffusion-models","gans","gaugan","image-editing","progressive-distillation","sparse","sparse-convolution"],"latest_commit_sha":null,"homepage":"https://www.cs.cmu.edu/~sige/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmxyy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-30T02:03:26.000Z","updated_at":"2025-03-25T00:29:37.000Z","dependencies_parsed_at":"2023-02-17T09:01:08.462Z","dependency_job_id":"d2c19e79-a0de-411d-8150-b9ab26d89dd6","html_url":"https://github.com/lmxyy/sige","commit_stats":{"total_commits":86,"total_committers":3,"mean_commits":"28.666666666666668","dds":0.08139534883720934,"last_synced_commit":"83ee1d330ea90068726da96c64cb5dd17640dadb"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lmxyy/sige","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmxyy%2Fsige","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmxyy%2Fsige/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmxyy%2Fsige/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmxyy%2Fsige/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmxyy","download_url":"https://codeload.github.com/lmxyy/sige/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmxyy%2Fsige/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260197107,"owners_count":22973091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acceleration","conditional-gan","conditional-gans","ddim","ddpm","diffusion-models","gans","gaugan","image-editing","progressive-distillation","sparse","sparse-convolution"],"created_at":"2025-05-21T04:18:58.709Z","updated_at":"2025-06-16T16:31:22.795Z","avatar_url":"https://github.com/lmxyy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spatially Incremental Generative Engine (SIGE)\n\n### [Paper](https://arxiv.org/abs/2211.02048) | [Project](https://www.cs.cmu.edu/~sige/) | [Slides](https://www.cs.cmu.edu/~sige/resources/slides.key) | [YouTube](https://youtu.be/rDPotGoPPkQ)\n\n**[NEW!]** SIGE is accepted by T-PAMI!\n\n**[NEW!]** SIGE supports [Stable Diffusion](./stable_diffusion) and Mac MPS backend! We also release the codes of [an interactive demo for DDPM](diffusion_demo) on M1 Macbook Pro!\n\n**[NEW!]** SIGE is accepted by NeurIPS 2022! Our code and benchmark datasets are publicly available!\n\n![teaser](https://github.com/lmxyy/sige/raw/main/assets/teaser.jpg)\n*We introduce Spatially Incremental Generative Engine (SIGE),an engine that selectively performs computations at the edited regions for image editing applications. The computation and latency are measured for a single forward. For the above examples, SIGE significantly reduces the computation of [SDEdit](https://github.com/ermongroup/SDEdit) with [DDPM](https://github.com/lucidrains/denoising-diffusion-pytorch) (4-6x) and [Stable Diffusion](https://github.com/CompVis/stable-diffusion) (8x), and [GauGAN](https://github.com/NVlabs/SPADE) (15x) while preserving the image quality. When combined with existing model compression methods such as [GAN Compression](https://github.com/mit-han-lab/gan-compression), it further reduces the computation of GauGAN by 47x. On NVIDIA RTX 3090, SIGE achieves up to 7.2x speedups.*\n\nEfficient Spatially Sparse Inference for Conditional GANs and Diffusion Models\u003c/br\u003e\n[Muyang Li](https://lmxyy.me/), [Ji Lin](http://linji.me/), [Chenlin Meng](https://cs.stanford.edu/~chenlin/), [Stefano Ermon](https://cs.stanford.edu/~ermon/), [Song Han](https://songhan.mit.edu/), and [Jun-Yan Zhu](https://www.cs.cmu.edu/~junyanz/)\u003c/br\u003e\nCMU, MIT, and Stanford\u003c/br\u003e\nIn NeurIPS 2022.\n\n## Demo\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/lmxyy/sige/raw/main/diffusion_demo/assets/demo.gif\" width=600\u003e\n\u003c/p\u003e\n\n\nSIGE achieves 2x less conversion time compared to original DDPM on M1 MacBook Pro GPU as we selectively perform computation on the edited regions.\n\n## Overview\n\n![overview](https://github.com/lmxyy/sige/raw/main/assets/method.gif)*Tiling-based sparse convolution overview. For each convolution \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e in the network, we wrap it into SIGE Conv\u003csub\u003e\u003ci\u003el\u003c/i\u003e\u003c/sub\u003e. The activations of the original image are already pre-computed. When getting the edited image, we first compute a difference mask between the original and edited image and reduce the mask to the active block indices to locate the edited regions. In each SIGE Conv\u003csub\u003e\u003ci\u003el\u003c/i\u003e\u003c/sub\u003e, we directly gather the active blocks from the edited activation \u003ci\u003eA\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e\u003csup\u003eedited\u003c/sup\u003e according to the reduced indices, stack the blocks along the batch dimension, and feed them into \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e. The gathered blocks have an overlap of width 2 if \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e is 3×3 convolution with stride 1. After getting the output blocks from \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e, we scatter them back into \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e(\u003ci\u003eA\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e\u003csup\u003eoriginal\u003c/sup\u003e) to get the edited output, which approximates \u003ci\u003eF\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e(\u003ci\u003eA\u003csub\u003el\u003c/sub\u003e\u003c/i\u003e\u003csup\u003eedited\u003c/sup\u003e).*\n\n## Performance\n\n### Efficiency\n\n![overview](https://github.com/lmxyy/sige/raw/main/assets/results.jpg)\n*With 1.2% edits, SIGE could reduce the computation of DDPM, Progressive Distillation and GauGAN by 7-18x, achieve a 2-4x speedup on NVIDIA RTX 3090, 3-5x speedup on Apple M1 Pro GPU and 4-14x on M1 Pro CPU. When combined with GAN Compression, it further reduces 50x computation on GauGAN, achieving 38x speedup on M1 Pro CPU. Please check our paper for more details and results.*\n\n### Quality\n\n![overview](https://github.com/lmxyy/sige/raw/main/assets/quality.jpg)*Qualitative results under different edit sizes. PD is Progressive Distillation. Our method well preserves the visual fidelity of the original model without losing global context.*\n\n![quality-stable-diffusion](https://github.com/lmxyy/sige/raw/main/assets/quality-stable-diffusion.jpg)\n\n*More qualitative results of Stable Diffusion on both image inpainting and editing, measured on NVIDIA RTX 3090.*\n\nReferences:\n\n* Denoising Diffusion Probabilistic Models (DDPM), Ho et al., ICLR 2020\n* Denoising Diffusion Implicit Model (DDIM), Song et al., ICLR 2021\n* Progressive Distillation for Fast Sampling of Diffusion Models, Salimans et al., ICLR 2022\n* Semantic Image Synthesis with Spatially-Adaptive Normalization (GauGAN), Park et al., CVPR 2019\n* GAN Compression: Efficient Architectures for Interactive Conditional GANs, Li et al., CVPR 2020\n* High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al., CVPR 2022\n\n## Prerequisites\n\n* Python3\n* CPU, M1 GPU, or NVIDIA GPU + CUDA CuDNN\n* [PyTorch](https://pytorch.org) \u003e= 1.7. For M1 GPU support, please install [PyTorch](https://pytorch.org)\u003e=2.0.\n\n## Getting Started\n\n### Installation\n\nAfter installing [PyTorch](https://pytorch.org), you should be able to install SIGE with PyPI\n\n```shell\npip install sige\n```\n\nor via GitHub:\n\n```shell\npip install git+https://github.com/lmxyy/sige.git\n```\n\nor locally for development\n\n```shell\ngit clone git@github.com:lmxyy/sige.git\ncd sige\npip install -e .\n```\n\nFor MPS backend, please set the environment variables:\n\n```shell\nexport PYTORCH_ENABLE_MPS_FALLBACK=1\n```\n\n### Usage Example\n\nSee [example.py](https://github.com/lmxyy/sige/tree/main/example.py) for the minimal SIGE convolution example. Please first install SIGE with the above instructions and [torchprofile](https://github.com/zhijian-liu/torchprofile) with\n\n```shell\npip install torchprofile\n```\n\nThen you can run it with\n\n```shell\npython example.py\n```\n\nWe also have [![colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lmxyy/sige/blob/main/example.ipynb) example.\n\n### Benchmark\n\nTo reproduce the results of [DDPM](https://github.com/ermongroup/ddim) and [Progressive Distillation](https://github.com/google-research/google-research/tree/master/diffusion_distillation) or download the LSUN Church editing datasets, please follow the instructions in [diffusion/README.md](https://github.com/lmxyy/sige/tree/main/diffusion/README.md).\n\nTo reproduce the results of [GauGAN](https://github.com/NVlabs/SPADE) and [GAN Compression](https://github.com/mit-han-lab/gan-compression) or download the Cityscapes editing datasets, please follow the instructions in [gaugan/README.md](https://github.com/lmxyy/sige/tree/main/gaugan/README.md).\n\n## Citation\n\nIf you use this code for your research, please cite our paper.\n\n```bibtex\n@inproceedings{li2022efficient,\n  title={Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models},\n  author={Li, Muyang and Lin, Ji and Meng, Chenlin and Ermon, Stefano and Han, Song and Zhu, Jun-Yan},\n  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},\n  year={2022}\n}\n```\n\n## Acknowledgments\n\nOur code is developed based on [SDEdit](https://github.com/ermongroup/SDEdit), [ddim](https://github.com/ermongroup/ddim), [diffusion_distillation](https://github.com/google-research/google-research/tree/master/diffusion_distillation), [gan-compression](https://github.com/mit-han-lab/gan-compression), [dpm-solver](https://github.com/LuChengTHU/dpm-solver), and [stable-diffusion](https://github.com/CompVis/stable-diffusion). We refer to [sbnet](https://github.com/uber-research/sbnet) for the tiling-based sparse convolution algorithm implementation. Our work is also inspired by the gather/scatter implementations in [torchsparse](https://github.com/mit-han-lab/torchsparse).\n\nWe thank [torchprofile](https://github.com/zhijian-liu/torchprofile) for MACs measurement, [clean-fid](https://github.com/GaParmar/clean-fid) for FID computation and [drn](https://github.com/fyu/drn) for Cityscapes mIoU computation.\n\nWe thank Yaoyao Ding, Zihao Ye, Lianmin Zheng, Haotian Tang, and Ligeng Zhu for the helpful comments on the engine design. We also thank George Cazenavette, Kangle Deng, Ruihan Gao, Daohan Lu, Sheng-Yu Wang and Bingliang Zhang for their valuable feedback. The project is partly supported by NSF, MIT-IBM Watson AI Lab, Kwai Inc, and Sony Corporation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmxyy%2Fsige","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmxyy%2Fsige","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmxyy%2Fsige/lists"}