{"id":17714305,"url":"https://github.com/pkulwj1994/diff_instruct","last_synced_at":"2025-03-13T22:32:21.621Z","repository":{"id":231511754,"uuid":"743391781","full_name":"pkulwj1994/diff_instruct","owner":"pkulwj1994","description":"official code for Diff-Instruct algorithm for one-step diffusion distillation","archived":false,"fork":false,"pushed_at":"2025-01-09T15:41:34.000Z","size":116,"stargazers_count":63,"open_issues_count":3,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-09T16:55:12.608Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pkulwj1994.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-15T06:12:28.000Z","updated_at":"2025-01-09T15:41:38.000Z","dependencies_parsed_at":"2024-04-06T08:41:37.527Z","dependency_job_id":null,"html_url":"https://github.com/pkulwj1994/diff_instruct","commit_stats":null,"previous_names":["pkulwj1994/diff_instruct"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkulwj1994%2Fdiff_instruct","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkulwj1994%2Fdiff_instruct/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkulwj1994%2Fdiff_instruct/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pkulwj1994%2Fdiff_instruct/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pkulwj1994","download_url":"https://codeload.github.com/pkulwj1994/diff_instruct/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243494534,"owners_count":20299830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-25T11:02:20.983Z","updated_at":"2025-03-13T22:32:21.616Z","avatar_url":"https://github.com/pkulwj1994.png","language":"Python","funding_links":[],"categories":["Accelerate"],"sub_categories":[],"readme":"## Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models (Diff-Instruct)\u003cbr\u003e\u003csub\u003eOfficial PyTorch implementation of the NeurIPS 2023 paper\u003c/sub\u003e\n\n**Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models**\u003cbr\u003e\nWeijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li and Zhihua Zhang.\n\u003cbr\u003ehttps://openreview.net/forum?id=MLIs5iRq4w\u003cbr\u003e\n\nAbstract: *Due to the ease of training, ability to scale, and high sample quality, diffusion models (DMs) have become the preferred option for generative modeling, with numerous pre-trained models available for a wide variety of datasets. Containing intricate information about data distributions, pre-trained DMs are valuable assets for downstream applications. In this work, we consider learning from pre-trained DMs and transferring their knowledge to other generative models in a data-free fashion. Specifically, we propose a general framework called Diff-Instruct to instruct the training of arbitrary generative models as long as the generated samples are differentiable with respect to the model parameters. Our proposed Diff-Instruct is built on a rigorous mathematical foundation where the instruction process directly corresponds to minimizing a novel divergence we call Integral Kullback-Leibler (IKL) divergence. IKL is tailored for DMs by calculating the integral of the KL divergence along a diffusion process, which we show to be more robust in comparing distributions with misaligned supports. We also reveal non-trivial connections of our method to existing works such as DreamFusion \\citep{poole2022dreamfusion}, and generative adversarial training. To demonstrate the effectiveness and universality of Diff-Instruct, we consider two scenarios: distilling pre-trained diffusion models and refining existing GAN models. The experiments on distilling pre-trained diffusion models show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. The experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models across various settings. Our official code is released through \\url{https://github.com/pkulwj1994/diff_instruct}.*\n\nCode was based on Pytorch implementation of EDM diffusion model: https://github.com/NVlabs/edm.\n\n## Prepare conda env\n\ngit clone https://github.com/pkulwj1994/diff_instruct.git\ncd diff_instruct\n\nsource activate\nconda create -n di_v100 python=3.8\nconda activate di_v100\npip install torch==1.12.1 torchvision==0.13.1 tqdm click psutil scipy\n\n## Pre-trained models\n\nWe use pre-trained EDM models:\n\n- [https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/](https://nvlabs-fi-cdn.nvidia.com/edm/pretrained/)\n\n\n## Preparing datasets\n\nDatasets are stored in the same format as in [StyleGAN](https://github.com/NVlabs/stylegan3): uncompressed ZIP archives containing uncompressed PNG files and a metadata file `dataset.json` for labels. Custom datasets can be created from a folder containing images; see [`python dataset_tool.py --help`](./docs/dataset-tool-help.txt) for more information.\n\n**CIFAR-10:** Download the [CIFAR-10 python version](https://www.cs.toronto.edu/~kriz/cifar.html) and convert to ZIP archive:\n\n```.bash\npython dataset_tool_edm.py --source=/data/downloads/cifar-10-python.tar.gz --dest=/data/datasets/cifar10-32x32.zip\n```\n\n**ImageNet:** Download the [ImageNet Object Localization Challenge](https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data) and convert to ZIP archive at 64x64 resolution:\n\n```.bash\npython dataset_tool.py --source=/data/downloads/imagenet/ILSVRC/Data/CLS-LOC/train --dest=/data/datasets/imagenet-64x64.zip --resolution=64x64 --transform=center-crop\n```\n\n## Distill single-step models for CIFAR10 unconditional generation on a single V100 GPU (result in an FID \u003c= 4.5)\n\nYou can run diffusion distillation using `di_train.py`. For example:\n\n```.bash\n# Train one-step DI model for unconditional CIFAR-10 using 1 GPUs\nCUDA_VISIBLE_DEVICES=0 torchrun --standalone --nproc_per_node=1 --master_port=25678 di_train.py --outdir=/logs/di/ci10-uncond --data=/data/datasets/cifar10-32x32.zip --arch=ddpmpp --batch 128 --edm_model cifar10-uncond --cond=0 --metrics fid50k_full --tick 10 --snap 50 --lr 0.00001 --glr 0.00001 --init_sigma 1.0 --fp16=0 --lr_warmup_kimg -1 --ls 1.0 --sgls 1.0\n```\n\nIn the experiment, the FID will be calculated automatically for each \"snap\" of rounds. \n\n## License\n\nAll material, including source code and pre-trained models, is licensed under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).\n\n## Citation\n\n```\n@article{luo2024diffinstruct,\n  title={Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models},\n  author={Luo, Weijian and Hu, Tianyang and Zhang, Shifeng and Sun, Jiacheng and Li, Zhenguo and Zhang, Zhihua},\n  journal={Advances in Neural Information Processing Systems},\n  volume={36},\n  year={2024}\n}\n```\n\n## Development\n\nThis is a research reference implementation and is treated as a one-time code drop. As such, we do not accept outside code contributions in the form of pull requests.\n\n## Acknowledgments\n\nWe thank EDM paper \"\"Elucidating the Design Space of Diffusion-Based Generative Models\"\" for its great implementation of EDM diffusion models in https://github.com/NVlabs/edm. We thank Shuchen Xue, and Zhengyang Geng for constructive feedback on code implementations.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkulwj1994%2Fdiff_instruct","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpkulwj1994%2Fdiff_instruct","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpkulwj1994%2Fdiff_instruct/lists"}