{"id":24191210,"url":"https://github.com/alpha-innovator/adaptivediffusion","last_synced_at":"2025-04-09T14:13:53.685Z","repository":{"id":258461888,"uuid":"871416097","full_name":"Alpha-Innovator/AdaptiveDiffusion","owner":"Alpha-Innovator","description":"[NeurIPS'24] Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy","archived":false,"fork":false,"pushed_at":"2025-01-22T18:32:28.000Z","size":9045,"stargazers_count":64,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T14:13:47.341Z","etag":null,"topics":["adaptive-inference","diffusion-models","efficient-inference","model-acceleration","stable-diffusion","training-free"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-Innovator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-11T23:26:59.000Z","updated_at":"2025-03-20T18:15:37.000Z","dependencies_parsed_at":"2025-01-10T06:35:41.457Z","dependency_job_id":null,"html_url":"https://github.com/Alpha-Innovator/AdaptiveDiffusion","commit_stats":null,"previous_names":["unimodal4reasoning/adaptivediffusion","alpha-innovator/adaptivediffusion"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FAdaptiveDiffusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FAdaptiveDiffusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FAdaptiveDiffusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FAdaptiveDiffusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-Innovator","download_url":"https://codeload.github.com/Alpha-Innovator/AdaptiveDiffusion/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054193,"owners_count":21039952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adaptive-inference","diffusion-models","efficient-inference","model-acceleration","stable-diffusion","training-free"],"created_at":"2025-01-13T15:17:38.376Z","updated_at":"2025-04-09T14:13:53.666Z","avatar_url":"https://github.com/Alpha-Innovator.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![arXiv](https://img.shields.io/badge/arXiv-2410.09873-b31b1b.svg)](https://arxiv.org/abs/2410.09873)\n[![GitHub issues](https://img.shields.io/github/issues/UniModal4Reasoning/AdaptiveDiffusion)](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/issues)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/pulls)\n\n\u003cdiv align=\"center\"\u003e\n\u003ch1 style=\"text-align: center; font-size: 2.5rem; font-weight: bolders\"\u003e\nNeurIPS-2024: Noise Prediction Can Be Adaptively Skipped for Different Prompts Without Training!\n\u003c/h1\u003e\n\n\n\u003cfont size=4\u003e[[Paper]](https://arxiv.org/pdf/2410.09873)\u003c/font\u003e \u0026nbsp; \u0026nbsp;\u003cfont size=4\u003e[[Project page]](https://jiakangyuan.github.io/AdaptiveDiffusion-project-page/)\u003c/font\u003e \u0026nbsp;\u0026nbsp;\u003cfont size=4\u003e[[Huggingface]](https://huggingface.co/datasets/HankYe/Sampled_AIGCBench_text2image_ar_0.625)\u003c/font\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/framework.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n## Introduction\nThis is the up-to-date official implementation of AdaptiveDiffusion in the paper, [**Training-free Adaptive Diffusion with Bounded Difference Approximation Strategy**](https://arxiv.org/abs/2410.09873). AdaptiveDiffusion is a novel adaptive inference paradigm containing a third-order latent differential estimator to determine whether to reuse the noise prediction from previous timesteps for the denoising of the current timestep. The developed skipping strategy adaptively approximates the optimal skipping strategy for various prompts based on the third-order latent differential value.\n\nAdaptiveDiffusion offers three core components:\n\n- Training-free adaptive diffusion acceleration [pipelines](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/acceleration/sparse_pipeline.py) from the step number reduction of noise predictions that makes different skipping paths for different prompts.\n- Unified skipping strategy for both image and video generation models.\n- Interchangeable noise [schedulers](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/generate.py) for different diffusion speeds and output quality.\n\n## Installation\n\nPlease follow the [installation](https://github.com/huggingface/diffusers/blob/main/README.md) to complete the installation. If the evaluation is required, `clean-fid` should be installed for images and videos.\n\n```\npip install git+https://github.com/zhijian-liu/torchprofile datasets torchmetrics dominate clean-fid\n```\n\n## Quickstart\n\nThanks to the unified inference pipelines in [diffusers](https://github.com/huggingface/diffusers), it is easy to deploy the third-order estimator on various diffusion pipelines to achieve adaptive diffusion.\n\n### Step One\nSelect the target pipeline that you attempt to accelerate. For the comparison with original diffusion results, you can copy the [pipeline](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/src/diffusers/pipelines/) classes to [sparse_pipeline](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/acceleration/sparse_pipeline.py).\n\n### Step Two\nModify the pipeline you just copied into the [sparse_pipeline](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/acceleration/sparse_pipeline.py). There are four places that need modification.\n\n1. Pipeline Initialization\n```python\nclass TargetPipeline(\n    #... existing code...\n):\n    def __init__(\n          #... existing code...\n          threshold: float = 0.01, # default_threshold\n          max_skip_steps: int = 4, # default max skipping time steps\n        )\n        #... existing code...\n        self.prev_latents = []\n        self.mask = []\n        self.diff_list = []\n        self.max_skip_steps = max_skip_steps\n        self.threshold = threshold\n```\n2. `Estimator` function design and `Reset` function definition in the target class.\n```python\nclass TargetPipeline(\n    #... existing code...\n):\n    #... existing code...\n    def estimate_skipping(self, latent):\n        prev_latent = self.prev_latents[-1]\n        \n        prev_diff = self.diff_list[-1]\n        prev_prev_diff = self.diff_list[-2]\n        cur_diff = (latent - prev_latent).abs().mean()\n        self.diff_list.append(cur_diff)\n        if len(self.mask) \u003e 4 and not any(self.mask[-self.max_skip_steps:]):\n            return True\n        if abs((cur_diff + prev_prev_diff) / 2 - prev_diff) \u003c= prev_diff * self.threshold:\n            return False\n        return True\n\n    def reset_cache(self):\n        self.noise_pred = None\n        self.prev_latents = []\n        self.mask = []\n        self.diff_list = []\n      \n    def __call__(\n        #... existing code...\n    ):\n        #... existing code...\n```\n3. Replace the denoising code.\n```python\nclass TargetPipeline(\n    #... existing code...\n):\n    #... existing code...\n    \n    def __call__(\n        #... existing code...\n    ):\n        #... existing code...\n\n        with self.progress_bar(total=num_inference_steps) as progress_bar:\n            #... existing code...\n            # original: noise_pred = self.unet(...)\n            # replaced with:\n            ###### estimate whether to skip steps #######\n            if len(self.prev_latents) \u003c= 3:\n                noise_pred = self.unet(...)[0]\n                self.noise_pred = noise_pred\n                if len(self.prev_latents) \u003e 1:\n                    self.diff_list.append((self.prev_latents[-1] - self.prev_latents[-2]).abs().mean())\n            else:\n                if self.mask[-1] == True:\n                    noise_pred = self.unet(...)[0]\n                    self.noise_pred = noise_pred\n                else:\n                    noise_pred = self.noise_pred\n            #... existing code...\n            latents = self.scheduler.step(...)[0]\n\n            if len(self.prev_latents) \u003e= 3:\n                self.mask.append(self.estimate_skipping(latents))\n            self.prev_latents.append(latents)\n            #... existing code...\n```\n4. Modify the inference code.\n```python\nimport sys\nsys.path.append('/path/to/examples/AdaptiveDiffusion')\nfrom acceleration.sparse_pipeline import TargetPipeline as AdaptiveTargetPipeline\nimport torch\n\nthreshold = 0.01\nmax_skip_steps = 4\npipeline = AdaptiveTargetPipeline.from_pretrained(..., threshold=threshold, max_skip_steps=max_skip_steps)\npipeline.scheduler = ... # in case you want to try more schedulers\npipeline.to(\"cuda\")\npipeline(\"An image of a squirrel in Picasso style\").images[0]\n```\n### Evaluation\nTo evaluate the generation quality of AdaptiveDiffusion, we follow [Distrifuser](https://github.com/mit-han-lab/distrifuser) to evaluate the generation similarity between the original and our adaptive diffusion model. After you generate all the images, you can use our script [`compute_metrics_image.py`](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/compute_metrics_image.py) and [`compute_metrics_video.py`](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/compute_metrics_video.py) to calculate PSNR, LPIPS and FID. The usage is\n```python\npython scripts/compute_metrics_image.py --input_root0 $IMAGE_ROOT0 --input_root1 $IMAGE_ROOT1\n```\nwhere `$IMAGE_ROOT0` and `$IMAGE_ROOT1` are paths to the image folders you are trying to compare.\n\n### Evaluation on AIGCBench\nFor the evaluation on the image-to-video generation task, we randomly select 100 samples from the validation set of [AIGCBench](https://arxiv.org/abs/2401.01651). The sample list is provided in [Huggingface](https://huggingface.co/datasets/HankYe/Sampled_AIGCBench_text2image_ar_0.625). After generating all the videos by [`generate_video.py`](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/generate_video.py), you can use our script [`compute_metrics_video.py`](https://github.com/UniModal4Reasoning/AdaptiveDiffusion/blob/master/examples/AdaptiveDiffusion/compute_metrics_video.py) to calculate PSNR, LPIPS and FVD. The usage is \n```python\npython scripts/compute_metrics_video.py --input_root0 $VIDEO_ROOT0 --input_root1 $VIDEO_ROOT1\n```\nwhere `$VIDEO_ROOT0` and `$VIDEO_ROOT1` are paths to the video folders you are trying to compare.\n\n## Demo\n\nYou can also try our demo by \n```python\ncd examples/AdaptiveDiffusion \u0026\u0026 python demo.py\n```\nThen, open the URL displayed in the terminal (For example, http://127.0.0.1:7860) and you can change the `model`, `seed`, `threshold`, and so on in the WebUI. The additional package required for the demo is `gradio`, and you can use `pip install gradio` to install it.\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"assets/demo.png\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n\n## Citation\n\n```bibtex\n@misc{adaptivediffusion24ye,\n  author = {Hancheng Ye and Jiakang Yuan and Renqiu Xia and Xiangchao Yan and Tao Chen and Junchi Yan and Botian Shi and Bo Zhang},\n  title = {Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy},\n  year = {2024},\n  booktitle = {The Thirty-Eighth Annual Conference on Neural Information Processing Systems}\n}\n```\n\n\n\n## Acknowledgements\nWe greatly acknowledge the authors of *Distrifuser*, *Torchsparse*, and *Diffusers* for their open-source codes. Visit the following links to access their more contributions.\n\n- [Distrifuser](https://github.com/mit-han-lab/distrifuser)\n- [Torchsparse](https://github.com/mit-han-lab/torchsparse)\n- [Diffusers](https://github.com/huggingface/diffusers)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fadaptivediffusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-innovator%2Fadaptivediffusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fadaptivediffusion/lists"}