{"id":22677200,"url":"https://github.com/redaigc/target-driven-distillation","last_synced_at":"2025-04-07T11:04:59.308Z","repository":{"id":254548895,"uuid":"845452016","full_name":"RedAIGC/Target-Driven-Distillation","owner":"RedAIGC","description":" Consistency Distillation with Target Timestep Selection and Decoupled Guidance","archived":false,"fork":false,"pushed_at":"2025-01-04T06:57:05.000Z","size":14590,"stargazers_count":76,"open_issues_count":12,"forks_count":11,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-07T11:04:50.473Z","etag":null,"topics":["consistency-models","diffusion-models","flux","lcm-lora","lora","sdxl-lightning","stable-diffusion","stable-video-diffusion"],"latest_commit_sha":null,"homepage":"https://redaigc.github.io/TDD/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedAIGC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-21T09:27:03.000Z","updated_at":"2025-04-03T02:53:29.000Z","dependencies_parsed_at":"2024-12-09T17:59:24.293Z","dependency_job_id":"0df60873-7ebe-49ce-898d-8384036503f2","html_url":"https://github.com/RedAIGC/Target-Driven-Distillation","commit_stats":{"total_commits":71,"total_committers":6,"mean_commits":"11.833333333333334","dds":0.4084507042253521,"last_synced_commit":"cb8776cb82d5d79eaf8a77a084c3c260dea0ba60"},"previous_names":["redaigc/target-driven-distillation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FTarget-Driven-Distillation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FTarget-Driven-Distillation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FTarget-Driven-Distillation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedAIGC%2FTarget-Driven-Distillation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedAIGC","download_url":"https://codeload.github.com/RedAIGC/Target-Driven-Distillation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247640462,"owners_count":20971557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consistency-models","diffusion-models","flux","lcm-lora","lora","sdxl-lightning","stable-diffusion","stable-video-diffusion"],"created_at":"2024-12-09T17:59:16.133Z","updated_at":"2025-04-07T11:04:59.291Z","avatar_url":"https://github.com/RedAIGC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# ✨Target-Driven Distillation✨\n\n[![Arxiv](https://img.shields.io/badge/arXiv-2402.19159-b31b1b)](https://arxiv.org/abs/2409.01347)\n[![Project page](https://img.shields.io/badge/Web-Project%20Page-green)](https://redaigc.github.io/TDD)\n[![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97HuggingFace-Model-purple)](https://huggingface.co/RED-AIGC/TDD)\n[![Hugging Face Space SDXL](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-FLUX_BETA-blue)](https://huggingface.co/spaces/RED-AIGC/FLUX-TDD-BETA)\n[![Hugging Face Space SDXL](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-SDXL-blue)](https://huggingface.co/spaces/RED-AIGC/TDD)\n[![Hugging Face Space SVD](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-SVD-blue)](https://huggingface.co/spaces/RED-AIGC/SVD-TDD)\n\n\u003c/div\u003e\n\nTarget-Driven Distillation (TDD) is a state-of-the-art consistency distillation model that largely accelerates the inference processes of diffusion models. Using its delicate strategies of *target timestep selection* and *decoupled guidance*, models distilled by TDD can generated highly detailed images with only a few steps.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/teaser.jpg\" alt=\"teaser\" style=\"zoom:80%;\" /\u003e\n\n  Samples generated by TDD-distilled SDXL, with only 4--8 steps.\n\u003c/div\u003e\n\n\n\n## News!\n\n- **Jan. 4, 2025**:  We have update codes with adv config for training on [FLUX](https://github.com/RedAIGC/Target-Driven-Distillation/tree/main/train/FLUX) \n- **Jan. 1, 2025**:  Demo of FLUX-TDD(training with adv) is now available on Hugging Face [![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97HuggingFace-Model-purple)](https://huggingface.co/RED-AIGC/TDD)\n- **Dec. 10, 2024**: Our paper is accepted by **AAAI25**!\n- **Sept. 21, 2024**: Demo of FLUX-TDD-BETA(4-8-steps) is now available on Hugging Face [![Hugging Face Space FLUX](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-FLUX-blue)](https://huggingface.co/spaces/RED-AIGC/FLUX-TDD-BETA)\n- **Sept. 20, 2024**: We have released codes for training on [FLUX](https://github.com/RedAIGC/Target-Driven-Distillation/tree/main/train/FLUX) , Our 4-8-steps FLUX.1-dev-related LoRAs are coming soon!\n- **Sept. 12, 2024**: Demos of TDD-SDXL and TDD-SVD are now available on Hugging Face [![Hugging Face Space SDXL](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-SDXL-blue)](https://huggingface.co/spaces/RED-AIGC/TDD)\n[![Hugging Face Space SVD](https://img.shields.io/badge/%F0%9F%A4%97HF%20Space-SVD-blue)](https://huggingface.co/spaces/RED-AIGC/SVD-TDD). Give them a try!\n- **Sept. 4, 2024**: Our detailed research paper is now on arXiv [![Arxiv](https://img.shields.io/badge/arXiv-2402.19159-b31b1b)](https://arxiv.org/abs/2409.01347).\n- **Aug. 29, 2024**: We have released codes for training and inference, as well as the pretrained models both w/ and w/o adv, on SDXL.\n- **Aug. 22, 2024**: Project launched.\n\n## Demos\n\n### Comparison with Previous Works(LCM, PCM, TCD). From the same seeds, our method(TDD) demonstrates advantages in both image complexity and clarity.\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/compare.png\" alt=\"comparison\" style=\"zoom:80%;\" /\u003e\n\u003c/div\u003e\n\n### Video samples generated by AnimateLCM-distilled (top) and TDD-distilled (bottom) SVD-xt 1.1, also with 4--8 steps.\nhttps://github.com/user-attachments/assets/09fcfc83-fbb8-45da-8ecf-18fa11a6bf82\n\n### Samples generated by TDD-distilled different base models, and by SDXL with different LoRA adapters or ControlNets.  \n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/other_1.jpg\" alt=\"other\"/\u003e\n\u003c/div\u003e\n\n## Usage\n\n### Inference\n- Clone this repository.\n```shell\ngit clone https://github.com/RedAIGC/Target-Driven-Distillation.git\ncd Target-Driven-Distillation\n```\n\n- FLUX Download pretrained models with the script below or from [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/RED-AIGC/TDD).\n```python\nfrom huggingface_hub import hf_hub_download\nfrom diffusers import FluxPipeline\n\npipe = FluxPipeline.from_pretrained(\"black-forest-labs/FLUX.1-dev\", torch_dtype=torch.bfloat16)\npipe.load_lora_weights(hf_hub_download(\"RED-AIGC/TDD\", \"FLUX.1-dev_tdd_lora_weights.safetensors\"))\npipe.fuse_lora(lora_scale=0.125)\npipe.to(\"cuda\")\n\nimage_flux = pipe(\n    prompt=[prompt],\n    generator=torch.Generator().manual_seed(int(3413)),\n    num_inference_steps=8,\n    guidance_scale=2.0,\n    height=1024,\n    width=1024,\n    max_sequence_length=256\n).images[0]\n```\n\n- SDXL Download pretrained models with the script below or from [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/RED-AIGC/TDD).\n```python\nfrom huggingface_hub import hf_hub_download\nhf_hub_download(repo_id=\"RedAIGC/TDD\", filename=\"sdxl_tdd_lora_weights.safetensors\", local_dir=\"./tdd_lora\")\n```\n\n- Generate images.\n```python\n# !pip install opencv-python transformers accelerate \nimport torch\nimport diffusers\nfrom diffusers import StableDiffusionXLPipeline\nfrom tdd_scheduler import TDDScheduler\n\ndevice = \"cuda\"\ntdd_lora_path = \"tdd_lora/sdxl_tdd_lora_weights.safetensors\"\n\npipe = StableDiffusionXLPipeline.from_pretrained(\"stabilityai/stable-diffusion-xl-base-1.0\", torch_dtype=torch.float16, variant=\"fp16\").to(device)\n\npipe.scheduler = TDDScheduler.from_config(pipe.scheduler.config)\npipe.load_lora_weights(tdd_lora_path, adapter_name=\"accelerate\")\npipe.fuse_lora()\n\nprompt=\"A photo of a cat made of water.\"\n\nimage = pipe(\n    prompt=prompt,\n    num_inference_steps=4,\n    guidance_scale=1.7,\n    eta=0.2, \n    generator=torch.Generator(device=device).manual_seed(546237),\n).images[0]\n\nimage.save(\"tdd.png\")\n```\n\n### Training\n\nSee scripts under [train](https://github.com/RedAIGC/Target-Driven-Distillation/tree/main/train).\n\n\n## Introduction\n\nTarget-Driven Distillation (TDD) features three key designs, that differ from previous consistency distillation methods.\n1. **TDD adopts a delicate selection strategy of target timesteps, increasing the training efficiency.** Specifically, it first chooses from a predefined set of equidistant denoising schedules (*e.g.* 4--8 steps), then adds a stochatic offset to accomodate non-deterministic sampling (*e.g.* $\\gamma$-sampling).\n2. **TDD utilizes decoupled guidances during training, making itself open to post-tuning on guidance scale during inference periods.** Specifically, it replaces a portion of the text conditions with unconditional (*i.e.* empty) prompts, in order to align with the standard training process using CFG.\n3. **TDD can be optionally equipped with non-equidistant sampling and x0 clipping, enabling a more flexible and accurate way for image sampling.**\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"assets/tdd_overview.jpg\" alt=\"overview\"/\u003e\n\n  An overview of TDD. (a) The training process features target timestep selection and decoupled guidance. (b) The inference process can optionally adopt non-equidistant denoising schedules.\n\u003c/div\u003e\n\nFor further details of TDD, please refer to our paper: [![Arxiv](https://img.shields.io/badge/arXiv-2402.19159-b31b1b)](https://arxiv.org/abs/2409.01347).\n\n## Acknowledgements\n- Thanks [sdbds](https://github.com/sdbds) help us in the training FLUX, This allows us to distill FLUX with a larger batch size.\n- Thanks [PSNbst](https://huggingface.co/PSNbst/PAseer-TDD-Accelerator) provide the compressed version of TDD, which is less than 20MB. Truly impressive.\n- Thanks to the [PCM](https://github.com/G-U-N/Phased-Consistency-Model) PCM team for their ADV_loss support!\n- Thanks to the [HuggingFace](https://github.com/huggingface) gradio team for their free GPU support!\n\n\n## Concact, Collaboration, and Citation![visitors](https://visitor-badge.laobi.icu/badge?page_id=RedAIGC.Target-Driven-Distillation)\n\nIf you have any questions about the code, please do not hesitate to contact me!\n\nEmail: polu@xiaohongshu.com\nEmail: wangcunzheng2000@163.com\n\n\u003c!-- If you find TDD helpful to your research, please cite our paper:\n```\n@article{Wang2024TDD,\n  title     = {Target-Driven Distillation: Consistency Distillation with Target Timestep Selection and Decoupled Guidance},\n  author    = {Cunzheng Wang and Ziyuan Guo and Yuxuan Duan and Huaxia Li and Nemo Chen and Xu Tang and Yao Hu},\n  journal   = {arXiv preprint arXiv:xxxx.xxxxx},\n  year      = {2024}\n}\n``` --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredaigc%2Ftarget-driven-distillation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredaigc%2Ftarget-driven-distillation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredaigc%2Ftarget-driven-distillation/lists"}