{"id":13455937,"url":"https://github.com/PRIS-CV/DemoFusion","last_synced_at":"2025-03-24T09:31:05.412Z","repository":{"id":208916819,"uuid":"711650878","full_name":"PRIS-CV/DemoFusion","owner":"PRIS-CV","description":"Let us democratise high-resolution generation! (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-04-15T05:26:23.000Z","size":61176,"stargazers_count":2003,"open_issues_count":22,"forks_count":227,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-03-17T03:15:32.925Z","etag":null,"topics":["aigc","genai","high-resolution","low-resource","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://ruoyidu.github.io/demofusion/demofusion.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRIS-CV.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-29T22:31:09.000Z","updated_at":"2025-03-16T14:12:00.000Z","dependencies_parsed_at":"2024-09-25T00:50:58.934Z","dependency_job_id":null,"html_url":"https://github.com/PRIS-CV/DemoFusion","commit_stats":null,"previous_names":["pris-cv/demofusion"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRIS-CV%2FDemoFusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRIS-CV%2FDemoFusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRIS-CV%2FDemoFusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRIS-CV%2FDemoFusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRIS-CV","download_url":"https://codeload.github.com/PRIS-CV/DemoFusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245232833,"owners_count":20581699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","genai","high-resolution","low-resource","stable-diffusion"],"created_at":"2024-07-31T08:01:13.709Z","updated_at":"2025-03-24T09:31:05.403Z","avatar_url":"https://github.com/PRIS-CV.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","T2I Diffusion Model augmentation","Repos","Other AI Tools"],"sub_categories":[],"readme":"# DemoFusion\n[![Project Page](https://img.shields.io/badge/Project-Page-green.svg)](https://ruoyidu.github.io/demofusion/demofusion.html)\n[![arXiv](https://img.shields.io/badge/arXiv-2311.16973-b31b1b.svg)](https://arxiv.org/pdf/2311.16973.pdf)\n[![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/lucataco/demofusion)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/DemoFusion-colab/blob/main/DemoFusion_colab.ipynb)\n[![Hugging Face](https://img.shields.io/badge/i2i-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/radames/Enhance-This-DemoFusion-SDXL)\n[![Page Views Count](https://badges.toozhao.com/badges/01HFMAPCVTA1T32KN2PASNYGYK/blue.svg)](https://badges.toozhao.com/stats/01HFMAPCVTA1T32KN2PASNYGYK \"Get your own page views count badge on badges.toozhao.com\")\n\nCode release for \"DemoFusion: Democratising High-Resolution Image Generation With No 💰\" \n\n\u003cimg src=\"figures/illustration.jpg\" width=\"800\"/\u003e\n\n**Abstract**: High-resolution image generation with Generative Artificial Intelligence (GenAI) has immense potential but, due to the enormous capital investment required for training, it is increasingly centralised to a few large corporations, and hidden behind paywalls. This paper aims to democratise high-resolution GenAI by advancing the frontier of high-resolution generation while remaining accessible to a broad audience. We demonstrate that existing Latent Diffusion Models (LDMs) possess untapped potential for higher-resolution image generation. Our novel DemoFusion framework seamlessly extends open-source GenAI models, employing Progressive Upscaling, Skip Residual, and Dilated Sampling mechanisms to achieve higher-resolution image generation. The progressive nature of DemoFusion requires more passes, but the intermediate results can serve as \"previews\", facilitating rapid prompt iteration.\n\n# News\n- **2024.02.27**: 🔥 DemoFusion has been accepted to CVPR'24!\n- **2023.12.15**: 🚀 A [ComfyUI Demofusion Custom Node](https://github.com/deroberon/demofusion-comfyui) is available! Thank [Andre](https://github.com/deroberon) for the implementation!\n- **2023.12.12**: ✨ DemoFusion with ControNet is availabe now! Check it out at `pipeline_demofusion_sdxl_controlnet`! The local [Gradio Demo](https://github.com/PRIS-CV/DemoFusion#DemoFusionControlNet-with-local-Gradio-demo) is also available.\n- **2023.12.10**: ✨ Image2Image is supported by `pipeline_demofusion_sdxl` now! The local [Gradio Demo](https://github.com/PRIS-CV/DemoFusion#Image2Image-with-local-Gradio-demo) is also available.\n- **2023.12.08**: 🚀 A HuggingFace Demo for Img2Img is now available! [![Hugging Face](https://img.shields.io/badge/i2i-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/radames/Enhance-This-DemoFusion-SDXL) Thank [Radamés](https://github.com/radames) for the implementation and [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-Diffusers-orange.svg)](https://huggingface.co/docs/diffusers/index) for the support!\n- **2023.12.07**: 🚀 Add Colab demo [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/DemoFusion-colab/blob/main/DemoFusion_colab.ipynb). Check it out! Thank [camenduru](https://github.com/camenduru) for the implementation!\n- **2023.12.06**: ✨ The local [Gradio Demo](https://github.com/PRIS-CV/DemoFusion#Text2Image-with-local-Gradio-demo) is now available! Better interaction and presentation!\n- **2023.12.04**: ✨ A [low-vram version](https://github.com/PRIS-CV/DemoFusion#Text2Image-on-Windows-with-8-GB-of-VRAM) of DemoFusion is available! Thank [klimaleksus](https://github.com/klimaleksus) for the implementation!\n- **2023.12.01**: 🚀 Integrated to [Replicate](https://replicate.com/explore). Check out the online demo: [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/lucataco/demofusion) Thank [Luis C.](https://github.com/lucataco) for the implementation!\n- **2023.11.29**: 💰 `pipeline_demofusion_sdxl` is released.\n\n# Usage\n## A quick try with integrated demos\n- HuggingFace Space: Try Text2Image generation at [![Hugging Face](https://img.shields.io/badge/t2i-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/fffiloni/DemoFusion) and Image2Image enhancement at [![Hugging Face](https://img.shields.io/badge/i2i-%F0%9F%A4%97%20Hugging%20Face-blue)](https://huggingface.co/spaces/radames/Enhance-This-DemoFusion-SDXL).\n- Colab: Try Text2Image generation at [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/DemoFusion-colab/blob/main/DemoFusion_colab.ipynb) and Image2Image enhancement at [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/camenduru/DemoFusion-colab/blob/main/DemoFusion_img2img_colab.ipynb).\n- Replicate: Try Text2Image generation at [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/lucataco/demofusion) and Image2Image enhancement at [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/lucataco/demofusion-enhance).\n\n## Starting with our code\n### Hyper-parameters\n- `view_batch_size` (`int`, defaults to 16):\n  The batch size for multiple denoising paths. Typically, a larger batch size can result in higher efficiency but comes with increased GPU memory requirements.\n- `stride` (`int`, defaults to 64):\n  The stride of moving local patches. A smaller stride is better for alleviating seam issues, but it also introduces additional computational overhead and inference time.\n- `cosine_scale_1` (`float`, defaults to 3):\n  Control the decreasing rate of skip-residual. A smaller value results in better consistency with low-resolution results, but it may lead to more pronounced upsampling noise. Please refer to Appendix C in the DemoFusion paper.\n- `cosine_scale_2` (`float`, defaults to 1):\n  Control the decreasing rate of dilated sampling. A smaller value can better address the repetition issue, but it may lead to grainy images. For specific impacts, please refer to Appendix C in the DemoFusion paper.\n- `cosine_scale_3` (`float`, defaults to 1):\n  Control the decrease rate of the Gaussian filter. A smaller value results in less grainy images, but it may lead to over-smoothing images. Please refer to Appendix C in the DemoFusion paper.\n- `sigma` (`float`, defaults to 1):\n  The standard value of the Gaussian filter. A larger sigma promotes the global guidance of dilated sampling, but it has the potential of over-smoothing.\n- `multi_decoder` (`bool`, defaults to True):\n  Determine whether to use a tiled decoder. Generally, a tiled decoder becomes necessary when the resolution exceeds 3072*3072 on an RTX 3090 GPU.\n- `show_image` (`bool`, defaults to False):\n  Determine whether to show intermediate results during generation.\n\n### Text2Image (will take about 17 GB of VRAM)\n- Set up the dependencies as:\n```\nconda create -n demofusion python=3.9\nconda activate demofusion\npip install -r requirements.txt\n```\n- Download `pipeline_demofusion_sdxl.py` and run it as follows. A use case can be found in `demo.ipynb`.\n```\nfrom pipeline_demofusion_sdxl import DemoFusionSDXLPipeline\nimport torch\n\nmodel_ckpt = \"stabilityai/stable-diffusion-xl-base-1.0\"\npipe = DemoFusionSDXLPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16)\npipe = pipe.to(\"cuda\")\n\nprompt = \"Envision a portrait of an elderly woman, her face a canvas of time, framed by a headscarf with muted tones of rust and cream. Her eyes, blue like faded denim. Her attire, simple yet dignified.\"\nnegative_prompt = \"blurry, ugly, duplicate, poorly drawn, deformed, mosaic\"\n\nimages = pipe(prompt, negative_prompt=negative_prompt,\n              height=3072, width=3072, view_batch_size=16, stride=64,\n              num_inference_steps=50, guidance_scale=7.5,\n              cosine_scale_1=3, cosine_scale_2=1, cosine_scale_3=1, sigma=0.8,\n              multi_decoder=True, show_image=True\n             )\n\nfor i, image in enumerate(images):\n    image.save('image_' + str(i) + '.png')\n```\n- ⚠️ When you have enough VRAM (e.g., generating 2048*2048 images on hardware with more than 18GB RAM), you can set `multi_decoder=False`, which can make the decoding process faster.\n- Please feel free to try different prompts and resolutions.\n- Default hyper-parameters are recommended, but they may not be optimal for all cases. For specific impacts of each hyper-parameter, please refer to Appendix C in the DemoFusion paper.\n- The code was cleaned before the release. If you encounter any issues, please contact us.\n\n### Text2Image on Windows with 8 GB of VRAM\n\n- Set up the environment as:\n\n```\ncmd\ngit clone \"https://github.com/PRIS-CV/DemoFusion\"\ncd DemoFusion\npython -m venv venv\nvenv\\Scripts\\activate\npip install -U \"xformers==0.0.22.post7+cu118\" --index-url https://download.pytorch.org/whl/cu118\npip install \"diffusers==0.21.4\" \"matplotlib==3.8.2\" \"transformers==4.35.2\" \"accelerate==0.25.0\"\n```\n\n- Launch DemoFusion as follows. The use case can be found in `demo_lowvram.py`.\n\n```\npython\nfrom pipeline_demofusion_sdxl import DemoFusionSDXLPipeline\n\nimport torch\nfrom diffusers.models import AutoencoderKL\nvae = AutoencoderKL.from_pretrained(\"madebyollin/sdxl-vae-fp16-fix\", torch_dtype=torch.float16)\n\nmodel_ckpt = \"stabilityai/stable-diffusion-xl-base-1.0\"\npipe = DemoFusionSDXLPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16, vae=vae)\npipe = pipe.to(\"cuda\")\n\nprompt = \"Envision a portrait of an elderly woman, her face a canvas of time, framed by a headscarf with muted tones of rust and cream. Her eyes, blue like faded denim. Her attire, simple yet dignified.\"\nnegative_prompt = \"blurry, ugly, duplicate, poorly drawn, deformed, mosaic\"\n\nimages = pipe(prompt, negative_prompt=negative_prompt,\n              height=2048, width=2048, view_batch_size=4, stride=64,\n              num_inference_steps=40, guidance_scale=7.5,\n              cosine_scale_1=3, cosine_scale_2=1, cosine_scale_3=1, sigma=0.8,\n              multi_decoder=True, show_image=False, lowvram=True\n             )\n\nfor i, image in enumerate(images):\n    image.save('image_' + str(i) + '.png')\n```\n### Text2Image with local Gradio demo\n- Make sure you have installed `gradio` and `gradio_imageslider`.\n- Launch DemoFusion via Gradio demo now -- try `python gradio_demo.py`! Better Interaction and Presentation！\n\u003cimg src=\"figures/gradio_demo.png\" width=\"600\"/\u003e\n\n### Image2Image with local Gradio demo\n- Make sure you have installed `gradio` and `gradio_imageslider`.\n- Launch DemoFusion Image2Image by `python gradio_demo_img2img.py`.\n\u003cimg src=\"figures/gradio_demo_img2img.png\" width=\"600\"/\u003e\n- ⚠️ Please note that, as a tuning-free framework, DemoFusion's Image2Image capability is strongly correlated with the SDXL's training data distribution and will show a significant bias. An accurate prompt to describe the content and style of the input also significantly improves performance. Have fun and regard it as a side application of text+image based generation.\n\n### DemoFusion+ControlNet with local Gradio demo\n- Make sure you have installed `gradio` and `gradio_imageslider`.\n- Launch DemoFusion+ControNet Text2Image by `python gradio_demo.py`.\n- \u003cimg src=\"figures/gradio_demo_controlnet.png\" width=\"600\"/\u003e\n- Launch DemoFusion+ControNet Image2Image by `python gradio_demo_img2img.py`.\n- \u003cimg src=\"figures/gradio_demo_controlnet_img2img.png\" width=\"600\"/\u003e\n\n## Citation\nIf you find this paper useful in your research, please consider citing:\n```\n@inproceedings{du2024demofusion,\n  title={DemoFusion: Democratising High-Resolution Image Generation With No \\$\\$\\$},\n  author={Du, Ruoyi and Chang, Dongliang and Hospedales, Timothy and Song, Yi-Zhe and Ma, Zhanyu},\n  booktitle={CVPR},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRIS-CV%2FDemoFusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPRIS-CV%2FDemoFusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRIS-CV%2FDemoFusion/lists"}