{"id":13932097,"url":"https://github.com/instantX-research/InstantStyle","last_synced_at":"2025-07-19T16:31:04.441Z","repository":{"id":231041455,"uuid":"734573996","full_name":"instantX-research/InstantStyle","owner":"instantX-research","description":"InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation 🔥","archived":false,"fork":false,"pushed_at":"2024-09-18T09:33:36.000Z","size":38953,"stargazers_count":1603,"open_issues_count":34,"forks_count":99,"subscribers_count":22,"default_branch":"main","last_synced_at":"2024-09-19T12:34:46.778Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://instantstyle.github.io/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/instantX-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-22T03:21:58.000Z","updated_at":"2024-09-18T14:14:00.000Z","dependencies_parsed_at":"2024-04-13T07:31:15.138Z","dependency_job_id":"1af33570-3872-4164-ba0b-0d825e658427","html_url":"https://github.com/instantX-research/InstantStyle","commit_stats":null,"previous_names":["instantstyle/instantstyle","instantx-research/instantstyle"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantX-research%2FInstantStyle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantX-research%2FInstantStyle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantX-research%2FInstantStyle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/instantX-research%2FInstantStyle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/instantX-research","download_url":"https://codeload.github.com/instantX-research/InstantStyle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226635591,"owners_count":17662051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T21:00:48.681Z","updated_at":"2024-11-26T22:31:10.811Z","avatar_url":"https://github.com/instantX-research.png","language":"Jupyter Notebook","funding_links":[],"categories":["Repos","图像生成","Jupyter Notebook"],"sub_categories":["资源传输下载"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eInstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation\u003c/h1\u003e\n\n[**Haofan Wang**](https://haofanwang.github.io/)\u003csup\u003e*\u003c/sup\u003e · [**Matteo Spinelli**](https://github.com/cubiq) · [**Qixun Wang**](https://github.com/wangqixun) · [**Xu Bai**](https://huggingface.co/baymin0220) · [**Zekui Qin**](https://github.com/ZekuiQin) · [**Anthony Chen**](https://antonioo-c.github.io/)\n\nInstantX Team \n\n\u003csup\u003e*\u003c/sup\u003ecorresponding authors\n\n\u003ca href='https://instantstyle.github.io/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-green'\u003e\u003c/a\u003e\n\u003ca href='https://arxiv.org/abs/2404.02733'\u003e\u003cimg src='https://img.shields.io/badge/Technique-Report-red'\u003e\u003c/a\u003e\n[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-red)](https://huggingface.co/spaces/InstantX/InstantStyle)\n[![ModelScope](https://img.shields.io/badge/ModelScope-Studios-blue)](https://modelscope.cn/studios/instantx/InstantStyle/summary)\n[![GitHub](https://img.shields.io/github/stars/InstantStyle/InstantStyle?style=social)](https://github.com/InstantStyle/InstantStyle)\n\n\u003c/div\u003e\n\nInstantStyle is a general framework that employs two straightforward yet potent techniques for achieving an effective disentanglement of style and content from reference images.\n\n\u003c!-- \u003cimg src='assets/pipe.png'\u003e --\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src='assets/page0.png' width = 900 \u003e\n\u003c/div\u003e\n\n\n## Principle\n\nSeparating Content from Image. Benefit from the good characterization of CLIP global features, after subtracting the content text fea- tures from the image features, the style and content can be explicitly decoupled. Although simple, this strategy is quite effective in mitigating content leakage.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/subtraction.png\"\u003e\n\u003c/p\u003e\n\nInjecting into Style Blocks Only. Empirically, each layer of a deep network captures different semantic information the key observation in our work is that there exists two specific attention layers handling style. Specifically, we find up blocks.0.attentions.1 and down blocks.2.attentions.1 capture style (color, material, atmosphere) and spatial layout (structure, composition) respectively.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/tree.png\"\u003e\n\u003c/p\u003e\n\n## Release\n- [2024/07/06] 🔥 We release [CSGO](https://github.com/instantX-research/CSGO) page for content-style composition. Code will be released soon.\n- [2024/07/01] 🔥 We release [InstantStyle-Plus](https://instantstyle-plus.github.io/) report for content preserving.\n- [2024/04/29] 🔥 We support InstantStyle natively in diffusers, usage can be found [here](https://github.com/InstantStyle/InstantStyle?tab=readme-ov-file#use-in-diffusers)\n- [2024/04/24] 🔥 InstantStyle for fast generation, find demos at [InstantStyle-SDXL-Lightning](https://huggingface.co/spaces/radames/InstantStyle-SDXL-Lightning) and [InstantStyle-Hyper-SDXL](https://huggingface.co/spaces/radames/InstantStyle-Hyper-SDXL).\n- [2024/04/24] 🔥 We support [HiDiffusion](https://github.com/megvii-research/HiDiffusion) for generating highres images, find more information [here](https://github.com/InstantStyle/InstantStyle/tree/main?tab=readme-ov-file#high-resolution-generation).\n- [2024/04/23] 🔥 InstantStyle has been natively supported in diffusers, more information can be found [here](https://github.com/huggingface/diffusers/pull/7668).\n- [2024/04/20] 🔥 InstantStyle is supported in [Mikubill/sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet/discussions/2770).\n- [2024/04/11] 🔥 We add the experimental distributed inference feature. Check it [here](https://github.com/InstantStyle/InstantStyle?tab=readme-ov-file#distributed-inference).\n- [2024/04/10] 🔥 We support an [online demo](https://modelscope.cn/studios/instantx/InstantStyle/summary) on ModelScope.\n- [2024/04/09] 🔥 We support an [online demo](https://huggingface.co/spaces/InstantX/InstantStyle) on Huggingface.\n- [2024/04/09] 🔥 We support SDXL-inpainting, more information can be found [here](https://github.com/InstantStyle/InstantStyle/blob/main/infer_style_inpainting.py).\n- [2024/04/08] 🔥 InstantStyle is supported in [AnyV2V](https://tiger-ai-lab.github.io/AnyV2V/) for stylized video-to-video editing, demo can be found [here](https://twitter.com/vinesmsuic/status/1777170927500787782).\n- [2024/04/07] 🔥 We support image-based stylization, more information can be found [here](https://github.com/InstantStyle/InstantStyle/blob/main/infer_style_controlnet.py).\n- [2024/04/07] 🔥 We support an experimental version for SD1.5, more information can be found [here](https://github.com/InstantStyle/InstantStyle/blob/main/infer_style_sd15.py).\n- [2024/04/03] 🔥 InstantStyle is supported in [ComfyUI_IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus) developed by our co-author.\n- [2024/04/03] 🔥 We release the [technical report](https://arxiv.org/abs/2404.02733).\n\n## Demos\n\n### Stylized Synthesis\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/example1.png\"\u003e\n  \u003cimg src=\"assets/example2.png\"\u003e\n\u003c/p\u003e\n\n### Image-based Stylized Synthesis\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/example3.png\"\u003e\n\u003c/p\u003e\n\n### Comparison with Previous Works\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/comparison.png\"\u003e\n\u003c/p\u003e\n\n## Download\nFollow [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter?tab=readme-ov-file#download-models) to download pre-trained checkpoints from [here](https://huggingface.co/h94/IP-Adapter).\n\n```\ngit clone https://github.com/InstantStyle/InstantStyle.git\ncd InstantStyle\n\n# download the models\ngit lfs install\ngit clone https://huggingface.co/h94/IP-Adapter\nmv IP-Adapter/models models\nmv IP-Adapter/sdxl_models sdxl_models\n```\n\n## Usage\n\nOur method is fully compatible with [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter). For feature subtraction, it only works for global feature instead of patch features. For SD1.5, you can find a demo at [infer_style_sd15.py](https://github.com/InstantStyle/InstantStyle/blob/main/infer_style_sd15.py), but we find that SD1.5 has weaker perception and understanding of style information, thus this demo is experimental only. All block names can be found in [attn_blocks.py](https://github.com/InstantStyle/InstantStyle/blob/main/attn_blocks.py) and [attn_blocks_sd15.py](https://github.com/InstantStyle/InstantStyle/blob/main/attn_blocks_sd15.py) for SDXL and SD1.5 respectively.\n\n```python\nimport torch\nfrom diffusers import StableDiffusionXLPipeline\nfrom PIL import Image\n\nfrom ip_adapter import IPAdapterXL\n\nbase_model_path = \"stabilityai/stable-diffusion-xl-base-1.0\"\nimage_encoder_path = \"sdxl_models/image_encoder\"\nip_ckpt = \"sdxl_models/ip-adapter_sdxl.bin\"\ndevice = \"cuda\"\n\n# load SDXL pipeline\npipe = StableDiffusionXLPipeline.from_pretrained(\n    base_model_path,\n    torch_dtype=torch.float16,\n    add_watermarker=False,\n)\n\n# reduce memory consumption\npipe.enable_vae_tiling()\n\n# load ip-adapter\n# target_blocks=[\"block\"] for original IP-Adapter\n# target_blocks=[\"up_blocks.0.attentions.1\"] for style blocks only\n# target_blocks = [\"up_blocks.0.attentions.1\", \"down_blocks.2.attentions.1\"] # for style+layout blocks\nip_model = IPAdapterXL(pipe, image_encoder_path, ip_ckpt, device, target_blocks=[\"up_blocks.0.attentions.1\"])\n\nimage = \"./assets/0.jpg\"\nimage = Image.open(image)\nimage.resize((512, 512))\n\n# generate image variations with only image prompt\nimages = ip_model.generate(pil_image=image,\n                            prompt=\"a cat, masterpiece, best quality, high quality\",\n                            negative_prompt= \"text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry\",\n                            scale=1.0,\n                            guidance_scale=5,\n                            num_samples=1,\n                            num_inference_steps=30, \n                            seed=42,\n                            #neg_content_prompt=\"a rabbit\",\n                            #neg_content_scale=0.5,\n                          )\n\nimages[0].save(\"result.png\")\n```\n\n## Use in diffusers\nInstantStyle has already been integrated into [diffusers](https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#style--layout-control) (please make sure that you have installed diffusers\u003e=0.28.0.dev0), making the usage significantly simpler. You can now control the per-transformer behavior of each IP-Adapter with the set_ip_adapter_scale() method, using a configuration dictionary as shown below:\n\n```python\nfrom diffusers import StableDiffusionXLPipeline\nfrom PIL import Image\nimport torch\n\n# load SDXL pipeline\npipe = StableDiffusionXLPipeline.from_pretrained(\n    \"stabilityai/stable-diffusion-xl-base-1.0\",\n    torch_dtype=torch.float16,\n    add_watermarker=False,\n)\n\n# load ip-adapter\npipe.load_ip_adapter(\"h94/IP-Adapter\", subfolder=\"sdxl_models\", weight_name=\"ip-adapter_sdxl.bin\")\npipe.enable_vae_tiling()\n\n# configure ip-adapter scales.\nscale = {\n    \"down\": {\"block_2\": [0.0, 1.0]},\n    \"up\": {\"block_0\": [0.0, 1.0, 0.0]},\n}\npipeline.set_ip_adapter_scale(scale)\n```\n\nIn this example. We set ```scale=1.0``` for IP-Adapter in the second transformer of down-part, block 2, and the second in up-part, block 0. Note that there are 2 transformers in down-part block 2 so the list is of length 2, and so do the up-part block 0. The rest IP-Adapter will have a zero scale which means disable them in all the other layers.\n\nWith the help of ```set_ip_adapter_scale()```, we can now configure IP-Adapters without a need of reloading them everytime we want to test the IP-Adapter behaviors.\n\n```python\n# for original IP-Adapter\nscale = 1.0\npipeline.set_ip_adapter_scale(scale)\n\n# for style blocks only\nscale = {\n    \"up\": {\"block_0\": [0.0, 1.0, 0.0]},\n}\npipeline.set_ip_adapter_scale(scale)\n```\n\n### Multiple IP-Adapter images with masks\nYou can also load multiple IP-Adapters, together with multiple IP-Adapter images with masks for more precisely layout control just as that in [IP-Adapter](https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter#ip-adapter-masking) do.\n\n```python\nfrom diffusers import StableDiffusionXLPipeline\nfrom diffusers.image_processor import IPAdapterMaskProcessor\nfrom transformers import CLIPVisionModelWithProjection\nfrom PIL import Image\nimport torch\n\nimage_encoder = CLIPVisionModelWithProjection.from_pretrained(\n    \"h94/IP-Adapter\", subfolder=\"models/image_encoder\", torch_dtype=torch.float16\n).to(\"cuda\")\n\npipe = StableDiffusionXLPipeline.from_pretrained(\n    \"RunDiffusion/Juggernaut-XL-v9\", torch_dtype=torch.float16, image_encoder=image_encoder, variant=\"fp16\"\n).to(\"cuda\")\n\npipe.load_ip_adapter(\n    [\"ostris/ip-composition-adapter\", \"h94/IP-Adapter\"],\n    subfolder=[\"\", \"sdxl_models\"],\n    weight_name=[\n        \"ip_plus_composition_sdxl.safetensors\",\n        \"ip-adapter_sdxl_vit-h.safetensors\",\n    ],\n    image_encoder_folder=None,\n)\n\nscale_1 = {\n    \"down\": [[0.0, 0.0, 1.0]],\n    \"mid\": [[0.0, 0.0, 1.0]],\n    \"up\": {\"block_0\": [[0.0, 0.0, 1.0], [1.0, 1.0, 1.0], [0.0, 0.0, 1.0]], \"block_1\": [[0.0, 0.0, 1.0]]},\n}\n# activate the first IP-Adapter in everywhere in the model,\n# configure the second one for precise style control to each masked input.\npipe.set_ip_adapter_scale([1.0, scale_1])\n\nprocessor = IPAdapterMaskProcessor()\nfemale_mask = Image.open(\"./assets/female_mask.png\")\nmale_mask = Image.open(\"./assets/male_mask.png\")\nbackground_mask = Image.open(\"./assets/background_mask.png\")\ncomposition_mask = Image.open(\"./assets/composition_mask.png\")\nmask1 = processor.preprocess([composition_mask], height=1024, width=1024)\nmask2 = processor.preprocess([female_mask, male_mask, background_mask], height=1024, width=1024)\nmask2 = mask2.reshape(1, mask2.shape[0], mask2.shape[2], mask2.shape[3])   # output -\u003e (1, 3, 1024, 1024)\n\nip_female_style = Image.open(\"./assets/ip_female_style.png\")\nip_male_style = Image.open(\"./assets/ip_male_style.png\")\nip_background = Image.open(\"./assets/ip_background.png\")\nip_composition_image = Image.open(\"./assets/ip_composition_image.png\")\n\nimage = pipe(\n    prompt=\"high quality, cinematic photo, cinemascope, 35mm, film grain, highly detailed\",\n    negative_prompt=\"\",\n    ip_adapter_image=[ip_composition_image, [ip_female_style, ip_male_style, ip_background]],\n    cross_attention_kwargs={\"ip_adapter_masks\": [mask1, mask2]},\n    guidance_scale=6.5,\n    num_inference_steps=25,\n).images[0]\nimage\n\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/multi_instantstyle.png\"\u003e\n\u003c/p\u003e\n\n## High Resolution Generation\nWe employ [HiDiffusion](https://github.com/megvii-research/HiDiffusion) to seamlessly generate high-resolution images, you can install via `pip install hidiffusion`.\n\n```python\nfrom hidiffusion import apply_hidiffusion, remove_hidiffusion\n\n# reduce memory consumption\npipe.enable_vae_tiling()\n\n# apply hidiffusion with a single line of code.\napply_hidiffusion(pipe)\n\n...\n\n# generate image at higher resolution\nimages = ip_model.generate(pil_image=image,\n                           prompt=\"a cat, masterpiece, best quality, high quality\",\n                           negative_prompt= \"text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry\",\n                           scale=1.0,\n                           guidance_scale=5,\n                           num_samples=1,\n                           num_inference_steps=30, \n                           seed=42,\n                           height=2048,\n                           width=2048\n                          )\n```\n\n## Distributed Inference\nOn distributed setups, you can run inference across multiple GPUs with 🤗 Accelerate or PyTorch Distributed, which is useful for generating with multiple prompts in parallel, in case you have limited VRAM on each GPU. More information can be found [here](https://huggingface.co/docs/diffusers/main/en/training/distributed_inference#device-placement). Make sure you have installed diffusers from the source and the lastest accelerate.\n\n```python\nmax_memory = {0:\"10GB\", 1:\"10GB\"}\npipe = StableDiffusionXLPipeline.from_pretrained(\n    base_model_path,\n    torch_dtype=torch.float16,\n    add_watermarker=False,\n    device_map=\"balanced\",\n    max_memory=max_memory\n)\n```\n\n## Start a local gradio demo \u003ca href='https://github.com/gradio-app/gradio'\u003e\u003cimg src='https://img.shields.io/github/stars/gradio-app/gradio'\u003e\u003c/a\u003e\nRun the following command:\n```sh\ngit clone https://github.com/InstantStyle/InstantStyle.git\ncd ./InstantStyle/gradio_demo/\npip install -r requirements.txt\npython app.py\n```\n\n## Resources\n- [InstantStyle for WebUI](https://github.com/Mikubill/sd-webui-controlnet/discussions/2770)\n- [InstantStyle for ComfyUI](https://github.com/cubiq/ComfyUI_IPAdapter_plus)\n- [InstantID](https://github.com/InstantID/InstantID)\n\n## Disclaimer\nThe pretrained checkpoints follow the license in [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter?tab=readme-ov-file#download-models). Users are granted the freedom to create images using this tool, but they are obligated to comply with local laws and utilize it responsibly. The developers will not assume any responsibility for potential misuse by users.\n\n## Acknowledgements\nInstantStyle is developed by the InstantX team and is highly built on [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter), which has been unfairly compared by many other works. We at InstantStyle make IP-Adapter great again. Additionally, we acknowledge [Hu Ye](https://github.com/xiaohu2015) for his valuable discussion.\n\n## Star History\n[![Star History Chart](https://api.star-history.com/svg?repos=InstantStyle/InstantStyle\u0026type=Date)](https://star-history.com/#InstantStyle/InstantStyle\u0026Date)\n\n## Cite\nIf you find InstantStyle useful for your research and applications, please cite us using this BibTeX:\n\n```bibtex\n@article{wang2024instantstyle,\n  title={InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation},\n  author={Wang, Haofan and Xing, Peng and Huang, Renyuan and Ai, Hao and Wang, Qixun and Bai, Xu},\n  journal={arXiv preprint arXiv:2407.00788},\n  year={2024}\n}\n\n@article{wang2024instantstyle,\n  title={InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation},\n  author={Wang, Haofan and Wang, Qixun and Bai, Xu and Qin, Zekui and Chen, Anthony},\n  journal={arXiv preprint arXiv:2404.02733},\n  year={2024}\n}\n```\n\nFor any question, feel free to contact us via haofanwang.ai@gmail.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FinstantX-research%2FInstantStyle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FinstantX-research%2FInstantStyle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FinstantX-research%2FInstantStyle/lists"}