{"id":15029693,"url":"https://github.com/tencentarc/t2i-adapter","last_synced_at":"2025-05-14T10:11:12.989Z","repository":{"id":65934861,"uuid":"602115048","full_name":"TencentARC/T2I-Adapter","owner":"TencentARC","description":"T2I-Adapter","archived":false,"fork":false,"pushed_at":"2024-06-21T20:51:36.000Z","size":4445,"stargazers_count":3457,"open_issues_count":89,"forks_count":206,"subscribers_count":39,"default_branch":"main","last_synced_at":"2024-10-29T15:33:58.310Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TencentARC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-15T14:27:49.000Z","updated_at":"2024-10-28T23:52:47.000Z","dependencies_parsed_at":"2024-11-19T08:16:23.634Z","dependency_job_id":null,"html_url":"https://github.com/TencentARC/T2I-Adapter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FT2I-Adapter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FT2I-Adapter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FT2I-Adapter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TencentARC%2FT2I-Adapter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TencentARC","download_url":"https://codeload.github.com/TencentARC/T2I-Adapter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027400,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:11:23.045Z","updated_at":"2025-04-09T11:02:14.490Z","avatar_url":"https://github.com/TencentARC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/TencentARC/T2I-Adapter/assets/17445847/452793ae-dcf7-4c1a-b2ee-f465f8f85236\" height=120\u003e\n\u003c/p\u003e\n\n\n### \u003cdiv align=\"center\"\u003e👉 T2I-Adapter for \u003cb\u003e\u003ca href=\"https://github.com/TencentARC/T2I-Adapter/tree/SD\"\u003e[SD-1.4/1.5]\u003c/a\u003e, for \u003ca href=\"https://github.com/TencentARC/T2I-Adapter\"\u003e[SDXL]\u003c/a\u003e\u003c/b\u003e\u003c/div\u003e \n\n\u003cdiv align=\"center\"\u003e\n\n[![Huggingface T2I-Adapter-SDXL](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Gradio\u0026color=orange)](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL) \u0026ensp;  [![Blog T2I-Adapter-SDXL](https://img.shields.io/static/v1?label=Blog\u0026message=HuggingFace\u0026color=orange)](https://huggingface.co/blog/t2i-sdxl-adapters) \u0026ensp;   [![arXiv](https://img.shields.io/badge/arXiv-2302.08453-b31b1b.svg?style=flat-square)](https://arxiv.org/abs/2302.08453)\n\n\u003c/div\u003e \n\n---\n\nOfficial implementation of **[T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.08453)** based on [Stable Diffusion-XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).\n\nThe diffusers team and the T2I-Adapter authors have been collaborating to bring the support of T2I-Adapters for Stable Diffusion XL (SDXL) in diffusers! It achieves impressive results in both performance and efficiency. \n\n---\n![image](https://github.com/TencentARC/T2I-Adapter/assets/54032224/d249f699-b6d5-461d-9fdf-f0d009f14f4d)\n\n## 🚩 **New Features/Updates**\n- ✅ Sep. 8, 2023. We collaborate with the diffusers team to bring the support of T2I-Adapters for Stable Diffusion XL (SDXL) in diffusers! It achieves impressive results in both performance and efficiency. We release T2I-Adapter-SDXL models for [sketch](https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0), [canny](https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0), [lineart](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0), [openpose](https://huggingface.co/TencentARC/t2i-adapter-openpose-sdxl-1.0), [depth-zoe](https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0), and [depth-mid](https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0). We release two online demos: [![Huggingface T2I-Adapter-SDXL](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Gradio\u0026color=orange)](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL) and [![Huggingface T2I-Adapter-SDXL Doodle](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Doodly%20Demo\u0026color=orange)](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL-Sketch).\n- ✅ Aug. 21, 2023. We release [T2I-Adapter-SDXL](https://github.com/TencentARC/T2I-Adapter/), including sketch, canny, and keypoint. We still use the original recipe (77M parameters, a single inference) to drive [StableDiffusion-XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). Due to the limited computing resources, those adapters still need further improvement. We are collaborating with [HuggingFace](https://huggingface.co/), and a more powerful adapter is in the works.\n\n- ✅ Jul. 13, 2023. [Stability AI](https://stability.ai/) release [Stable Doodle](https://stability.ai/blog/clipdrop-launches-stable-doodle), a groundbreaking sketch-to-image tool based on T2I-Adapter and [SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9). It makes drawing easier.\n\n- ✅ Mar. 16, 2023. We add **CoAdapter** (**Co**mposable **Adapter**). The online Huggingface Gadio has been updated [![Huggingface Gradio (CoAdapter)](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Gradio\u0026color=orange)](https://huggingface.co/spaces/Adapter/CoAdapter). You can also try the [local gradio demo](app_coadapter.py).\n- ✅ Mar. 16, 2023. We have shrunk the git repo with [bfg](https://rtyley.github.io/bfg-repo-cleaner/).  If you encounter any issues when pulling or pushing, you can try re-cloning the repository. Sorry for the inconvenience.\n- ✅ Mar. 3, 2023. Add a [*color adapter (spatial palette)*](https://huggingface.co/TencentARC/T2I-Adapter/tree/main/models), which has only **17M parameters**.\n- ✅ Mar. 3, 2023. Add four new adapters [*style, color, openpose and canny*](https://huggingface.co/TencentARC/T2I-Adapter/tree/main/models). See more info in the **[Adapter Zoo](https://github.com/TencentARC/T2I-Adapter/blob/SD/docs/AdapterZoo.md)**.\n- ✅ Feb. 23, 2023. Add the depth adapter [*t2iadapter_depth_sd14v1.pth*](https://huggingface.co/TencentARC/T2I-Adapter/tree/main/models). See more info in the **[Adapter Zoo](https://github.com/TencentARC/T2I-Adapter/blob/SD/docs/AdapterZoo.md)**.\n- ✅ Feb. 15, 2023. Release [T2I-Adapter](https://github.com/TencentARC/T2I-Adapter/tree/SD).\n\n---\n\n# 🔥🔥🔥 Why T2I-Adapter-SDXL? \n## The Original Recipe Drives Larger SD.\n\n|   | SD-V1.4/1.5 | SD-XL | T2I-Adapter | T2I-Adapter-SDXL |\n| --- | --- |--- |--- |--- |\n| Parameters | 860M | 2.6B |77 M | 77/79 M | |\n\n## Inherit High-quality Generation from SDXL.\n\n- Lineart-guided\n\nModel from [TencentARC/t2i-adapter-lineart-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0)\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/t_lineart.PNG\" height=420\u003e\n\u003c/p\u003e\n  \n- Keypoint-guided\n\nModel from [openpose_sdxl_1.0](https://huggingface.co/Adapter/t2iadapter/tree/main/openpose_sdxl_1.0) \n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/t_pose.PNG\" height=520\u003e\n\u003c/p\u003e\n\n- Sketch-guided\n\nModel from [TencentARC/t2i-adapter-sketch-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0)\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/t_sketch.PNG\" height=520\u003e\n\u003c/p\u003e\n\n- Canny-guided\nModel from [TencentARC/t2i-adapter-canny-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0)\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/t_canny.PNG\" height=520\u003e\n\u003c/p\u003e\n\n- Depth-guided\n\nDepth guided models from [TencentARC/t2i-adapter-depth-midas-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0) and [TencentARC/t2i-adapter-depth-zoe-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0) respectively\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/t_depth.PNG\" height=520\u003e\n\u003c/p\u003e\n\n# 🔧 Dependencies and Installation\n\n- Python \u003e= 3.8 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))\n- [PyTorch \u003e= 2.0.1](https://pytorch.org/)\n```bash\npip install -r requirements.txt\n```\n\n# ⏬ Download Models \nAll models will be automatically downloaded. You can also choose to download manually from this [url](https://huggingface.co/TencentARC).\n\n# 🔥 How to Train\nHere we take sketch guidance as an example, but of course, you can also prepare your own dataset following this method.\n```bash\naccelerate launch train_sketch.py --pretrained_model_name_or_path stabilityai/stable-diffusion-xl-base-1.0 --output_dir experiments/adapter_sketch_xl --config configs/train/Adapter-XL-sketch.yaml --mixed_precision=\"fp16\" --resolution=1024 --learning_rate=1e-5 --max_train_steps=60000 --train_batch_size=1 --gradient_accumulation_steps=4 --report_to=\"wandb\" --seed=42 --num_train_epochs 100\n```\n\nWe train with `FP16` data precision on `4` NVIDIA `A100` GPUs.\n\n# 💻 How to Test\nInference requires at least `15GB` of GPU memory.\n\n## Quick start with [diffusers](https://github.com/huggingface/diffusers)\n\nTo get started, first install the required dependencies:\n\n```bash\npip install git+https://github.com/huggingface/diffusers.git@t2iadapterxl # for now\npip install -U controlnet_aux==0.0.7 # for conditioning models and detectors  \npip install transformers accelerate safetensors\n```\n\n1. Images are first downloaded into the appropriate *control image* format.\n 2. The *control image* and *prompt* are passed to the [`StableDiffusionXLAdapterPipeline`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L125).\n\nLet's have a look at a simple example using the [LineArt Adapter](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0).\n\n- Dependency\n```py\nfrom diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler, AutoencoderKL\nfrom diffusers.utils import load_image, make_image_grid\nfrom controlnet_aux.lineart import LineartDetector\nimport torch\n\n# load adapter\nadapter = T2IAdapter.from_pretrained(\n  \"TencentARC/t2i-adapter-lineart-sdxl-1.0\", torch_dtype=torch.float16, varient=\"fp16\"\n).to(\"cuda\")\n\n# load euler_a scheduler\nmodel_id = 'stabilityai/stable-diffusion-xl-base-1.0'\neuler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder=\"scheduler\")\nvae=AutoencoderKL.from_pretrained(\"madebyollin/sdxl-vae-fp16-fix\", torch_dtype=torch.float16)\npipe = StableDiffusionXLAdapterPipeline.from_pretrained(\n    model_id, vae=vae, adapter=adapter, scheduler=euler_a, torch_dtype=torch.float16, variant=\"fp16\", \n).to(\"cuda\")\npipe.enable_xformers_memory_efficient_attention()\n\nline_detector = LineartDetector.from_pretrained(\"lllyasviel/Annotators\").to(\"cuda\")\n```\n\n- Condition Image\n```py\nurl = \"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_lin.jpg\"\nimage = load_image(url)\nimage = line_detector(\n    image, detect_resolution=384, image_resolution=1024\n)\n```\n\u003ca href=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_lin.png\"\u003e\u003cimg width=\"480\" style=\"margin:0;padding:0;\" src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_lin.png\"/\u003e\u003c/a\u003e\n\n- Generation\n```py\nprompt = \"Ice dragon roar, 4k photo\"\nnegative_prompt = \"anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured\"\ngen_images = pipe(\n    prompt=prompt,\n    negative_prompt=negative_prompt,\n    image=image,\n    num_inference_steps=30,\n    adapter_conditioning_scale=0.8,\n    guidance_scale=7.5, \n).images[0]\ngen_images.save('out_lin.png')\n```\n\u003ca href=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_lin.png\"\u003e\u003cimg width=\"480\" style=\"margin:0;padding:0;\" src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_lin.png\"/\u003e\u003c/a\u003e\n\n## Online Demo [![Huggingface T2I-Adapter-SDXL](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Gradio\u0026color=orange)](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL) \n\u003ca href=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_dragon.png\"\u003e\u003cimg width=\"520\" style=\"margin:0;padding:0;\" src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_dragon.png\"/\u003e\u003c/a\u003e\n\n## Online Doodly Demo [![Huggingface T2I-Adapter-SDXL](https://img.shields.io/static/v1?label=Demo\u0026message=Huggingface%20Gradio\u0026color=orange)](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL-Sketch) \n\u003ca href=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_tree.png\"\u003e\u003cimg width=\"520\" style=\"margin:0;padding:0;\" src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_tree.png\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_doodle_dragon.png\"\u003e\u003cimg width=\"520\" style=\"margin:0;padding:0;\" src=\"https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/demo_doodle_dragon.png\"/\u003e\u003c/a\u003e\n\n\n# Tutorials on HuggingFace:\n- Sketch: [https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0)\n- Canny: [https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0)\n- Lineart: [https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0)\n- Openpose: [https://huggingface.co/TencentARC/t2i-adapter-openpose-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-openpose-sdxl-1.0)\n- Depth-mid: [https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0)\n- Depth-zoe: [https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0)\n\n...\n\n# Other Source\nJul. 13, 2023. [Stability AI](https://stability.ai/) release [Stable Doodle](https://stability.ai/blog/clipdrop-launches-stable-doodle), a groundbreaking sketch-to-image tool based on T2I-Adapter and [SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9). It makes drawing easier.\n\nhttps://user-images.githubusercontent.com/73707470/253800159-c7e12362-1ea1-4b20-a44e-bd6c8d546765.mp4\n\n# 🤗 Acknowledgements\n- Thanks to HuggingFace for their support of T2I-Adapter.\n- T2I-Adapter is co-hosted by Tencent ARC Lab and Peking University [VILLA](https://villa.jianzhang.tech/).\n\n# BibTeX\n\n    @article{mou2023t2i,\n      title={T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models},\n      author={Mou, Chong and Wang, Xintao and Xie, Liangbin and Wu, Yanze and Zhang, Jian and Qi, Zhongang and Shan, Ying and Qie, Xiaohu},\n      journal={arXiv preprint arXiv:2302.08453},\n      year={2023}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Ft2i-adapter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftencentarc%2Ft2i-adapter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftencentarc%2Ft2i-adapter/lists"}