{"id":18949912,"url":"https://github.com/salesforce/unicontrol","last_synced_at":"2025-10-08T06:38:26.147Z","repository":{"id":169909164,"uuid":"642399315","full_name":"salesforce/UniControl","owner":"salesforce","description":"Unified Controllable Visual Generation Model","archived":false,"fork":false,"pushed_at":"2025-01-27T13:40:10.000Z","size":152196,"stargazers_count":643,"open_issues_count":19,"forks_count":35,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-05-22T14:07:44.422Z","etag":null,"topics":["aigc","generation","multi-modal"],"latest_commit_sha":null,"homepage":"https://canqin001.github.io/UniControl-Page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-18T13:33:11.000Z","updated_at":"2025-05-17T16:30:32.000Z","dependencies_parsed_at":"2025-02-22T06:10:26.714Z","dependency_job_id":null,"html_url":"https://github.com/salesforce/UniControl","commit_stats":{"total_commits":50,"total_committers":4,"mean_commits":12.5,"dds":"0.21999999999999997","last_synced_commit":"472c6cd347df2d2bccba3875d66d27946e847290"},"previous_names":["salesforce/unicontrol"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/salesforce/UniControl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FUniControl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FUniControl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FUniControl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FUniControl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce","download_url":"https://codeload.github.com/salesforce/UniControl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2FUniControl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278903014,"owners_count":26065785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","generation","multi-modal"],"created_at":"2024-11-08T13:19:34.417Z","updated_at":"2025-10-08T06:38:26.126Z","avatar_url":"https://github.com/salesforce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [UniControl](https://arxiv.org/abs/2305.11147) [![arXiv](https://img.shields.io/badge/📃-arXiv-ff69b4)](https://arxiv.org/pdf/2305.11147.pdf) [![webpage](https://img.shields.io/badge/🖥-Website-9cf)](https://canqin001.github.io/UniControl-Page/) [![HuggingFace space](https://img.shields.io/badge/🤗-Huggingface%20Space-cyan.svg)](https://huggingface.co/spaces/Robert001/UniControl-Demo)\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/salesforce.png\"  height=\"100px\" \u003e\u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"figs/northeastern.png\"  height=\"100px\" \u003e\u003c/a\u003e\n    \u003ca\u003e\u003cimg src=\"figs/stanford.png\"  height=\"100px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nThis repository is for the paper:\n\u003e **[UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild](https://arxiv.org/abs/2305.11147)** \\\n\u003e Can Qin \u003csup\u003e1,2\u003c/sup\u003e, Shu Zhang\u003csup\u003e1\u003c/sup\u003e, Ning Yu \u003csup\u003e1\u003c/sup\u003e, Yihao Feng\u003csup\u003e1\u003c/sup\u003e, Xinyi Yang\u003csup\u003e1\u003c/sup\u003e, Yingbo Zhou \u003csup\u003e1\u003c/sup\u003e, Huan Wang \u003csup\u003e1\u003c/sup\u003e, Juan Carlos Niebles\u003csup\u003e1\u003c/sup\u003e, Caiming Xiong \u003csup\u003e1\u003c/sup\u003e, Silvio Savarese \u003csup\u003e1\u003c/sup\u003e, Stefano Ermon \u003csup\u003e3\u003c/sup\u003e, Yun Fu \u003csup\u003e2\u003c/sup\u003e,  Ran Xu \u003csup\u003e1\u003c/sup\u003e \\\n\u003e \u003csup\u003e1\u003c/sup\u003e Salesforce AI \u003csup\u003e2\u003c/sup\u003e Northeastern University  \u003csup\u003e3\u003c/sup\u003e Stanford University \\\n\u003e Work done when Can Qin was an intern at Salesforce AI Research.\n\n\n![img](figs/demo_simple.png)\n\n\n\n## Introduction\nWe introduce **UniControl**, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation. \n\n\n![img](figs/method.png)\n\n\n\n\n## Updates\n* **05/18/23**: ***[UniControl](https://arxiv.org/abs/2305.11147) paper uploaded to arXiv.***\n* **05/26/23**: ***UniControl inference code and checkpoint open to public.*** \n* **05/28/23**: ***Latest UniControl model [checkpoint](https://console.cloud.google.com/storage/browser/_details/sfr-unicontrol-data-research/unicontrol.ckpt) (1.4B #params, 5.78GB) updated.*** \n* **06/08/23**: ***Latest UniControl model [checkpoint](https://console.cloud.google.com/storage/browser/_details/sfr-unicontrol-data-research/unicontrol.ckpt) updated which supports 12 tasks now (***Canny***, ***HED***, ***Sketch***, ***Depth***, ***Normal***, ***Skeleton***, ***Bbox***, ***Seg***, ***Outpainting***, ***Inpainting***, ***Deblurring*** and ***Colorization***) !*** \n* **06/08/23**: ***Training dataset ([MultiGen-20M](https://console.cloud.google.com/storage/browser/sfr-unicontrol-data-research/dataset)) is fully released.*** \n* **06/08/23**: ***Training code is public.***:blush:\n* **07/06/23**: ***Latest UniControl model v1.1 [checkpoint](https://console.cloud.google.com/storage/browser/_details/sfr-unicontrol-data-research/unicontrol_v1.1.ckpt) updated which supports 12 tasks now (***Canny***, ***HED***, ***Sketch***, ***Depth***, ***Normal***, ***Skeleton***, ***Bbox***, ***Seg***, ***Outpainting***, ***Inpainting***, ***Deblurring*** and ***Colorization***) !***\n* **07/25/23**: ***Huggingface Demo API is available! [![HuggingFace space](https://img.shields.io/badge/🤗-Huggingface%20Space-cyan.svg)](https://huggingface.co/spaces/Robert001/UniControl-Demo)***\n* **07/25/23**: ***Safetensors model is available! [checkpoint](https://storage.googleapis.com/sfr-unicontrol-data-research/unicontrol_v1.1.st)***\n* **09/21/23**: ***UniControl is accepted to NeurIPS 2023.***:blush:\n\n\n## MultiGen-20M Datasets\nThere are more than 20M image-prompt-condition triplets [here](https://console.cloud.google.com/storage/browser/sfr-unicontrol-data-research/dataset) with total size ***\u003e 2TB***. It includes all 12 tasks (`Canny, HED, Sketch, Depth, Normal, Skeleton, Bbox, Seg, Outpainting, Inpainting, Deblurring, Colorization`) which are fully released.\n\n## Instruction\n### Environment Preparation\nSetup the env first (need to wait a few minutes).\n```\nconda env create -f environment.yaml\nconda activate unicontrol\n```\n### Checkpoint Preparation\nThe checkpoint of pre-trained UniControl model is saved at `./ckpts/unicontrol.ckpt`.\n```\ncd ckpts\nwget https://storage.googleapis.com/sfr-unicontrol-data-research/unicontrol.ckpt \n```\n\nYou can also use the latest trained model (ckpt and safetensors)\n```\nwget https://storage.googleapis.com/sfr-unicontrol-data-research/unicontrol_v1.1.ckpt\nwget https://storage.googleapis.com/sfr-unicontrol-data-research/unicontrol_v1.1.st\n```\n\nIf you want to train from scratch, please follow the ControlNet to prepare the checkpoint initialization. ControlNet provides a simple script for you to achieve this easily. If your SD filename is `./ckpts/v1-5-pruned.ckpt` and you want the script to save the processed model (SD+ControlNet) at location `./ckpts/control_sd15_ini.ckpt`, you can just run:\n```\npython tool_add_control.py ./ckpts/v1-5-pruned.ckpt ./ckpts/control_sd15_ini.ckpt\n```\n\n### Data Preparation \nPlease download the training dataset ([MultiGen-20M](https://console.cloud.google.com/storage/browser/sfr-unicontrol-data-research/dataset)) to `./multigen20m`. Please:\n```\ncd multigen20m\ngsutil -m cp -r gs://sfr-unicontrol-data-research/dataset ./\n```\nThen unzip the all the files.\n\n### Model Training (CUDA 11.0 and Conda 4.12.0 work)\nTraining from Scratch:\n``` \npython train_unicontrol.py --ckpt ./ckpts/control_sd15_ini.ckpt --config ./models/cldm_v15_unicontrol_v11.yaml --lr 1e-5\n```\n\nModel Finetuning:\n``` \npython train_unicontrol.py --ckpt ./ckpts/unicontrol.ckpt  --config ./models/cldm_v15_unicontrol.yaml --lr 1e-7\n```\n\n### Model Inference (CUDA 11.0 and Conda 4.12.0 work)\nFor different tasks, please run the code as follows. If you meet OOM error, please decrease the \"--num_samples\".\n\nIf you use safetensors model, you can load the model following ./load_model/load_safetensors_model.py\n\nCanny to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task canny\n```\n\nHED Edge to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task hed \n```\n\nHED-like Skech to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task hedsketch\n```\n\nDepth Map to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task depth \n```\n\nNormal Surface Map to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task normal\n```\n\nSegmentation Map to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task seg\n```\n\n\nHuman Skeleton to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task openpose\n```\n\n\nObject Bounding Boxes to Image Generation:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task bbox\n```\n\n\nImage Outpainting:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task outpainting\n```\n\n\nImage Inpainting:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task inpainting\n```\n\n\nImage Deblurring:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task blur\n```\n\n\nImage Colorization:\n```\npython inference_demo.py --ckpt ./ckpts/unicontrol.ckpt --task grayscale\n```\n\n### Gradio Demo ([App Demo Video](https://github.com/salesforce/UniControl/issues/1), CUDA 11.0 and Conda 4.12.0 work)\nWe have provided gradio demos for different tasks to use. The example images are saved at `./test_imgs`. \n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_all_tasks.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nFor all the tasks (`Canny, HED, Sketch, Depth, Normal, Human Pose, Seg, Bbox, Outpainting, Colorization, Deblurring, Inpainting`) please run the following code: \n```\npython app/gradio_all_tasks.py\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_all_task_sketch.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nWe support the direct condition-to-image generation (as shown above). Please unmark the `Condition Extraction` in UI if you want to upload condition image directly. \n\n***\n\nOr, we provide the task-specifc gradio demos:\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_canny.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nCanny to Image Generation:\n```\npython app/gradio_canny2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_hed.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nHED Edge to Image Generation:\n```\npython app/gradio_hed2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_hedsketch.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nHED-like Skech to Image Generation:\n```\npython app/gradio_hedsketch2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_depth.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nDepth Map to Image Generation:\n```\npython app/gradio_depth2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_normal.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nNormal Surface Map to Image Generation:\n```\npython app/gradio_normal2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_seg.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nSegmentation Map to Image Generation:\n```\npython app/gradio_seg2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_pose.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nHuman Skeleton to Image Generation:\n```\npython app/gradio_pose2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_bbox.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nObject Bounding Boxes to Image Generation:\n```\npython app/gradio_bbox2image.py\n```\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_outpainting.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nImage Outpainting:\n```\npython app/gradio_outpainting.py\n```\n\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_colorization.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nImage Colorization:\n```\npython app/gradio_colorization.py\n```\n\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_deblur.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nImage Deblurring:\n```\npython app/gradio_deblur.py\n```\n\n\n***\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/gradio_inpainting.png\"  height=\"450px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nImage Inpainting:\n```\npython app/gradio_inpainting.py\n```\n\n\n## To Do\n- [x] Data Preparation\n- [x] Pre-training Tasks Inference\n- [x] Gradio Demo\n- [x] Model Training\n- [x] HF Space\n- [ ] Colab\n\n## Tips\n* Negative prompts are very useful sometimes: `monochrome, lowres, bad anatomy, worst quality, low quality` are example negative prompts.\n\n* UniControl can work well on some tasks (ie, Colorization and Deblurring) without ANY text prompts.\n\n* If OOM, let `--num_samples 1` may be helpful\n\n## Citation\nIf you find this project useful for your research, please kindly cite our paper:\n\n```bibtex\n@article{qin2023unicontrol,\n  title={UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild},\n  author={Qin, Can and Zhang, Shu and Yu, Ning and Feng, Yihao and Yang, Xinyi and Zhou, Yingbo and Wang, Huan and Niebles, Juan Carlos and Xiong, Caiming and Savarese, Silvio and others},\n  journal={arXiv preprint arXiv:2305.11147},\n  year={2023}\n}\n```\n\n## Acknowledgement\n*This project is built upon the gaint sholders of ControlNet and Stable Diffusion. Great thanks to them!*\n\nStable Diffusion https://github.com/CompVis/stable-diffusion\n\nControlNet https://github.com/lllyasviel/ControlNet \n\nStyleGAN3 https://github.com/NVlabs/stylegan3\n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Funicontrol","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce%2Funicontrol","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Funicontrol/lists"}