{"id":13488746,"url":"https://github.com/ali-vilab/AnyDoor","last_synced_at":"2025-03-28T01:37:39.780Z","repository":{"id":190282927,"uuid":"667423270","full_name":"ali-vilab/AnyDoor","owner":"ali-vilab","description":"Official implementations for paper: Anydoor: zero-shot object-level image customization","archived":false,"fork":false,"pushed_at":"2024-04-08T06:31:26.000Z","size":50616,"stargazers_count":3729,"open_issues_count":46,"forks_count":348,"subscribers_count":84,"default_branch":"main","last_synced_at":"2024-05-15T23:59:48.481Z","etag":null,"topics":["image-composition","image-customization","image-editing","image-generation"],"latest_commit_sha":null,"homepage":"https://ali-vilab.github.io/AnyDoor-Page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ali-vilab.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-17T13:21:41.000Z","updated_at":"2024-06-11T05:58:02.325Z","dependencies_parsed_at":"2023-08-23T23:48:30.772Z","dependency_job_id":"9e62503b-c9ef-4b74-a272-3303f691be3b","html_url":"https://github.com/ali-vilab/AnyDoor","commit_stats":null,"previous_names":["damo-vilab/anydoor","ali-vilab/anydoor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ali-vilab%2FAnyDoor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ali-vilab%2FAnyDoor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ali-vilab%2FAnyDoor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ali-vilab%2FAnyDoor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ali-vilab","download_url":"https://codeload.github.com/ali-vilab/AnyDoor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222333976,"owners_count":16968058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-composition","image-customization","image-editing","image-generation"],"created_at":"2024-07-31T18:01:21.128Z","updated_at":"2025-03-28T01:37:39.769Z","avatar_url":"https://github.com/ali-vilab.png","language":"Python","funding_links":[],"categories":["SD-inpaint","Python"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\n  \u003ch2 align=\"center\"\u003eAnyDoor: Zero-shot Object-level Image Customization\u003c/h2\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://xavierchen34.github.io/\"\u003e\u003cstrong\u003eXi Chen\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=JYVCn3AAAAAJ\u0026hl=en\"\u003e\u003cstrong\u003eLianghua Huang\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=8zksQb4AAAAJ\u0026hl=zh-CN\"\u003e\u003cstrong\u003eYu Liu\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://shenyujun.github.io/\"\u003e\u003cstrong\u003eYujun Shen\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://scholar.google.com/citations?user=7LhjCn0AAAAJ\u0026hl=en\"\u003e\u003cstrong\u003eDeli Zhao\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://hszhao.github.io/\"\u003e\u003cstrong\u003eHengshuang Zhao\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n        \u003ca href=\"https://arxiv.org/abs/2307.09481\"\u003e\u003cimg src='https://img.shields.io/badge/arXiv-AnyDoor-red' alt='Paper PDF'\u003e\u003c/a\u003e\n        \u003ca href='https://ali-vilab.github.io/AnyDoor-Page/'\u003e\u003cimg src='https://img.shields.io/badge/Project_Page-AnyDoor-green' alt='Project Page'\u003e\u003c/a\u003e\n        \u003ca href='https://modelscope.cn/studios/damo/AnyDoor-online/summary'\u003e\u003cimg src='https://img.shields.io/badge/ModelScope-AnyDoor-yellow'\u003e\u003c/a\u003e\n        \u003ca href='https://huggingface.co/spaces/xichenhku/AnyDoor-online'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'\u003e\u003c/a\u003e\n        \u003ca href='https://replicate.com/lucataco/anydoor'\u003e\u003cimg src='https://replicate.com/lucataco/anydoor/badge'\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cb\u003eThe University of Hong Kong \u0026nbsp; | \u0026nbsp;  Alibaba Group  | \u0026nbsp;  Ant Group \u003c/b\u003e\n  \u003c/p\u003e\n  \n  \u003ctable align=\"center\"\u003e\n    \u003ctr\u003e\n    \u003ctd\u003e\n      \u003cimg src=\"assets/Figures/Teaser.png\"\u003e\n    \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\n## News\n* **[2023.12.17]** Release train \u0026 inference \u0026 demo code, and pretrained checkpoint.\n* **[2023.12.24]** 🔥 Support online demo on [ModelScope](https://modelscope.cn/studios/damo/AnyDoor-online/summary) and \n [HuggingFace](https://huggingface.co/spaces/xichenhku/AnyDoor-online).\n* **[Soon]** Release the new version paper.\n* **[On-going]** Scale-up the training data and release stronger models as the foundaition model for downstream region-to-region generation tasks.\n* **[On-going]** Release specific-designed models for downstream tasks like virtual tryon, face swap, text and logo transfer, etc.\n\n\n## Installation\nInstall with `conda`: \n```bash\nconda env create -f environment.yaml\nconda activate anydoor\n```\nor `pip`:\n```bash\npip install -r requirements.txt\n```\nAdditionally, for training, you need to install panopticapi, pycocotools, and lvis-api.\n```bash\npip install git+https://github.com/cocodataset/panopticapi.git\n\npip install pycocotools -i https://pypi.douban.com/simple\n\npip install lvis\n```\n## Download Checkpoints\nDownload AnyDoor checkpoint: \n* [ModelScope](https://modelscope.cn/models/damo/AnyDoor/files)\n* [HuggingFace](https://huggingface.co/spaces/xichenhku/AnyDoor/tree/main)\n\n**Note:** We include all the optimizer params for Adam, so the checkpoint is big. You could only keep the \"state_dict\" to make it much smaller.\n\n\nDownload DINOv2 checkpoint and revise `/configs/anydoor.yaml` for the path (line 83)\n* URL: https://github.com/facebookresearch/dinov2?tab=readme-ov-file\n\nDownload Stable Diffusion V2.1 if you want to train from scratch.\n* URL: https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main\n\n\n## Inference\nWe provide inference code in `run_inference.py` (from Line 222 - ) for both inference single image and inference a dataset (VITON-HD Test). You should modify the data path and run the following code. The generated results are provided in `examples/TestDreamBooth/GEN` for single image, and `VITONGEN` for VITON-HD Test.\n\n```bash\npython run_inference.py\n```\nThe inferenced results on VITON-Test would be like [garment, ground truth, generation].\n\n*Noticing that AnyDoor does not contain any specific design/tuning for tryon, we think it would be helpful to add skeleton infos or warped garment, and tune on tryon data to make it better :)*\n  \u003ctable align=\"center\"\u003e\n    \u003ctr\u003e\n    \u003ctd\u003e\n      \u003cimg src=\"assets/Figures/tryon.png\"\u003e\n    \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\n\nOur evaluation data for DreamBooth an COCOEE coud be downloaded at Google Drive:\n* URL: [to be released]\n\n\n\n\n\n## Gradio demo \nCurrently, we suport local gradio demo. To launch it, you should firstly modify `/configs/demo.yaml` for the path to the pretrained model, and `/configs/anydoor.yaml` for the path to DINOv2(line 83). \n\nAfterwards, run the script:\n```bash\npython run_gradio_demo.py\n```\nThe gradio demo would look like the UI shown below:\n\n* 📢 This version requires users to annotate the mask of the target object, too coarse mask would influence the generation quality. We plan to add mask refine module or interactive segmentation modules in the demo.\n\n* 📢 We provide an segmentation module to refine the user annotated reference mask. We could chose to disable it by setting  `use_interactive_seg: False` in `/configs/demo.yaml`.\n\n\u003ctable align=\"center\"\u003e\n  \u003ctr\u003e\n  \u003ctd\u003e\n    \u003cimg src=\"assets/Figures/gradio.png\"\u003e\n  \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n## Train\n\n### Prepare datasets\n* Download the datasets that present in `/configs/datasets.yaml` and modify the corresponding paths.\n* You could prepare you own datasets according to the formates of files in `./datasets`.\n* If you use UVO dataset, you need to process the json following `./datasets/Preprocess/uvo_process.py`\n* You could refer to `run_dataset_debug.py` to verify you data is correct.\n\n\n### Prepare initial weight\n* If your would like to train from scratch, convert the downloaded SD weights to control copy by running:\n```bash\nsh ./scripts/convert_weight.sh  \n```\n### Start training\n* Modify the training hyper-parameters in `run_train_anydoor.py` Line 26-34 according to your training resources. We verify that using 2-A100 GPUs with batch accumulation=1 could get satisfactory results after 300,000 iterations.\n\n\n* Start training by executing: \n```bash\nsh ./scripts/train.sh  \n```\n\n## 🔥 Community Contributions\n@bdsqlsz\n\n* AnyDoor for windows: https://github.com/sdbds/AnyDoor-for-windows\n* Pruned model: https://modelscope.cn/models/bdsqlsz/AnyDoor-Pruned/summary\n\n## Acknowledgements\nThis project is developped on the codebase of [ControlNet](https://github.com/lllyasviel/ControlNet). We  appreciate this great work! \n\n\n## Citation\nIf you find this codebase useful for your research, please use the following entry.\n```BibTeX\n@article{chen2023anydoor,\n  title={Anydoor: Zero-shot object-level image customization},\n  author={Chen, Xi and Huang, Lianghua and Liu, Yu and Shen, Yujun and Zhao, Deli and Zhao, Hengshuang},\n  journal={arXiv preprint arXiv:2307.09481},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fali-vilab%2FAnyDoor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fali-vilab%2FAnyDoor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fali-vilab%2FAnyDoor/lists"}