{"id":13487796,"url":"https://github.com/Zheng-Chong/CatVTON","last_synced_at":"2025-03-27T23:31:45.114Z","repository":{"id":247878807,"uuid":"823175328","full_name":"Zheng-Chong/CatVTON","owner":"Zheng-Chong","description":"[ICLR 2025] CatVTON is a simple and efficient virtual try-on diffusion model with 1) Lightweight Network (899.06M parameters totally), 2) Parameter-Efficient Training (49.57M parameters trainable) and 3) Simplified Inference (\u003c 8G VRAM for 1024X768 resolution).","archived":false,"fork":false,"pushed_at":"2025-02-20T08:38:35.000Z","size":17133,"stargazers_count":1187,"open_issues_count":46,"forks_count":146,"subscribers_count":15,"default_branch":"edited","last_synced_at":"2025-02-20T09:34:54.268Z","etag":null,"topics":["diffusion-models","fashion","try-on"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zheng-Chong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-02T14:29:30.000Z","updated_at":"2025-02-20T08:53:46.000Z","dependencies_parsed_at":"2024-12-19T12:33:22.889Z","dependency_job_id":"b7f67958-e4fe-433c-8d96-9853e5d72638","html_url":"https://github.com/Zheng-Chong/CatVTON","commit_stats":null,"previous_names":["zheng-chong/catvton"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zheng-Chong%2FCatVTON","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zheng-Chong%2FCatVTON/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zheng-Chong%2FCatVTON/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zheng-Chong%2FCatVTON/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zheng-Chong","download_url":"https://codeload.github.com/Zheng-Chong/CatVTON/tar.gz/refs/heads/edited","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245944020,"owners_count":20697945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","fashion","try-on"],"created_at":"2024-07-31T18:01:04.028Z","updated_at":"2025-03-27T23:31:40.098Z","avatar_url":"https://github.com/Zheng-Chong.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"image\"\u003eImage\u003c/span\u003e","Personalized Restoration"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"# 🐈 CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models\n\n\u003cdiv style=\"display: flex; justify-content: center; align-items: center;\"\u003e\n  \u003ca href=\"http://arxiv.org/abs/2407.15886\" style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/arXiv-2407.15886-red?style=flat\u0026logo=arXiv\u0026logoColor=red' alt='arxiv'\u003e\n  \u003c/a\u003e\n  \u003ca href='https://huggingface.co/zhengchong/CatVTON' style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/Hugging Face-ckpts-orange?style=flat\u0026logo=HuggingFace\u0026logoColor=orange' alt='huggingface'\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/Zheng-Chong/CatVTON\" style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/GitHub-Repo-blue?style=flat\u0026logo=GitHub' alt='GitHub'\u003e\n  \u003c/a\u003e\n  \u003ca href=\"http://120.76.142.206:8888\" style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/Demo-Gradio-gold?style=flat\u0026logo=Gradio\u0026logoColor=red' alt='Demo'\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/spaces/zhengchong/CatVTON\" style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/Space-ZeroGPU-orange?style=flat\u0026logo=Gradio\u0026logoColor=red' alt='Demo'\u003e\n  \u003c/a\u003e\n  \u003ca href='https://zheng-chong.github.io/CatVTON/' style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/Webpage-Project-silver?style=flat\u0026logo=\u0026logoColor=orange' alt='webpage'\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/Zheng-Chong/CatVTON/LICENCE\" style=\"margin: 0 2px;\"\u003e\n    \u003cimg src='https://img.shields.io/badge/License-CC BY--NC--SA--4.0-lightgreen?style=flat\u0026logo=Lisence' alt='License'\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n\n**CatVTON** is a simple and efficient virtual try-on diffusion model with ***1) Lightweight Network (899.06M parameters totally)***, ***2) Parameter-Efficient Training (49.57M parameters trainable)*** and ***3) Simplified Inference (\u003c 8G VRAM for 1024X768 resolution)***.\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"resource/img/teaser.jpg\" width=\"100%\" height=\"100%\"/\u003e\n\u003c/div\u003e\n\n\n\n## Updates\n- **`2024/10/17`**:[**Mask-free version**](https://huggingface.co/zhengchong/CatVTON-MaskFree)🤗 of CatVTON is release and please try it in our [**Online Demo**](http://120.76.142.206:8888). \n- **`2024/10/13`**: We have built a repo [**Awesome-Try-On-Models**](https://github.com/Zheng-Chong/Awesome-Try-On-Models) that focuses on image, video, and 3D-based try-on models published after 2023, aiming to provide insights into the latest technological trends. If you're interested, feel free to contribute or give it a 🌟 star!\n- **`2024/08/13`**: We localize DensePose \u0026 SCHP to avoid certain environment issues.\n- **`2024/08/10`**: Our 🤗 [**HuggingFace Space**](https://huggingface.co/spaces/zhengchong/CatVTON) is available now! Thanks for the grant from [**ZeroGPU**](https://huggingface.co/zero-gpu-explorers)！\n- **`2024/08/09`**: [**Evaluation code**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#3-calculate-metrics) is provided to calculate metrics 📚.\n- **`2024/07/27`**: We provide code and workflow for deploying CatVTON on [**ComfyUI**](https://github.com/Zheng-Chong/CatVTON?tab=readme-ov-file#comfyui-workflow) 💥.\n- **`2024/07/24`**: Our [**Paper on ArXiv**](http://arxiv.org/abs/2407.15886) is available 🥳!\n- **`2024/07/22`**: Our [**App Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/app.py) is released, deploy and enjoy CatVTON on your mechine 🎉!\n- **`2024/07/21`**: Our [**Inference Code**](https://github.com/Zheng-Chong/CatVTON/blob/main/inference.py) and [**Weights** 🤗](https://huggingface.co/zhengchong/CatVTON) are released.\n- **`2024/07/11`**: Our [**Online Demo**](http://120.76.142.206:8888) is released 😁.\n\n\n\n\n## Installation\n\nCreate a conda environment \u0026 Install requirments\n```shell\nconda create -n catvton python==3.9.0\nconda activate catvton\ncd CatVTON-main  # or your path to CatVTON project dir\npip install -r requirements.txt\n```\n\n## Deployment \n### ComfyUI Workflow\nWe have modified the main code to enable easy deployment of CatVTON on [ComfyUI](https://github.com/comfyanonymous/ComfyUI). Due to the incompatibility of the code structure, we have released this part in the [Releases](https://github.com/Zheng-Chong/CatVTON/releases/tag/ComfyUI), which includes the code placed under `custom_nodes` of ComfyUI and our workflow JSON files.\n\nTo deploy CatVTON to your ComfyUI, follow these steps:\n1. Install all the requirements for both CatVTON and ComfyUI, refer to [Installation Guide for CatVTON](https://github.com/Zheng-Chong/CatVTON/blob/main/INSTALL.md) and [Installation Guide for ComfyUI](https://github.com/comfyanonymous/ComfyUI?tab=readme-ov-file#installing).\n2. Download [`ComfyUI-CatVTON.zip`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/ComfyUI-CatVTON.zip) and unzip it in the `custom_nodes` folder under your ComfyUI project (clone from [ComfyUI](https://github.com/comfyanonymous/ComfyUI)).\n3. Run the ComfyUI.\n4. Download [`catvton_workflow.json`](https://github.com/Zheng-Chong/CatVTON/releases/download/ComfyUI/catvton_workflow.json) and drag it into you ComfyUI webpage and enjoy 😆!\n\n\u003e Problems under Windows OS, please refer to [issue#8](https://github.com/Zheng-Chong/CatVTON/issues/8).\n\u003e \nWhen you run the CatVTON workflow for the first time, the weight files will be automatically downloaded, usually taking dozens of minutes.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"resource/img/comfyui-1.png\" width=\"100%\" height=\"100%\"/\u003e\n\u003c/div\u003e\n\n\u003c!-- \u003cdiv align=\"center\"\u003e\n \u003cimg src=\"resource/img/comfyui.png\" width=\"100%\" height=\"100%\"/\u003e\n\u003c/div\u003e --\u003e\n\n### Gradio App\n\nTo deploy the Gradio App for CatVTON on your machine, run the following command, and checkpoints will be automatically downloaded from HuggingFace.\n\n```PowerShell\nCUDA_VISIBLE_DEVICES=0 python app.py \\\n--output_dir=\"resource/demo/output\" \\\n--mixed_precision=\"bf16\" \\\n--allow_tf32 \n```\nWhen using `bf16` precision, generating results with a resolution of `1024x768` only requires about `8G` VRAM.\n\n## Inference\n### 1. Data Preparation\nBefore inference, you need to download the [VITON-HD](https://github.com/shadow2496/VITON-HD) or [DressCode](https://github.com/aimagelab/dress-code) dataset.\nOnce the datasets are downloaded, the folder structures should look like these:\n```\n├── VITON-HD\n|   ├── test_pairs_unpaired.txt\n│   ├── test\n|   |   ├── image\n│   │   │   ├── [000006_00.jpg | 000008_00.jpg | ...]\n│   │   ├── cloth\n│   │   │   ├── [000006_00.jpg | 000008_00.jpg | ...]\n│   │   ├── agnostic-mask\n│   │   │   ├── [000006_00_mask.png | 000008_00.png | ...]\n...\n```\n\n```\n├── DressCode\n|   ├── test_pairs_paired.txt\n|   ├── test_pairs_unpaired.txt\n│   ├── [dresses | lower_body | upper_body]\n|   |   ├── test_pairs_paired.txt\n|   |   ├── test_pairs_unpaired.txt\n│   │   ├── images\n│   │   │   ├── [013563_0.jpg | 013563_1.jpg | 013564_0.jpg | 013564_1.jpg | ...]\n│   │   ├── agnostic_masks\n│   │   │   ├── [013563_0.png| 013564_0.png | ...]\n...\n```\nFor the DressCode dataset, we provide script to preprocessed agnostic masks, run the following command:\n```PowerShell\nCUDA_VISIBLE_DEVICES=0 python preprocess_agnostic_mask.py \\\n--data_root_path \u003cyour_path_to_DressCode\u003e \n```\n\n### 2. Inference on VTIONHD/DressCode\nTo run the inference on the DressCode or VITON-HD dataset, run the following command, checkpoints will be automatically downloaded from HuggingFace.\n\n```PowerShell\nCUDA_VISIBLE_DEVICES=0 python inference.py \\\n--dataset [dresscode | vitonhd] \\\n--data_root_path \u003cpath\u003e \\\n--output_dir \u003cpath\u003e \n--dataloader_num_workers 8 \\\n--batch_size 8 \\\n--seed 555 \\\n--mixed_precision [no | fp16 | bf16] \\\n--allow_tf32 \\\n--repaint \\\n--eval_pair  \n```\n### 3. Calculate Metrics\n\nAfter obtaining the inference results, calculate the metrics using the following command: \n\n```PowerShell\nCUDA_VISIBLE_DEVICES=0 python eval.py \\\n--gt_folder \u003cyour_path_to_gt_image_folder\u003e \\\n--pred_folder \u003cyour_path_to_predicted_image_folder\u003e \\\n--paired \\\n--batch_size=16 \\\n--num_workers=16 \n```\n\n-  `--gt_folder` and `--pred_folder` should be folders that contain **only images**.\n- To evaluate the results in a paired setting, use `--paired`; for an unpaired setting, simply omit it.\n- `--batch_size` and `--num_workers` should be adjusted based on your machine.\n\n\n## Acknowledgement\nOur code is modified based on [Diffusers](https://github.com/huggingface/diffusers). We adopt [Stable Diffusion v1.5 inpainting](https://huggingface.co/runwayml/stable-diffusion-inpainting) as the base model. We use [SCHP](https://github.com/GoGoDuck912/Self-Correction-Human-Parsing/tree/master) and [DensePose](https://github.com/facebookresearch/DensePose) to automatically generate masks in our [Gradio](https://github.com/gradio-app/gradio) App and [ComfyUI](https://github.com/comfyanonymous/ComfyUI) workflow. Thanks to all the contributors!\n\n## License\nAll the materials, including code, checkpoints, and demo, are made available under the [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. You are free to copy, redistribute, remix, transform, and build upon the project for non-commercial purposes, as long as you give appropriate credit and distribute your contributions under the same license.\n\n\n## Citation\n\n```bibtex\n@misc{chong2024catvtonconcatenationneedvirtual,\n title={CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models}, \n author={Zheng Chong and Xiao Dong and Haoxiang Li and Shiyue Zhang and Wenqing Zhang and Xujie Zhang and Hanqing Zhao and Xiaodan Liang},\n year={2024},\n eprint={2407.15886},\n archivePrefix={arXiv},\n primaryClass={cs.CV},\n url={https://arxiv.org/abs/2407.15886}, \n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZheng-Chong%2FCatVTON","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZheng-Chong%2FCatVTON","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZheng-Chong%2FCatVTON/lists"}