{"id":13487793,"url":"https://github.com/miccunifi/ladi-vton","last_synced_at":"2026-02-01T11:13:06.953Z","repository":{"id":166811274,"uuid":"641932376","full_name":"miccunifi/ladi-vton","owner":"miccunifi","description":"[ACM MM 2023] - LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On","archived":false,"fork":false,"pushed_at":"2023-09-05T08:16:33.000Z","size":1607,"stargazers_count":438,"open_issues_count":27,"forks_count":60,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-03-27T23:33:01.387Z","etag":null,"topics":["acmmm","acmmm2023","dresscode","fashionai","generative-model","latent-diffusion-models","stable-diffusion","textual-inversion","virtual-tryon","viton-hd"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miccunifi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-17T13:16:21.000Z","updated_at":"2025-03-18T05:34:48.000Z","dependencies_parsed_at":"2024-10-30T23:31:08.918Z","dependency_job_id":"39aeec94-d09f-42ce-9d93-1738bdf3f755","html_url":"https://github.com/miccunifi/ladi-vton","commit_stats":null,"previous_names":["miccunifi/ladi-vton"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/miccunifi/ladi-vton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miccunifi%2Fladi-vton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miccunifi%2Fladi-vton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miccunifi%2Fladi-vton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miccunifi%2Fladi-vton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miccunifi","download_url":"https://codeload.github.com/miccunifi/ladi-vton/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miccunifi%2Fladi-vton/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28977317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T09:57:52.632Z","status":"ssl_error","status_checked_at":"2026-02-01T09:57:49.143Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acmmm","acmmm2023","dresscode","fashionai","generative-model","latent-diffusion-models","stable-diffusion","textual-inversion","virtual-tryon","viton-hd"],"created_at":"2024-07-31T18:01:03.958Z","updated_at":"2026-02-01T11:13:06.893Z","avatar_url":"https://github.com/miccunifi.png","language":"Python","funding_links":[],"categories":["Personalized Restoration"],"sub_categories":[],"readme":"# LaDI-VTON (ACM Multimedia 2023)\n\n### Latent Diffusion Textual-Inversion Enhanced Virtual Try-On\n\n[**Davide Morelli**](https://scholar.google.com/citations?user=UJ4D3rYAAAAJ\u0026hl=en)**\\***,\n[**Alberto Baldrati**](https://scholar.google.com/citations?hl=en\u0026user=I1jaZecAAAAJ)**\\***,\n[**Giuseppe Cartella**](https://scholar.google.com/citations?hl=en\u0026user=0sJ4VCcAAAAJ),\n[**Marcella Cornia**](https://scholar.google.com/citations?hl=en\u0026user=DzgmSJEAAAAJ),\n[**Marco Bertini**](https://scholar.google.com/citations?user=SBm9ZpYAAAAJ\u0026hl=en),\n[**Rita Cucchiara**](https://scholar.google.com/citations?hl=en\u0026user=OM3sZEoAAAAJ)\n\n**\\*** Equal contribution.\n\n[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2305.13501)\n[![GitHub Stars](https://img.shields.io/github/stars/miccunifi/ladi-vton?style=social)](https://github.com/miccunifi/ladi-vton)\n\n🔥🔥 **[05/09/2023] Release of the training code**\n\nThis is the **official repository** for the [**paper**](https://arxiv.org/abs/2305.13501) \"*LaDI-VTON: Latent Diffusion\nTextual-Inversion Enhanced Virtual Try-On*\".\n\n## Overview\n\n![](assets/teaser.png \"Overview of our approach\")\n\n\n\u003e **Abstract**: \u003cbr\u003e\n\u003e The rapidly evolving fields of e-commerce and metaverse continue to seek innovative approaches to enhance the consumer\n\u003e experience. At the same time, recent advancements in the development of diffusion models have enabled generative\n\u003e networks to create remarkably realistic images. In this context, image-based virtual try-on, which consists in\n\u003e generating a novel image of a target model wearing a given in-shop garment, has yet to capitalize on the potential of\n\u003e these powerful generative solutions. This work introduces LaDI-VTON, the first Latent Diffusion textual\n\u003e Inversion-enhanced model for the Virtual Try-ON task. The proposed architecture relies on a latent diffusion model\n\u003e extended with a novel additional autoencoder module that exploits learnable skip connections to enhance the generation\n\u003e process preserving the model's characteristics. To effectively maintain the texture and details of the in-shop\n\u003e garment,\n\u003e we propose a textual inversion component that can map the visual features of the garment to the CLIP token embedding\n\u003e space and thus generate a set of pseudo-word token embeddings capable of conditioning the generation process.\n\u003e Experimental results on Dress Code and VITON-HD datasets demonstrate that our approach outperforms the competitors by\n\u003e a consistent margin, achieving a significant milestone for the task\n\n## Citation\n\nIf you make use of our work, please cite our paper:\n\n```bibtex\n@inproceedings{morelli2023ladi,\n  title={{LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-On}},\n  author={Morelli, Davide and Baldrati, Alberto and Cartella, Giuseppe and Cornia, Marcella and Bertini, Marco and Cucchiara, Rita},\n  booktitle={Proceedings of the ACM International Conference on Multimedia},\n  year={2023}\n}\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ch2\u003eGetting Started\u003c/h2\u003e\u003c/summary\u003e\n\nWe recommend using the [**Anaconda**](https://www.anaconda.com/) package manager to avoid dependency/reproducibility\nproblems.\nFor Linux systems, you can find a conda installation\nguide [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html).\n\n### Installation\n\n1. Clone the repository\n\n```sh\ngit clone https://github.com/miccunifi/ladi-vton\n```\n\n2. Install Python dependencies\n\n```sh\nconda env create -n ladi-vton -f environment.yml\nconda activate ladi-vton\n```\n\nAlternatively, you can create a new conda environment and install the required packages manually:\n\n```sh\nconda create -n ladi-vton -y python=3.10\nconda activate ladi-vton\npip install torch==2.0.1 torchvision==0.15.2 opencv-python==4.7.0.72 diffusers==0.14.0 transformers==4.27.3 accelerate==0.18.0 clean-fid==0.1.35 torchmetrics[image]==0.11.4 wandb==0.14.0 matplotlib==3.7.1 tqdm xformers\n```\n\n### Data Preparation\n\n#### DressCode\n\n1. Download the [DressCode](https://github.com/aimagelab/dress-code) dataset\n2. To enhance the performance of our warping module, we have discovered that using in-shop images with a white\n   background\n   yields better results. To facilitate this process, we now offer pre-extracted masks that can be used to remove the\n   background from the images. You can download the masks from the following\n   link: [here](https://github.com/miccunifi/ladi-vton/releases/download/weights/dresscode_masks.zip). Once downloaded,\n   please\n   extract the mask files and place them in the dataset folder alongside the corresponding images.\n\nOnce the dataset is downloaded, the folder structure should look like this:\n\n```\n├── DressCode\n|   ├── test_pairs_paired.txt\n|   ├── test_pairs_unpaired.txt\n|   ├── train_pairs.txt\n│   ├── [dresses | lower_body | upper_body]\n|   |   ├── test_pairs_paired.txt\n|   |   ├── test_pairs_unpaired.txt\n|   |   ├── train_pairs.txt\n│   │   ├── images\n│   │   │   ├── [013563_0.jpg | 013563_1.jpg | 013564_0.jpg | 013564_1.jpg | ...]\n│   │   ├── masks\n│   │   │   ├── [013563_1.png| 013564_1.png | ...]\n│   │   ├── keypoints\n│   │   │   ├── [013563_2.json | 013564_2.json | ...]\n│   │   ├── label_maps\n│   │   │   ├── [013563_4.png | 013564_4.png | ...]\n│   │   ├── skeletons\n│   │   │   ├── [013563_5.jpg | 013564_5.jpg | ...]\n│   │   ├── dense\n│   │   │   ├── [013563_5.png | 013563_5_uv.npz | 013564_5.png | 013564_5_uv.npz | ...]\n```\n\n#### VITON-HD\n\n1. Download the [VITON-HD](https://github.com/shadow2496/VITON-HD) dataset\n\nOnce the dataset is downloaded, the folder structure should look like this:\n\n```\n├── VITON-HD\n|   ├── test_pairs.txt\n|   ├── train_pairs.txt\n│   ├── [train | test]\n|   |   ├── image\n│   │   │   ├── [000006_00.jpg | 000008_00.jpg | ...]\n│   │   ├── cloth\n│   │   │   ├── [000006_00.jpg | 000008_00.jpg | ...]\n│   │   ├── cloth-mask\n│   │   │   ├── [000006_00.jpg | 000008_00.jpg | ...]\n│   │   ├── image-parse-v3\n│   │   │   ├── [000006_00.png | 000008_00.png | ...]\n│   │   ├── openpose_img\n│   │   │   ├── [000006_00_rendered.png | 000008_00_rendered.png | ...]\n│   │   ├── openpose_json\n│   │   │   ├── [000006_00_keypoints.json | 000008_00_keypoints.json | ...]\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ch2\u003eInference with Pre-trained Models\u003c/h2\u003e\u003c/summary\u003e\n\nTo run the inference on the Dress Code or VITON-HD dataset, run the following command:\n\n```sh\npython src/inference.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --output_dir \u003cpath\u003e --test_order [paired | unpaired] --category [all | lower_body | upper_body | dresses ] --mixed_precision [no | fp16 | bf16] --enable_xformers_memory_efficient_attention --use_png --compute_metrics\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired']\n    --category \u003cstr\u003e               category to test, options: ['all', 'lower_body', 'upper_body', 'dresses'] (default=all)\n    --output_dir \u003cstr\u003e             output directory\n    --batch_size \u003cint\u003e             batch size (default=8)\n    --mixed_precision \u003cstr\u003e        mixed precision (no, fp16, bf16) (default=no)\n    --enable_xformers_memory_efficient_attention \u003cstore_true\u003e\n                                   enable memory efficient attention in xformers (default=False)\n    --allow_tf32 \u003cstore_true\u003e      allow TF32 on Ampere GPUs (default=False)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n    --use_png \u003cstore_true\u003e         use png instead of jpg (default=False)\n    --compute_metrics              compute metrics at the end of inference (default=False)\n```\n\nSince we release the pre-trained models via torch.hub, the models will be automatically downloaded when running the\ninference script.\n\n### Metrics computation\n\nOnce you have run the inference script and extracted the images, you can compute the metrics by running the following\ncommand:\n\n```sh\npython src/utils/val_metrics.py --gen_folder \u003cpath\u003e --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --test_order [paired | unpaired] --category [all | lower_body | upper_body | dresses ]\n```\n\n```\n    --gen_folder \u003cstr\u003e             Path to the generated images folder.\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired']\n    --category \u003cstr\u003e               category to test, options: ['all', 'lower_body', 'upper_body', 'dresses'] (default=all)\n    --batch_size                   batch size (default=32)\n    --workers                      number of workers (default=8)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ch2\u003eTraining\u003c/h2\u003e\u003c/summary\u003e\n\nIn this section, you'll find instructions on how to train all the components of our model from scratch.\n\n### 1. Train Warping Module\n\nFirst of all, we need to train the warping module. To do so, run the following command:\n\n```sh\npython src/train_tps.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --checkpoints_dir \u003cpath\u003e --exp_name \u003cstr\u003e\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     dataroot of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       dataroot of vitonhd dataset (required when dataset=vitonhd)\n    --checkpoints_dir \u003cstr\u003e        checkpoints directory\n    --exp_name \u003cstr\u003e               experiment name\n    --batch_size \u003cint\u003e             batch size (default=16)\n    --workers \u003cint\u003e                number of workers (default=10)\n    --height \u003cint\u003e                 height of the input images (default=512)\n    --width \u003cint\u003e                  width of the input images (default=384)\n    --lr \u003cfloat\u003e                   learning rate (default=1e-4)\n    --const_weight \u003cfloat\u003e         weight for the TPS constraint loss (default=0.01)\n    --wandb_log \u003cstore_true\u003e       log training on wandb (default=False)\n    --wandb_project \u003cstr\u003e          wandb project name (default=LaDI_VTON_tps)\n    --dense \u003cstore_true\u003e           use dense uv map instead of keypoints (default=False)\n    --only_extraction \u003cstore_true\u003e only extract the images using the trained networks without training (default=False)\n    --vgg_weight \u003cint\u003e             weight for the VGG loss (refinement network) (default=0.25)\n    --l1_weight \u003cint\u003e              weight for the L1 loss (refinement network) (default=1.0)\n    --epochs_tps \u003cint\u003e             number of epochs for the TPS training (default=50)\n    --epochs_refinement \u003cint\u003e      number of epochs for the refinement network training (default=50)\n```\n\nAt the end of the training, the warped cloth images will be saved in the `data/warped_cloths`\nand `data/warped_cloths_unpaired` folders.\nTo save computation time, in the following steps, we will use the pre-extracted warped cloth images.\n\n### 2. Train EMASC\n\nTo train the EMASC module, run the following command:\n\n```sh\npython src/train_emasc.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --output_dir \u003cpath\u003e\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --output_dir \u003cstr\u003e             output directory where the model predictions and checkpoints will be written\n    --pretrained_model_name_or_path \u003cstr\u003e\n                                   model identifier from huggingface.co/models (default=stabilityai/stable-diffusion-2-inpainting)\n    --seed \u003cint\u003e                   seed for reproducible training (default=1234)\n    --train_batch_size \u003cint\u003e       batch size for training (default=16)\n    --test_batch_size \u003cint\u003e        batch size for testing (default=16)\n    --num_train_epochs \u003cint\u003e       number of training epochs (default=100)\n    --max_train_steps \u003cint\u003e        maximum number of training steps. If provided, overrides num_train_epochs (default=40k)\n    --gradient_accumulation_steps \u003cint\u003e\n                                   number of update steps to accumulate before performing a backward/update pass (default=1)\n    --learning_rate \u003cfloat\u003e        learning rate (default=1e-5)\n    --lr_scheduler \u003cstr\u003e           learning rate scheduler, options: ['linear', 'cosine', 'cosine_with_restarts', 'polynomial', 'constant', 'constant_with_warmup'] (default=constant_with_warmup)\n    --lr_warmup_steps \u003cint\u003e        number of warmup steps for learning rate scheduler (default=500)\n    --allow_tf32 \u003cstore_true\u003e      allow TF32 on Ampere GPUs (default=False)\n    --adam_beta1 \u003cfloat\u003e           value of beta_1 for Adam optimizer (default=0.9)\n    --adam_beta2 \u003cfloat\u003e           value of beta_2 for Adam optimizer (default=0.999)\n    --adam_weight_decay \u003cfloat\u003e    value of weight decay for Adam optimizer (default=1e-2)\n    --adam_epsilon \u003cfloat\u003e         value of epsilon for Adam optimizer (default=1e-8)\n    --max_grad_norm \u003cfloat\u003e        maximum value of gradient norm for gradient clipping (default=1.0)\n    --mixed_precision \u003cstr\u003e        mixed precision training, options: ['no', 'fp16', 'bf16'] (default=fp16)\n    --report_to \u003cstr\u003e              where to report metrics, options: ['wandb', 'tensorboard', 'comet_ml'] (default=wandb)\n    --checkpointing_steps \u003cint\u003e    number of steps between each checkpoint (default=10000)\n    --resume_from_checkpoint \u003cstr\u003e whether training should be resumed from a previous checkpoint. Use a \"latest\" to automatically select the last available checkpoint. (default=None)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n    --num_workers_test \u003cint\u003e       number of workers for test dataloader (default=8)\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired'] (default=paired)\n    --emasc_type \u003cstr\u003e             type of EMASC, options: ['linear', 'nonlinear'] (default=nonlinear)\n    --vgg_weight \u003cfloat\u003e           weight for the VGG loss (default=0.5)\n    --emasc_kernel \u003cint\u003e           kernel size for the EMASC module (default=3)\n    --emasc_padding \u003cint\u003e          padding for the EMASC module (default=1)\n```\n\nAt the end of the training, the EMASC checkpoints will be saved in the `output_dir` folder.\n\n### 2.5 (Optional) Extract clip cloth embeddings\n\nTo accelerate the training process for subsequent steps, consider pre-computing the CLIP cloth embeddings for each image\nin the dataset.\n\nTo do so, run the following command:\n\n```bash\npython src/utils/compute_cloth_clip_features.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --pretrained_model_name_or_path \u003cstr\u003e\n                                   model identifier from huggingface.co/models (default=stabilityai/stable-diffusion-2-inpainting)\n    --batch_size \u003cint\u003e             batch size (default=16)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n```\n\nThe computed features will be saved in the `data/clip_cloth_embeddings` folder.\n\nIn the following steps, to use the pre-computed features, make sure to use the `--use_clip_cloth_features` flag.\n\n### 3. Pre-train the inversion adapter\n\nTo pre-train the inversion adapter, run the following command:\n\n```sh\npython src/train_inversion_adapter.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --output_dir \u003cpath\u003e --gradient_checkpointing --enable_xformers_memory_efficient_attention --use_clip_cloth_features\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --output_dir \u003cstr\u003e             output directory where the model predictions and checkpoints will be written\n    --pretrained_model_name_or_path \u003cstr\u003e\n                                   model identifier from huggingface.co/models (default=stabilityai/stable-diffusion-2-inpainting)\n    --seed \u003cint\u003e                   seed for reproducible training (default=1234)\n    --train_batch_size \u003cint\u003e       batch size for training (default=16)\n    --test_batch_size \u003cint\u003e        batch size for testing (default=16)\n    --num_train_epochs \u003cint\u003e       number of training epochs (default=100)\n    --max_train_steps \u003cint\u003e        maximum number of training steps. If provided, overrides num_train_epochs (default=200k)\n    --gradient_accumulation_steps \u003cint\u003e\n                                   number of update steps to accumulate before performing a backward/update pass (default=1)\n    --gradient_checkpointing \u003cstore_true\u003e\n                                   use gradient checkpointing to save memory at the expense of slower backward pass (default=False)\n    --learning_rate \u003cfloat\u003e        learning rate (default=1e-5)\n    --lr_scheduler \u003cstr\u003e           learning rate scheduler, options: ['linear', 'cosine', 'cosine_with_restarts', 'polynomial', 'constant', 'constant_with_warmup'] (default=constant_with_warmup)\n    --lr_warmup_steps \u003cint\u003e        number of warmup steps for learning rate scheduler (default=500)\n    --allow_tf32 \u003cstore_true\u003e      allow TF32 on Ampere GPUs (default=False)\n    --adam_beta1 \u003cfloat\u003e           value of beta_1 for Adam optimizer (default=0.9)\n    --adam_beta2 \u003cfloat\u003e           value of beta_2 for Adam optimizer (default=0.999)\n    --adam_weight_decay \u003cfloat\u003e    value of weight decay for Adam optimizer (default=1e-2)\n    --adam_epsilon \u003cfloat\u003e         value of epsilon for Adam optimizer (default=1e-8)\n    --max_grad_norm \u003cfloat\u003e        maximum value of gradient norm for gradient clipping (default=1.0)\n    --mixed_precision \u003cstr\u003e        mixed precision training, options: ['no', 'fp16', 'bf16'] (default=fp16)\n    --report_to \u003cstr\u003e              where to report metrics, options: ['wandb', 'tensorboard', 'comet_ml'] (default=wandb)\n    --checkpointing_steps \u003cint\u003e    number of steps between each checkpoint (default=50000)\n    --resume_from_checkpoint \u003cstr\u003e whether training should be resumed from a previous checkpoint. Use a \"latest\" to automatically select the last available checkpoint. (default=None)\n    --enable_xformers_memory_efficient_attention \u003cstore_true\u003e\n                                   enable memory efficient attention in xformers (default=False)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n    --num_workers_test \u003cint\u003e       number of workers for test dataloader (default=8)\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired'] (default=paired)\n    --num_vstar \u003cint\u003e              number of predicted v* per image to use (default=16)\n    --num_encoder_layers \u003cint\u003e     number of ViT layers to use in inversion adapter (default=1)\n    --use_clip_cloth_features \u003cstore_true\u003e\n                                   use precomputed clip cloth features instead of computing them each iteration (default=False).\n```\n\nAt the end of the training, the inversion adapter checkpoints will be saved in the `output_dir` folder.\n\n**NOTE**: You can use the `--use_clip_cloth_features` flag only if you have previously computed the clip cloth features\nusing the `src/utils/compute_cloth_clip_features.py` script (step 2.5).\n\n### 4. Train VTO\n\nTo successfully train the VTO model, ensure that you specify the correct path to the pre-trained inversion adapter\ncheckpoint. If omitted, the inversion adapter will be trained from scratch. Additionally, don't forget to include the\n`--train_inversion_adapter` flag to enable the inversion adapter training during the VTO training process.\n\nTo train the VTO model, run the following command:\n\n```sh\npython src/train_vto.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --output_dir \u003cpath\u003e --inversion_adapter_dir \u003cpath\u003e --gradient_checkpointing --enable_xformers_memory_efficient_attention --use_clip_cloth_features --train_inversion_adapter\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --output_dir \u003cstr\u003e             output directory where the model predictions and checkpoints will be written\n    --inversion_adapter_dir \u003cstr\u003e  path to the inversion adapter checkpoint directory. Should be the same as `output_dir` of the inversion adapter training script. If not specified, the inversion adapter will be trained from scratch. (default=None)\n    --inversion_adapter_name \u003cstr\u003e name of the inversion adapter checkpoint. To load the latest checkpoint, use `latest`. (default=latest)\n     --pretrained_model_name_or_path \u003cstr\u003e\n                                   model identifier from huggingface.co/models (default=stabilityai/stable-diffusion-2-inpainting)\n    --seed \u003cint\u003e                   seed for reproducible training (default=1234)\n    --train_batch_size \u003cint\u003e       batch size for training (default=16)\n    --test_batch_size \u003cint\u003e        batch size for testing (default=16)\n    --num_train_epochs \u003cint\u003e       number of training epochs (default=100)\n    --max_train_steps \u003cint\u003e        maximum number of training steps. If provided, overrides num_train_epochs (default=200k)\n    --gradient_accumulation_steps \u003cint\u003e\n                                   number of update steps to accumulate before performing a backward/update pass (default=1)\n    --gradient_checkpointing \u003cstore_true\u003e\n                                   use gradient checkpointing to save memory at the expense of slower backward pass (default=False)\n    --learning_rate \u003cfloat\u003e        learning rate (default=1e-5)\n    --lr_scheduler \u003cstr\u003e           learning rate scheduler, options: ['linear', 'cosine', 'cosine_with_restarts', 'polynomial', 'constant', 'constant_with_warmup'] (default=constant_with_warmup)\n    --lr_warmup_steps \u003cint\u003e        number of warmup steps for learning rate scheduler (default=500)\n    --allow_tf32 \u003cstore_true\u003e      allow TF32 on Ampere GPUs (default=False)\n    --adam_beta1 \u003cfloat\u003e           value of beta_1 for Adam optimizer (default=0.9)\n    --adam_beta2 \u003cfloat\u003e           value of beta_2 for Adam optimizer (default=0.999)\n    --adam_weight_decay \u003cfloat\u003e    value of weight decay for Adam optimizer (default=1e-2)\n    --adam_epsilon \u003cfloat\u003e         value of epsilon for Adam optimizer (default=1e-8)\n    --max_grad_norm \u003cfloat\u003e        maximum value of gradient norm for gradient clipping (default=1.0)\n    --mixed_precision \u003cstr\u003e        mixed precision training, options: ['no', 'fp16', 'bf16'] (default=fp16)\n    --report_to \u003cstr\u003e              where to report metrics, options: ['wandb', 'tensorboard', 'comet_ml'] (default=wandb)\n    --checkpointing_steps \u003cint\u003e    number of steps between each checkpoint (default=50000)\n    --resume_from_checkpoint \u003cstr\u003e whether training should be resumed from a previous checkpoint. Use a \"latest\" to automatically select the last available checkpoint. (default=None)\n    --enable_xformers_memory_efficient_attention \u003cstore_true\u003e\n                                   enable memory efficient attention in xformers (default=False)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n    --num_workers_test \u003cint\u003e       number of workers for test dataloader (default=8)\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired'] (default=paired)\n    --uncond_fraction \u003cfloat\u003e      fraction of unconditioned training samples (default=0.2)\n    --text_usage \u003cstr\u003e             text features to use, options: ['none', 'noun_chunks', 'inversion_adapter'] (default=inversion_adapter)\n    --cloth_input_type \u003cstr\u003e       cloth input type, options: ['none', 'warped'], (default=warped)\n    --num_vstar \u003cint\u003e              number of predicted v* per image to use (default=16)\n    --num_encoder_layers \u003cint\u003e     number of ViT layers to use in inversion adapter (default=1)\n    --train_inversion_adapter \u003cstore_true\u003e\n                                   train the inversion adapter during the VTO training (default=False)\n    --use_clip_cloth_features \u003cstore_true\u003e\n                                   use precomputed clip cloth features instead of computing them each iteration (default=False).\n            \n```\n\nAt the end of the training, the checkpoints will be saved in the `output_dir` folder.\n\n**NOTE**: You can use the `--use_clip_cloth_features` flag only if you have previously computed the clip cloth features\nusing the `src/utils/compute_cloth_clip_features.py` script (step 2.5).\n\n### 5. Inference with the trained models\n\nBefore running the inference, make sure to specify the correct path to all the trained checkpoints.\nMake sure to also use coherent hyperparameters with the ones used during training.\n\nTo run the inference on the Dress Code or VITON-HD dataset, run the following command:\n\n```sh\npython src/eval.py --dataset [dresscode | vitonhd] --dresscode_dataroot \u003cpath\u003e --vitonhd_dataroot \u003cpath\u003e --output_dir \u003cpath\u003e --save_name \u003cstr\u003e --test_order [paired | unpaired]  --unet_dir \u003cpath\u003e --inversion_adapter_dir \u003cpath\u003e --emasc_dir \u003cpath\u003e  --category [all | lower_body | upper_body | dresses ] --enable_xformers_memory_efficient_attention --use_png --compute_metrics\n```\n\n```\n    --dataset \u003cstr\u003e                dataset to use, options: ['dresscode', 'vitonhd']\n    --dresscode_dataroot \u003cstr\u003e     data root of dresscode dataset (required when dataset=dresscode)\n    --vitonhd_dataroot \u003cstr\u003e       data root of vitonhd dataset (required when dataset=vitonhd)\n    --output_dir \u003cstr\u003e             output directory where the generated images will be written\n    --save_name \u003cstr\u003e              name of the generated images folder inside `output_dir`\n    --test_order \u003cstr\u003e             test setting, options: ['paired', 'unpaired']\n    --unet_dir \u003cstr\u003e               path to the UNet checkpoint directory. Should be the same as `output_dir` of the VTO training script\n    --unet_name \u003cstr\u003e              name of the UNet checkpoint. To load the latest checkpoint, use `latest`. (default=latest)\n    --inversion_adapter_dir \u003cstr\u003e  path to the inversion adapter checkpoint directory. Should be the same as `output_dir` of the VTO training script. Needed only if `--text_usage` is set to `inversion_adapter`. (default=None)\n    --inversion_adapter_name \u003cstr\u003e name of the inversion adapter checkpoint. To load the latest checkpoint, use `latest`. (default=latest)\n    --emasc_dir \u003cstr\u003e              path to the EMASC checkpoint directory. Should be the same as `output_dir` of the EMASC training script. Needed when --emasc_type!=none. (default=None)\n    --emasc_name \u003cstr\u003e             name of the EMASC checkpoint. To load the latest checkpoint, use `latest`. (default=latest)\n    --pretrained_model_name_or_path \u003cstr\u003e\n                                   model identifier from huggingface.co/models (default=stabilityai/stable-diffusion-2-inpainting)\n    --seed \u003cint\u003e                   seed for reproducible training (default=1234)\n    --batch_size \u003cint\u003e             batch size(default=8)\n    --allow_tf32 \u003cstore_true\u003e      allow TF32 on Ampere GPUs (default=False)\n    --enable_xformers_memory_efficient_attention \u003cstore_true\u003e\n                                   enable memory efficient attention in xformers (default=False)\n    --num_workers \u003cint\u003e            number of workers (default=8)\n    --category \u003cstr\u003e               category to test, options: ['all', 'lower_body', 'upper_body', 'dresses'] (default=all)\n    --emasc_type \u003cstr\u003e             type of EMASC, options: ['linear', 'nonlinear'] (default=nonlinear)\n    --emasc_kernel \u003cint\u003e           kernel size for the EMASC module (default=3)\n    --emasc_padding \u003cint\u003e          padding for the EMASC module (default=1)\n    --text_usage \u003cstr\u003e             text features to use, options: ['none', 'noun_chunks', 'inversion_adapter'] (default=inversion_adapter)\n    --cloth_input_type \u003cstr\u003e       cloth input type, options: ['none', 'warped'], (default=warped)\n    --num_vstar \u003cint\u003e              number of predicted v* per image to use (default=16)\n    --num_encoder_layers \u003cint\u003e     number of ViT layers to use in inversion adapter (default=1)\n    --use_png \u003cstore_true\u003e         use png instead of jpg (default=False)\n    --num_inference_steps \u003cint\u003e    number of diffusion steps at inference time (default=50)\n    --guidance_scale \u003cfloat\u003e       guidance scale of the diffusion (default=7.5)\n    --use_clip_cloth_features \u003cstore_true\u003e\n                                   use precomputed clip cloth features instead of computing them each iteration (default=False).\n    --compute_metrics              compute metrics at the end of inference (default=False)\n```\n\nThe generated images will be saved in the `output_dir/save_name_{test_order}` folder.\n\n**NOTE**: You can use the `--use_clip_cloth_features` flag only if you have previously computed the clip cloth features\nusing the `src/utils/compute_cloth_clip_features.py` script (step 2.5).\n\n\u003c/details\u003e\n\n## Acknowledgements\n\nThis work has partially been supported by the PNRR project “Future Artificial Intelligence Research (FAIR)”, by the PRIN\nproject “CREATIVE: CRoss-modal understanding and gEnerATIon of Visual and tExtual content” (CUP B87G22000460001), both\nco-funded by the Italian Ministry of University and Research, and by the European Commission under European Horizon 2020\nProgramme, grant number 101004545 - ReInHerit.\n\n## LICENSE\n\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" /\u003e\u003c/a\u003e\u003cbr /\u003e\nAll material is made available under [Creative Commons BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). You\ncan **use, redistribute, and adapt** the material for **non-commercial purposes**, as long as you give appropriate\ncredit by **citing our paper** and **indicate any changes** that you've made.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiccunifi%2Fladi-vton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiccunifi%2Fladi-vton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiccunifi%2Fladi-vton/lists"}