{"id":13487942,"url":"https://github.com/eric-ai-lab/swap-anything","last_synced_at":"2025-03-27T23:32:12.023Z","repository":{"id":232325576,"uuid":"783736494","full_name":"eric-ai-lab/swap-anything","owner":"eric-ai-lab","description":"Official implementation of the ECCV paper \"SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing\"","archived":false,"fork":false,"pushed_at":"2024-10-10T19:02:15.000Z","size":6299,"stargazers_count":228,"open_issues_count":1,"forks_count":7,"subscribers_count":29,"default_branch":"main","last_synced_at":"2024-10-30T23:36:16.249Z","etag":null,"topics":["diffusion-models","image-editing","personalization","photoswapping","subject-driven-generation","swap-anything"],"latest_commit_sha":null,"homepage":"https://swap-anything.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eric-ai-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-08T13:23:43.000Z","updated_at":"2024-10-22T08:13:14.000Z","dependencies_parsed_at":"2024-04-16T00:25:09.098Z","dependency_job_id":null,"html_url":"https://github.com/eric-ai-lab/swap-anything","commit_stats":null,"previous_names":["eric-ai-lab/swap-anything"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric-ai-lab%2Fswap-anything","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric-ai-lab%2Fswap-anything/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric-ai-lab%2Fswap-anything/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric-ai-lab%2Fswap-anything/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eric-ai-lab","download_url":"https://codeload.github.com/eric-ai-lab/swap-anything/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245944020,"owners_count":20697945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","image-editing","personalization","photoswapping","subject-driven-generation","swap-anything"],"created_at":"2024-07-31T18:01:06.845Z","updated_at":"2025-03-27T23:32:06.997Z","avatar_url":"https://github.com/eric-ai-lab.png","language":"Python","funding_links":[],"categories":["Text Guided Image Editing"],"sub_categories":[],"readme":"# SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing (ECCV 2024)\n\n[Jing Gu](https://g-jing.github.io/), [Yilin Wang](https://yilinwang.org/), [Nanxuan Zhao](http://nxzhao.com/), [Wei Xiong](https://wxiong.me/), [Qing Liu](https://qliu24.github.io/), [Zhifei Zhang](https://zzutk.github.io/), [He Zhang](https://sites.google.com/site/hezhangsprinter/), [Jianming Zhang](https://cs-people.bu.edu/jmzhang/), [HyunJoon Jung](https://polaris79.wixsite.com/hjung), [Xin Eric Wang](https://eric-xw.github.io/)\n\n[[Project Page](https://swap-anything.github.io/)] [[Paper](https://arxiv.org/abs/2404.05717)]\n\n![Teaser figure](figures/teaser.png)\n\n## Model Architecture\n![Teaser figure](figures/pipeline.png)\n\n## TODO\n- [ ] Release demo  \n- [x] Release code  \n\n## Crop-Edit-Paste Pipeline\nSince swap-anything could preserve the background perfectly, we could avoid the image size and ratio limitation of the backbone image diffusion model. Now we could edit object of any size in image of any size! Swap-anything support personalized object swapping, general object swapping, and object insertion.\n\n\n## 1. Git clone this repo\n```bash\ngit clone https://github.com/eric-ai-lab/swap-anything.git\ncd swap-anything\n```\n\n## 2. Image and Mask Preparation\n1. For a source image `filename.jpg`, please create `source_image/filename/`:\n   ```bash\n   mkdir source_image/filename\n   ```\n\n2. Put binary mask image into `source_image/filename`.\nThe mask image name should be `filename_mask.png`.\n\nWe accept general source image types like `.jpg`, `.jpeg`, `.png`. However, make sure the mask is in `.png` format so that the code can find the mask according to pattern matching. Please refer to the provided example in `source_image`.\n\n\n## 3. Train DreamBooth Model to Include the New Concept (Optional)\n1. Our method also works with other concept learning methods such as CustomDiffusion, Text Inversion, etc.\n2. You do not need this step for general object swapping.\n\n### Package Installation\n\nFollow the installation instructions from [Hugging Face Diffusers v0.25.0](https://github.com/huggingface/diffusers/tree/v0.25.0):\n\n```bash\ncd diffusers\npip install -e .\ncd examples/dreambooth\npip install -r requirements.txt\n```\n\n### Concept Data\nPlace images of the target concept into `dreambooth_data/{INSTANCE_NAME}`. The more images you provide, the better the results. Optimal performance is typically achieved with around 20 images.\n\n### Training DreamBooth\nIn script `train_dreambooth.sh`\n1. Set `INSTANCE_NAME` to the name of the image folder.\n2. Set `CLASS_NAME` to the class of the target object.\n\nRun the following command to start training:\n\n```bash\n./train_dreambooth.sh\n```\n\nThe script will generate checkpoints in the folder `checkpoints/checkpoint_$INSTANCE_NAME-$Model_IDENTITY`.\n\n## 4. Editing Image\n\n\n- **Personal object swapping**: \n  ```bash\n  python main.py --config config_personal_swap.yml\n  ```\n  To perform a personalized swap, set `concept_model_path` to the folder containing the diffusion model with the learned concept. Our model uses DreamBooth, so update `source_subject_word` with the object you want to replace and adjust `source_prompt` accordingly. Similarly, modify `target_subject_word` and `target_prompt` based on the tokens used during DreamBooth training.\n\n- **General object swapping**: \n  ```bash\n  python main.py --config config_general_swap.yml\n  ```\n  For general object swaps (non-personalized), you can use `\"runwayml/stable-diffusion-v1-5\"` as the `concept_model_path`.\n\n- **Object insertion**: \n  ```bash\n  python main.py --config config_insertion.yml\n  ```\n  For object insertion, follow the same process as the swapping task. The difference is to set `source_subject_word` to `'nothing'` and `source_prompt` to `\"a photo of nothing\"`.\n\n### Output\nThe editing results will be in the folder `photoswap_real_output_{cuda_id}`. In each `sample_*` subfolder, you will find the source image, edited image, mask, along with their cropped versions, and a JSON file containing all variable information. \n\nWe also provide a webpage in the `html` folder so you can browse all results at once.\n\n\n### config*.yml expanation\n| Variable               | Value                          | Description                                  |\n|------------------------|--------------------------------|----------------------------------------------|\n| `cuda_id`              | `0`                            | ID of the CUDA device.                       |\n| `do_not_crop`          | `False`                        | Use the whole image when `True`.             |\n| `pre_defined_crop`     | `[]`                           | Crop coordinates `[x1, y1, x2, y2]`.         |\n| `blend_width`          | `20`                           | Width of the blend area.                     |\n| `total_diffusion_steps`| `50`                           | Diffusion steps.                             |\n| `guidance_scale`       | `7.5`                          | Guidance scale for diffusion.                |\n| `source_image_path`    | `'source_image/person1.jpg'`   | Source image path.                           |\n| `source_subject_word`  | `'person'`                     | Source subject word.                         |\n| `source_prompt`        | `\"a photo of a person\"`        | Source image prompt.                         |\n| `target_subject_word`  | `'sks'`                        | Target subject word.                         |\n| `target_prompt`        | `\"a photo of sks man\"`         | Target image prompt.                         |\n| `concept_model_path`   | `'checkpoints_folder'`         | Concept model path.                          |\n| `self_output_range`    | `[0.1, 0.3, 0.5, 0.7]`         | Self-attention output range.                 |\n| `self_map_range`       | `[0.0]`                        | Self-attention map range.                    |\n| `cross_map_range`      | `[0.1, 0.3, 0.5, 0.7]`         | Cross-attention map range.                   |\n| `add_zero_to_range`    | `True`                         | Add zero to range.                           |\n| `end_blend`            | `52`                           | End point for blending.                      |\n| `is_show_result`       | `True`                         | Show result in notebook mode if `True`.      |\n\n\n\n## A Few Tips for Better Performance\n1. For face swapping tasks, usually a higher variable swapping ratio yields better performance.\n2. If the swapping results show significant shape deformation, it could be due to the automatic cropping issue. Manually input the crop coordinates `x1, y1, x2, y2` so that `(x2 - x1) ≈ (y2 - y1)`. This is also useful for detailed manipulation. For example:\n   - Setting `(x2 - x1) \u003e (y2 - y1)` will make the face wider.\n   - Setting `(x2 - x1) \u003c (y2 - y1)` will make the face narrower horizontally.\n3. If the swapping result is too similar to the source image and does not transfer the target identity, try decreasing the swapping ratio. If the result contains artifacts or is not harmonious, you may want to increase the swapping ratio.\n\n\n\n## Citation\n\n\n```\n@inproceedings{gu2024swapanything,\n      title={SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing}, \n      author={Jing Gu and Yilin Wang and Nanxuan Zhao and Wei Xiong and Qing Liu and Zhifei Zhang and He Zhang and Jianming Zhang and HyunJoon Jung and Xin Eric Wang},\n      booktitle={ECCV},\n      year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric-ai-lab%2Fswap-anything","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feric-ai-lab%2Fswap-anything","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric-ai-lab%2Fswap-anything/lists"}