{"id":13488721,"url":"https://github.com/Fantasy-Studio/Paint-by-Example","last_synced_at":"2025-03-28T01:37:23.494Z","repository":{"id":64101052,"uuid":"569687650","full_name":"Fantasy-Studio/Paint-by-Example","owner":"Fantasy-Studio","description":"Paint by Example: Exemplar-based Image Editing with Diffusion Models","archived":false,"fork":false,"pushed_at":"2023-11-28T02:32:20.000Z","size":18873,"stargazers_count":1104,"open_issues_count":32,"forks_count":98,"subscribers_count":23,"default_branch":"main","last_synced_at":"2024-10-29T20:59:31.415Z","etag":null,"topics":["computer-vision","deep-learning","diffusion-models","image-editing","image-generation","image-manipulation","paint-by-example","pytorch","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2211.13227","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Fantasy-Studio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-11-23T11:49:29.000Z","updated_at":"2024-10-29T11:45:15.000Z","dependencies_parsed_at":"2023-11-28T03:44:07.418Z","dependency_job_id":null,"html_url":"https://github.com/Fantasy-Studio/Paint-by-Example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantasy-Studio%2FPaint-by-Example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantasy-Studio%2FPaint-by-Example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantasy-Studio%2FPaint-by-Example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Fantasy-Studio%2FPaint-by-Example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Fantasy-Studio","download_url":"https://codeload.github.com/Fantasy-Studio/Paint-by-Example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222333976,"owners_count":16968058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","diffusion-models","image-editing","image-generation","image-manipulation","paint-by-example","pytorch","stable-diffusion"],"created_at":"2024-07-31T18:01:20.698Z","updated_at":"2024-10-31T00:32:30.615Z","avatar_url":"https://github.com/Fantasy-Studio.png","language":"Python","readme":"# Paint by Example: Exemplar-based Image Editing with Diffusion Models\n![Teaser](figure/teaser.png)\n### [Paper](https://arxiv.org/abs/2211.13227) | [Huggingface Demo](https://huggingface.co/spaces/Fantasy-Studio/Paint-by-Example) \n\u003c!-- \u003cbr\u003e --\u003e\n[Binxin Yang](https://orcid.org/0000-0003-4110-1986), [Shuyang Gu](http://home.ustc.edu.cn/~gsy777/), [Bo Zhang](https://bo-zhang.me/), [Ting Zhang](https://www.microsoft.com/en-us/research/people/tinzhan/), [Xuejin Chen](http://staff.ustc.edu.cn/~xjchen99/), [Xiaoyan Sun](http://staff.ustc.edu.cn/~xysun720/), [Dong Chen](https://www.microsoft.com/en-us/research/people/doch/) and [Fang Wen](https://www.microsoft.com/en-us/research/people/fangwen/).\n\u003c!-- \u003cbr\u003e --\u003e\n\n## Abstract\n\u003eLanguage-guided image editing has achieved great success recently. In this paper, for the first time, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. However, the naive approach will cause obvious fusing artifacts. We carefully analyze it and propose an information bottleneck and strong augmentations to avoid the trivial solution of directly copying and pasting the exemplar image. Meanwhile, to ensure the controllability of the editing process, we design an arbitrary shape mask for the exemplar image and leverage the classifier-free guidance to increase the similarity to the exemplar image. The whole framework involves a single forward of the diffusion model without any iterative optimization. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.\n\u003e\n## News\n- *2023-11-28* The recent work Asymmetric VQGAN improves the preservation of details in non-masked regions. For comprehensive details, please refer to the associated [paper](https://arxiv.org/abs/2306.04632), [github]( https://github.com/buxiangzhiren/Asymmetric_VQGAN).\n- *2023-05-13* Release code for quantitative results.\n- *2023-03-03* Release test benchmark.\n- *2023-02-23* Non-official 3rd party apps support by [ModelScope](https://www.modelscope.cn/models/damo/cv_stable-diffusion_paint-by-example/summary) (the largest Model Community in Chinese).\n- *2022-12-07* Release a [Gradio](https://gradio.app/) demo on [Hugging Face](https://huggingface.co/spaces/Fantasy-Studio/Paint-by-Example) Spaces.\n- *2022-11-29* Upload code.\n\n\n## Requirements\nA suitable [conda](https://conda.io/) environment named `Paint-by-Example` can be created\nand activated with:\n\n```\nconda env create -f environment.yaml\nconda activate Paint-by-Example\n```\n\n## Pretrained Model\nWe provide the checkpoint ([Google Drive](https://drive.google.com/file/d/15QzaTWsvZonJcXsNv-ilMRCYaQLhzR_i/view?usp=share_link) | [Hugging Face](https://huggingface.co/Fantasy-Studio/Paint-by-Example/resolve/main/model.ckpt)) that is trained on [Open-Images](https://storage.googleapis.com/openimages/web/index.html) for 40 epochs. By default, we assume that the pretrained model is downloaded and saved to the directory `checkpoints`.\n\n## Testing\n\nTo sample from our model, you can use `scripts/inference.py`. For example, \n```\npython scripts/inference.py \\\n--plms --outdir results \\\n--config configs/v1.yaml \\\n--ckpt checkpoints/model.ckpt \\\n--image_path examples/image/example_1.png \\\n--mask_path examples/mask/example_1.png \\\n--reference_path examples/reference/example_1.jpg \\\n--seed 321 \\\n--scale 5\n```\nor simply run:\n```\nsh test.sh\n```\nVisualization of inputs and output:\n\n![](figure/result_1.png)\n![](figure/result_2.png)\n![](figure/result_3.png)\n\n## Training\n\n### Data preparing\n- Download separate packed files of Open-Images dataset from [CVDF's site](https://github.com/cvdfoundation/open-images-dataset#download-images-with-bounding-boxes-annotations) and unzip them to the directory `dataset/open-images/images`.\n- Download bbox annotations of Open-Images dataset from [Open-Images official site](https://storage.googleapis.com/openimages/web/download_v7.html#download-manually) and save them to the directory `dataset/open-images/annotations`.\n- Generate bbox annotations of each image in txt format.\n    ```\n    python scripts/read_bbox.py\n    ```\n\nThe data structure is like this:\n```\ndataset\n├── open-images\n│  ├── annotations\n│  │  ├── class-descriptions-boxable.csv\n│  │  ├── oidv6-train-annotations-bbox.csv\n│  │  ├── test-annotations-bbox.csv\n│  │  ├── validation-annotations-bbox.csv\n│  ├── images\n│  │  ├── train_0\n│  │  │  ├── xxx.jpg\n│  │  │  ├── ...\n│  │  ├── train_1\n│  │  ├── ...\n│  │  ├── validation\n│  │  ├── test\n│  ├── bbox\n│  │  ├── train_0\n│  │  │  ├── xxx.txt\n│  │  │  ├── ...\n│  │  ├── train_1\n│  │  ├── ...\n│  │  ├── validation\n│  │  ├── test\n```\n\n### Download the pretrained model of Stable Diffusion\nWe utilize the pretrained Stable Diffusion v1-4 as initialization, please download the pretrained models from [Hugging Face](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original) and save the model to directory `pretrained_models`. Then run the following script to add zero-initialized weights for 5 additional input channels of the UNet (4 for the encoded masked-image and 1 for the mask itself).\n```\npython scripts/modify_checkpoints.py\n```\n\n### Training Paint by Example\nTo train a new model on Open-Images, you can use `main.py`. For example,\n```\npython -u main.py \\\n--logdir models/Paint-by-Example \\\n--pretrained_model pretrained_models/sd-v1-4-modified-9channel.ckpt \\\n--base configs/v1.yaml \\\n--scale_lr False\n```\nor simply run:\n```\nsh train.sh\n```\n\n## Test Benchmark\nWe build a test benchmark for quantitative analysis. Specifically, we manually select 3500 source images from MSCOCO validation set, each image contains only one bounding box. Then we manually retrieve a reference image patch from MSCOCO training set. The reference image usually shares a similar semantic with mask region to ensure the combination is reasonable. We named it as COCO Exemplar-based image Editing benchmark, abbreviated as COCOEE. This test benchmark can be downloaded from [Google Drive](https://drive.google.com/file/d/18wO_wSFF-GPNxWmO1bt6LdjubXcttqtO/view?usp=share_link).\n\n## Quantitative Results\nBy default, we assume that the COCOEE is downloaded and saved to the directory `test_bench`. To generate the results of test bench, you can use `scripts/inference_test_bench.py`. For example, \n```\npython scripts/inference_test_bench.py \\\n--plms \\\n--outdir results/test_bench \\\n--config configs/v1.yaml \\\n--ckpt checkpoints/model.ckpt \\\n--scale 5\n```\nor simply run:\n```\nbash inference_test_bench.sh\n```\n### FID Score\nBy default, we assume that the test set of COCO2017 is downloaded and saved to the directory `dataset`.\nThe data structure is like this:\n```\ndataset\n├── coco\n│  ├── test2017\n│  │  ├── xxx.jpg\n│  │  ├── xxx.jpg\n│  │  ├── ...\n│  │  ├── xxx.jpg\n```\nThen convert the images into square images with 512 solution.\n  ```\n  python scripts/create_square_gt_for_fid.py\n  ```\nTo calculate FID score, simply run:\n```\npython eval_tool/fid/fid_score.py --device cuda \\\ntest_bench/test_set_GT \\\nresults/test_bench/results\n```\n### QS Score\nPlease download the model weights for QS score from [Google Drive](https://drive.google.com/file/d/1Ce2cSQ8UttxcEk03cjfJgaBwdhSPyuHI/view?usp=share_link) and save the model to directory `eval_tool/gmm`.\nTo calculate QS score, simply run:\n```\npython eval_tool/gmm/gmm_score_coco.py results/test_bench/results \\\n--gmm_path eval_tool/gmm/coco2017_gmm_k20 \\\n--gpu 1\n```\n\n### CLIP Score\nTo calculate CLIP score, simply run:\n```\npython eval_tool/clip_score/region_clip_score.py \\\n--result_dir results/test_bench/results\n```\n\n\n## Citing Paint by Example\n\n```\n@article{yang2022paint,\n  title={Paint by Example: Exemplar-based Image Editing with Diffusion Models},\n  author={Binxin Yang and Shuyang Gu and Bo Zhang and Ting Zhang and Xuejin Chen and Xiaoyan Sun and Dong Chen and Fang Wen},\n  journal={arXiv preprint arXiv:2211.13227},\n  year={2022}\n}\n```\n\n## Acknowledgements\n\nThis code borrows heavily from [Stable Diffusion](https://github.com/CompVis/stable-diffusion). We also thank the contributors of [OpenAI's ADM codebase](https://github.com/openai/guided-diffusion) and [https://github.com/lucidrains/denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch).\n\n## Maintenance\n\nPlease open a GitHub issue for any help. If you have any questions regarding the technical details, feel free to contact us.\n\n## License\nThe codes and the pretrained model in this repository are under the CreativeML OpenRAIL M license as specified by the LICENSE file.\n\nThe test benchmark, COCOEE, belongs to the COCO Consortium and are licensed under a Creative Commons Attribution 4.0 License.\n","funding_links":[],"categories":["Python","SD-inpaint","Leaderboard"],"sub_categories":["Evaluating Your Results"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFantasy-Studio%2FPaint-by-Example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFantasy-Studio%2FPaint-by-Example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFantasy-Studio%2FPaint-by-Example/lists"}