{"id":13488737,"url":"https://github.com/ucsb-nlp-chang/copaint","last_synced_at":"2025-03-28T01:37:30.988Z","repository":{"id":111712880,"uuid":"607268556","full_name":"UCSB-NLP-Chang/CoPaint","owner":"UCSB-NLP-Chang","description":"Implementation of paper 'Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models'","archived":false,"fork":false,"pushed_at":"2024-04-03T09:45:30.000Z","size":1901,"stargazers_count":60,"open_issues_count":1,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-31T00:40:10.502Z","etag":null,"topics":["deblurring","diffusion-models","image-inpainting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UCSB-NLP-Chang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-27T16:53:18.000Z","updated_at":"2024-09-29T16:06:33.000Z","dependencies_parsed_at":"2024-01-18T22:37:02.282Z","dependency_job_id":"6b601fea-a785-4810-9858-e245cbfc2304","html_url":"https://github.com/UCSB-NLP-Chang/CoPaint","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UCSB-NLP-Chang%2FCoPaint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UCSB-NLP-Chang%2FCoPaint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UCSB-NLP-Chang%2FCoPaint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UCSB-NLP-Chang%2FCoPaint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UCSB-NLP-Chang","download_url":"https://codeload.github.com/UCSB-NLP-Chang/CoPaint/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245952926,"owners_count":20699562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deblurring","diffusion-models","image-inpainting"],"created_at":"2024-07-31T18:01:20.940Z","updated_at":"2025-03-28T01:37:30.959Z","avatar_url":"https://github.com/UCSB-NLP-Chang.png","language":"Python","funding_links":[],"categories":["SD-inpaint"],"sub_categories":[],"readme":"# Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models\n\nThis is the official impelmenation of the paper [Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models](https://arxiv.org/pdf/2304.03322.pdf).\n\n## Abstract\nImage inpainting refers to the task of generating a complete, natural image based on a partially revealed reference image. Recently, many research interests have been focused on addressing this problem using fixed diffusion models. These approaches typically directly replace the revealed region of the intermediate or final generated images with that of the reference image or its variants.  However, since the unrevealed regions are not directly modified to match the context, it results in incoherence between revealed and unrevealed regions. To address the incoherence problem, a small number of methods introduce a rigorous Bayesian framework, but they tend to introduce mismatches between the generated and the reference images due to the approximation errors in computing the posterior distributions. In this paper, we propose COPAINT, which can coherently inpaint the whole image without introducing mismatches. COPAINT also uses the Bayesian framework to jointly modify both revealed and unrevealed regions, but approximates the posterior distribution in a way that allows the errors to gradually drop to zero throughout the denoising steps, thus strongly penalizing any mismatches with the reference image. Our experiments verify that COPAINT can outperform the existing diffusion-based methods under both objective and subjective metrics.\n\n## Requirements\n### Environment\nInstall conda environment by running\n```\nconda env create -f environment.yaml\nconda activate copaint\n```\n### Download pretrained models and data\n```\nbash scripts/download.sh\n```\nThis script will download pretrained diffusion models from [guided-diffusion](https://github.com/openai/guided-diffusion) and [repaint](https://github.com/andreas128/RePaint).\n\nThen it will download and preprocess `CelebA-HQ` dataset according to the data split in [Lama](https://github.com/saic-mdal/lama). The images we used for `ImageNet` dataset should be placed in `datasets/imagenet100`, which could be downloaded [here](https://drive.google.com/drive/folders/1CTBHK8udyGejJEob-HTL1MrmtkqC_3gr?usp=sharing).\n\n## Usage\n### Inpainting with `CoPaint`\nTo inpaint a specific image with our algorithm `CoPaint`, you can run \n```text\npython main.py:\n    --config_file: The configuration file, which specifies the model to use and some hyper-parameters for our method\n    --input_image: The path to input image\n    --mask:        The path to mask file \n    --outdir:      The path to output folder\n    --n_samples:   The number of images to be generated\n    --algorithm:   The algorithm to be used\n```\nHere is an example result on `CelebA-HQ` dataset with `CoPaint`:\n```shell\npython main.py --config_file configs/celebahq.yaml --input_image examples/celeb-sample.jpg --mask examples/celeb-mask.jpg --outdir images/example --n_samples 1 --algorithm o_ddim \n```\nHere is an example result on `CelebA-HQ` dataset with `CoPaint-TT`:\n```shell\npython main.py --config_file configs/celebahq.yaml --input_image examples/celeb-sample.jpg --mask examples/celeb-mask.jpg --outdir images/example --n_samples 1 --algorithm o_ddim --ddim.schedule_params.jump_length 10 --ddim.schedule_params.jump_n_sample 2 --ddim.schedule_params.use_timetravel\n```\n\n\u003cimg src=\"assets/sample.png\" width=400px\u003e\n\n### Tip:\nYou can tune following hyper-parameters if you are not satisfied with the generated image:\n* `--optimize_xt.num_iteration_optimize_xt`: The number of optimization steps $G$.\n* `--optimize_xt.lr_xt`: The initial learning rate $\\mu_T$\nBelow are hyper-paremters for the time-travel trick:\n* `--ddim.schedule_params.use_timetravel`: Whether to use time-travel trick.\n* `--ddim.schedule_parms.jump_length` : The time-travel interval $\\tau$.\n* `--ddim.schedule_parms.jum_n_sample` : The time-travel frequency $K$.\n\nPlease refer to our paper for more details.\n\n## Reproduce results in our paper\nWe provide shell scripts for all algorithms discussed in our paper, `copaint, repaint, copaint-tt`, you can find them in `scripts` folder. To run a specific algorithm on both datasets:\n```shell\nbash scripts/{algorithm}.sh\n```\nThe generated images will be put in `images/{algorithm}`.\n\nBelow is an example to run `Copaint` on `CelebA-HQ` dataset with `half` mask:\n```shell\npython main.py --dataset_name celebahq --algorithm o_ddim --outdir celebresults --mask_type half --config_file configs/celebahq.yaml\n```\n\n## References\nIf you find our work useful for your research, please consider citing our paper:\n```bibtex\n@misc{zhang2023coherent,\n      title={Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models},\n      author={Guanhua Zhang and Jiabao Ji and Yang Zhang and Mo Yu and Tommi Jaakkola and Shiyu Chang},\n      year={2023},\n      eprint={2304.03322},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\nOur implementation is based on following repos:\n* https://github.com/andreas128/RePaint\n* https://github.com/bahjat-kawar/ddrm\n* https://github.com/wyhuai/DDNM\n* https://github.com/DPS2022/diffusion-posterior-sampling\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucsb-nlp-chang%2Fcopaint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fucsb-nlp-chang%2Fcopaint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucsb-nlp-chang%2Fcopaint/lists"}