{"id":13958402,"url":"https://github.com/andreas128/RePaint","last_synced_at":"2025-07-20T23:31:26.025Z","repository":{"id":37702622,"uuid":"433054511","full_name":"andreas128/RePaint","owner":"andreas128","description":"Official PyTorch Code and Models of \"RePaint: Inpainting using Denoising Diffusion Probabilistic Models\", CVPR 2022","archived":false,"fork":false,"pushed_at":"2022-08-20T13:01:44.000Z","size":94,"stargazers_count":1980,"open_issues_count":45,"forks_count":161,"subscribers_count":40,"default_branch":"main","last_synced_at":"2024-11-21T05:24:03.917Z","etag":null,"topics":["cvpr2022","diffusion-models","inpainting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreas128.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSES/LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-29T13:30:53.000Z","updated_at":"2024-11-19T14:39:36.000Z","dependencies_parsed_at":"2022-07-14T07:10:27.935Z","dependency_job_id":null,"html_url":"https://github.com/andreas128/RePaint","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas128%2FRePaint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas128%2FRePaint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas128%2FRePaint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreas128%2FRePaint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreas128","download_url":"https://codeload.github.com/andreas128/RePaint/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226845016,"owners_count":17691142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2022","diffusion-models","inpainting"],"created_at":"2024-08-08T13:01:31.949Z","updated_at":"2024-11-28T01:31:58.500Z","avatar_url":"https://github.com/andreas128.png","language":"Python","readme":"# RePaint\n**Inpainting using Denoising Diffusion Probabilistic Models**\n\n\nCVPR 2022 [[Paper]](https://bit.ly/3b1ABEb)\n\n[![Denoising_Diffusion_Inpainting_Animation](https://user-images.githubusercontent.com/11280511/150849757-5cd762cb-07a3-46aa-a906-0fe4606eba3b.gif)](#)\n\n## Setup\n\n### 1. Code\n\n```bash\ngit clone https://github.com/andreas128/RePaint.git\n```\n\n### 2. Environment\n```bash\npip install numpy torch blobfile tqdm pyYaml pillow    # e.g. torch 1.7.1+cu110.\n```\n\n### 3. Download models and data\n\n```bash\npip install --upgrade gdown \u0026\u0026 bash ./download.sh\n```\n\nThat downloads the models for ImageNet, CelebA-HQ, and Places2, as well as the face example and example masks.\n\n\n### 4. Run example\n```bash\npython test.py --conf_path confs/face_example.yml\n```\nFind the output in `./log/face_example/inpainted`\n\n*Note: After refactoring the code, we did not reevaluate all experiments.*\n\n\u003cbr\u003e\n\n# RePaint fills a missing image part using diffusion models\n\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e\u003cimg alt=\"RePaint Inpainting using Denoising Diffusion Probabilistic Models Demo 1\" src=\"https://user-images.githubusercontent.com/11280511/150766080-9f3d7bc9-99f2-472e-9e5d-b6ed456340d1.gif\"\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cimg alt=\"RePaint Inpainting using Denoising Diffusion Probabilistic Models Demo 2\" src=\"https://user-images.githubusercontent.com/11280511/150766125-adf5a3cb-17f2-432c-a8f6-ce0b97122819.gif\"\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n**What are the blue parts?** \u003cbr\u003e\nThose parts are missing and therefore have to be filled by RePaint. \u003cbr\u003e RePaint generates the missing parts inspired by the known parts.\n\n**How does it work?** \u003cbr\u003e\nRePaint starts from pure noise. Then the image is denoised step-by-step.  \u003cbr\u003e It uses the known part to fill the unknown part in each step.\n\n**Why does the noise level fluctuate during generation?** \u003cbr\u003e\nOur noise schedule improves the harmony between the generated and \u003cbr\u003e the known part [[4.2 Resampling]](https://bit.ly/3b1ABEb).\n\n\u003cbr\u003e\n\n## Details on data\n\n**Which datasets and masks have a ready-to-use config file?**\n\nWe provide config files for ImageNet (inet256), CelebA-HQ (c256) and Places2 (p256) for the masks \"thin\", \"thick\", \"every second line\", \"super-resolution\", \"expand\" and \"half\" in [`./confs`](https://github.com/andreas128/RePaint/tree/main/confs). You can use them as shown in the example above.\n\n**How to prepare the test data?**\n\nWe use [LaMa](https://github.com/saic-mdal/lama) for validation and testing. Follow their instructions and add the images as specified in the config files. When you download the data using `download.sh`, you can see examples of masks we used.\n\n**How to apply it to other images?**\n\nCopy the config file for the dataset that matches your data best (for faces aligned like CelebA-HQ `_c256`, for diverse images `_inet256`). Then set the [`gt_path`](https://github.com/andreas128/RePaint/blob/0fea066b52346c331cdf1bf7aed616c8c8896714/confs/face_example.yml#L70) and [`mask_path`](https://github.com/andreas128/RePaint/blob/0fea066b52346c331cdf1bf7aed616c8c8896714/confs/face_example.yml#L71) to where your input is. The masks have the value 255 for known regions and 0 for unknown areas (the ones that get generated).\n\n**How to apply it for other datasets?**\n\nIf you work with other data than faces, places or general images, train a model using the [guided-diffusion](https://github.com/openai/guided-diffusion) repository. Note that RePaint is an inference scheme. We do not train or finetune the diffusion model but condition pre-trained models.\n\n## Adapt the code\n\n**How to design a new schedule?**\n\nFill in your own parameters in this [line](https://github.com/andreas128/RePaint/blob/0fea066b52346c331cdf1bf7aed616c8c8896714/guided_diffusion/scheduler.py#L180) to visualize the schedule using `python guided_diffusion/scheduler.py`. Then copy a config file, set your parameters in these [lines](https://github.com/andreas128/RePaint/blob/0fea066b52346c331cdf1bf7aed616c8c8896714/confs/face_example.yml#L61-L65) and run the inference using `python test.py --conf_path confs/my_schedule.yml`. \n\n**How to speed up the inference?**\n\nThe following settings are in the [schedule_jump_params](https://github.com/andreas128/RePaint/blob/0fea066b52346c331cdf1bf7aed616c8c8896714/confs/face_example.yml#L61) key in the config files. You can visualize them as described above.\n\n- Reduce `t_T`, the total number of steps (without resampling). The lower it is, the more noise gets removed per step.\n- Reduce `jump_n_sample` to resample fewer times.\n- Apply resampling not from the beginning but only after a specific time by setting `start_resampling`.\n\n## Code overview\n\n- **Schedule:** The list of diffusion times t which will be traversed are obtained in this [line](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L503). e.g. times = [249, 248, 249, 248, 247, 248, 247, 248, 247, 246, ...]\n- **Denoise:** Reverse diffusion steps from x\u003csub\u003et\u003c/sub\u003e (more noise) to a x\u003csub\u003et-1\u003c/sub\u003e (less noisy) are done below this [line](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L515).\n- **Predict:** The model is called [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L237) and obtains x\u003csub\u003et\u003c/sub\u003e and the time t to predict a tensor with 6 channels containing information about the mean and variance of x\u003csub\u003et-1\u003c/sub\u003e. Then the value range of the variance is adjusted [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L252). The mean of x\u003csub\u003et-1\u003c/sub\u003e is obtained by the weighted sum of the estimated [x\u003csub\u003e0\u003c/sub\u003e](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L270) and x\u003csub\u003et\u003c/sub\u003e [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L189). The obtained mean and variance is used [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L402) to sample x\u003csub\u003et-1\u003c/sub\u003e. (This is the original reverse step from [guided-diffusion](https://github.com/openai/guided-diffusion.git). )\n- **Condition:** The known part of the input image needs to have the same amount of noise as the part that the diffusion model generates to join them. The required amount of noise is calculated [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L368) and added to the known part [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L371). The generated and sampled parts get joined using a maks [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L373).\n- **Undo:** The forward diffusion steps from x\u003csub\u003et-1\u003c/sub\u003e to x\u003csub\u003et\u003c/sub\u003e is done after this [line](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L536). The noise gets added to x\u003csub\u003et-1\u003c/sub\u003e [here](https://github.com/andreas128/RePaint/blob/76cb5b49d3f28715980f6e809c6859b148be9867/guided_diffusion/gaussian_diffusion.py#L176).\n\n## Issues\n\n**Do you have further questions?**\n\nPlease open an [issue](https://github.com/andreas128/RePaint/issues), and we will try to help you.\n\n**Did you find a mistake?**\n\nPlease create a pull request. For examply by clicking the pencil button on the top right on the github page.\n\n\u003cbr\u003e\n\n# RePaint on diverse content and shapes of missing regions\n\nThe blue region is unknown and filled by RePaint:\n\n![Denoising Diffusion Probabilistic Models Inpainting](https://user-images.githubusercontent.com/11280511/150803812-a4729ef8-6ad4-46aa-ae99-8c27fbb2ea2e.png)\n\n\n**Note: RePaint creates many meaningful fillings.** \u003cbr\u003e\n1) **Face:** Expressions and features like an earring or a mole. \u003cbr\u003e\n2) **Computer:** The computer screen shows different images, text, and even a logo. \u003cbr\u003e\n3) **Greens:** RePaint makes sense of the tiny known part and incorporates it in a beetle, spaghetti, and plants. \u003cbr\u003e\n4) **Garden:** From simple filling like a curtain to complex filling like a human. \u003cbr\u003e\n\n\n\u003cbr\u003e\n\n# Extreme Case 1: Generate every second line\n![Denoising_Diffusion_Probabilistic_Models_Inpainting_Every_Second_Line](https://user-images.githubusercontent.com/11280511/150818064-29789cbe-73c7-45de-a955-9fad5fb24c0e.png)\n\n- Every Second line of the input image is unknown.\n- Most inpainting methods fail on such masks.\n\n\n\u003cbr\u003e\n\n# Extreme Case 2: Upscale an image\n![Denoising_Diffusion_Probabilistic_Models_Inpainting_Super_Resolution](https://user-images.githubusercontent.com/11280511/150818741-5ed19a0b-1cf8-4f28-9e57-2e4c12303c3e.png)\n\n- The inpainting only knows pixels with a stridden access of 2.\n- A ratio of 3/4 of the image has to be filled.\n- This is equivalent to Super-Resolution with the Nearest Neighbor kernel.\n\n\u003cbr\u003e\n\n# RePaint conditions the diffusion model on the known part\n\n- RePaint uses unconditionally trained Denoising Diffusion Probabilistic Models.\n- We condition during inference on the given image content.\n\n![Denoising Diffusion Probabilistic Models Inpainting Method](https://user-images.githubusercontent.com/11280511/180631151-59b6674b-bf2c-4501-8307-03c9f5f593ae.gif)\n\n**Intuition of one conditioned denoising step:**\n1) **Sample the known part:** Add gaussian noise to the known regions of the image. \u003cbr\u003e We obtain a noisy image that follows the denoising process exactly.\n2) **Denoise one step:** Denoise the previous image for one step. This generates  \u003cbr\u003e content for the unknown region conditioned on the known region.\n3) **Join:** Merge the images from both steps.\n\nDetails are in Algorithm 1 on Page 5. [[Paper]](https://bit.ly/3b1ABEb)\n\n\n\u003cbr\u003e\n\n# How to harmonize the generated with the known part?\n\n- **Fail:** When using only the algorithm above, the filling is not well harmonized with the known part (n=1).\n- **Fix:** When applying the [[4.2 Resampling]](https://bit.ly/3b1ABEb) technique, the images are better harmonized (n\u003e1).\n\n\u003cimg width=\"1577\" alt=\"Diffusion Model Resampling\" src=\"https://user-images.githubusercontent.com/11280511/150822917-737c00b0-b6bb-439d-a5bf-e73238d30990.png\"\u003e\n\n\u003cbr\u003e\n\n# RePaint Fails\n- The ImageNet model is biased towards inpainting dogs.\n- This is due to the high ratio of dog images in ImageNet.\n\n\u003cimg width=\"1653\" alt=\"RePaint Fails\" src=\"https://user-images.githubusercontent.com/11280511/150853163-b965f59c-5ad4-485b-816e-4391e77b5199.png\"\u003e\n\n\u003cbr\u003e\n\n# User Study State-of-the-Art Comparison\n\n- Outperforms autoregression-based and GAN-based SOTA methods, \u003cbr\u003e with 95% significance for all masks except for two inconclusive cases.\n- The user study was done for six different masks on three datasets.\n- RePaint outperformed SOTA methods in 42 of 44 cases. [[Paper]](https://bit.ly/3b1ABEb)\n\n\u003cbr\u003e\n\n# Explore the Visual Examples\n- Datasets: CelebA-HQ, ImageNet, Places2\n- Masks: Random strokes, half image, huge, sparse\n- Explore more examples like this in the [[Appendix]](https://bit.ly/3b1ABEb).\n\n\n\u003cimg width=\"1556\" alt=\"Denosing Diffusion Inpainting Examples\" src=\"https://user-images.githubusercontent.com/11280511/150864677-0eb482ae-c114-4b0b-b1e0-9be9574da307.png\"\u003e\n\n\n\u003cbr\u003e\n\n\n# Acknowledgement\n\nThis work was supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Nvidia GPU grant.\n\nThis repository is based on [guided-diffuion](https://github.com/openai/guided-diffusion.git) from OpenAI.\n","funding_links":[],"categories":["图像恢复","Python"],"sub_categories":["网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreas128%2FRePaint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreas128%2FRePaint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreas128%2FRePaint/lists"}