{"id":15028326,"url":"https://github.com/wyhuai/ddnm","last_synced_at":"2025-05-16T06:07:13.828Z","repository":{"id":64089596,"uuid":"558368772","full_name":"wyhuai/DDNM","owner":"wyhuai","description":"[ICLR 2023 Oral] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model","archived":false,"fork":false,"pushed_at":"2024-04-25T08:28:06.000Z","size":14642,"stargazers_count":1259,"open_issues_count":51,"forks_count":94,"subscribers_count":25,"default_branch":"main","last_synced_at":"2025-05-16T06:07:03.448Z","etag":null,"topics":["diffusion-models","iclr","iclr2023","image-restoration","zero-shot"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wyhuai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-27T12:14:17.000Z","updated_at":"2025-05-16T02:04:26.000Z","dependencies_parsed_at":"2024-04-25T09:48:34.418Z","dependency_job_id":null,"html_url":"https://github.com/wyhuai/DDNM","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyhuai%2FDDNM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyhuai%2FDDNM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyhuai%2FDDNM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyhuai%2FDDNM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wyhuai","download_url":"https://codeload.github.com/wyhuai/DDNM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478190,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion-models","iclr","iclr2023","image-restoration","zero-shot"],"created_at":"2024-09-24T20:08:02.524Z","updated_at":"2025-05-16T06:07:08.810Z","avatar_url":"https://github.com/wyhuai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model\n## 📖[**Paper**](https://arxiv.org/pdf/2212.00490.pdf)|🖼️[**Project Page**](https://wyhuai.github.io/ddnm.io/)| \u003ca href=\"https://colab.research.google.com/drive/1SRSD6GXGqU0eO2CoTNY-2WykB9qRZHJv?usp=sharing\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/hysts/DDNM-HQ)\n\n[Yinhuai Wang*](https://wyhuai.github.io/info/), [Jiwen Yu*](https://scholar.google.com.hk/citations?user=uoRPLHIAAAAJ), [Jian Zhang](https://jianzhang.tech/)  \nPeking University and PCL  \n\\*denotes equal contribution\n\n\n\nThis repository contains the code release for *Zero-Shot Image Restoration Using ***D***enoising ***D***iffusion ***N***ull-Space ***M***odel*. **DDNM** can solve various image restoration tasks **without any optimization or training! Yes, in a zero-shot manner**.\n\n\n***Supported Applications:***\n- **Arbitrary Size**🆕\n- **Old Photo Restoration**🆕\n- Super-Resolution\n- Denoising\n- Colorization\n- Inpainting\n- Deblurring\n- Compressed Sensing\n\n\n![front](https://user-images.githubusercontent.com/95485229/227095293-1024f337-1fde-494b-ae82-97d6139bbefe.jpg)\n\n\n\n## 🧩News\n- A Colab demo for high-quality results is now avaliable! \u003ca href=\"https://colab.research.google.com/drive/1SRSD6GXGqU0eO2CoTNY-2WykB9qRZHJv?usp=sharing\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e\n\n## Installation\n### Code\n```\ngit clone https://github.com/wyhuai/DDNM.git\n```\n### Environment\n```\npip install numpy torch blobfile tqdm pyYaml pillow    # e.g. torch 1.7.1+cu110.\n```\n### Pre-Trained Models\nTo restore human face images, download this [model](https://drive.google.com/file/d/1wSoA5fm_d6JBZk4RZ1SzWLMgev4WqH21/view?usp=share_link)(from [SDEdit](https://github.com/ermongroup/SDEdit)) and put it into `DDNM/exp/logs/celeba/`. \n```\nhttps://drive.google.com/file/d/1wSoA5fm_d6JBZk4RZ1SzWLMgev4WqH21/view?usp=share_link\n```\nTo restore general images, download this [model](https://openaipublic.blob.core.windows.net/diffusion/jul-2021/256x256_diffusion_uncond.pt)(from [guided-diffusion](https://github.com/openai/guided-diffusion)) and put it into `DDNM/exp/logs/imagenet/`.\n```\nwget https://openaipublic.blob.core.windows.net/diffusion/jul-2021/256x256_diffusion_uncond.pt\n```\n### Quick Start\nRun below command to get 4x SR results immediately. The results should be in `DDNM/exp/image_samples/demo`.\n```\npython main.py --ni --simplified --config celeba_hq.yml --path_y celeba_hq --eta 0.85 --deg \"sr_averagepooling\" --deg_scale 4.0 --sigma_y 0 -i demo\n```\n\n## Setting\nThe detailed sampling command is here:\n```\npython main.py --ni --simplified --config {CONFIG}.yml --path_y {PATH_Y} --eta {ETA} --deg {DEGRADATION} --deg_scale {DEGRADATION_SCALE} --sigma_y {SIGMA_Y} -i {IMAGE_FOLDER}\n```\nwith following options:\n- We implement **TWO** versions of DDNM in this repository. One is SVD-based version, which is more precise in solving noisy tasks. Another one is the simplified version, which does not involve SVD and is flexible for users to define their own degradations. Use `--simplified` to activate the simplified DDNM. Without `--simplified` will turn to the SVD-based DDNM.\n- `PATH_Y` is the folder name of the test dataset, in `DDNM/exp/datasets`.\n- `ETA` is the DDIM hyperparameter. (default: `0.85`)\n- `DEGREDATION` is the supported tasks including `cs_walshhadamard`, `cs_blockbased`, `inpainting`, `denoising`, `deblur_uni`, `deblur_gauss`, `deblur_aniso`, `sr_averagepooling`,`sr_bicubic`, `colorization`, `mask_color_sr`, and user-defined `diy`.\n- `DEGRADATION_SCALE` is the scale of degredation. e.g., `--deg sr_bicubic --deg_scale 4` lead to 4xSR.\n- `SIGMA_Y` is the noise observed in y.\n- `CONFIG` is the name of the config file (see `configs/` for a list), including hyperparameters such as batch size and sampling step.\n- `IMAGE_FOLDER` is the folder name of the results.\n\nFor the config files, e.g., celeba_hq.yml, you may change following properties:\n```\nsampling:\n    batch_size: 1\n    \ntime_travel:\n    T_sampling: 100     # sampling steps\n    travel_length: 1    # time-travel parameters l and s, see section 3.3 of the paper.\n    travel_repeat: 1    # time-travel parameter r, see section 3.3 of the paper.\n```\n\n## Reproduce The Results In The Paper\n### Quantitative Evaluation\nDataset download link: [[Google drive](https://drive.google.com/drive/folders/1cSCTaBtnL7OIKXT4SVME88Vtk4uDd_u4?usp=sharing)] [[Baidu drive](https://pan.baidu.com/s/1tQaWBqIhE671v3rrB-Z2mQ?pwd=twq0)]\n\nDownload the CelebA testset and put it into `DDNM/exp/datasets/celeba/`.\n\nDownload the ImageNet testset and put it into `DDNM/exp/datasets/imagenet/` and replace the file `DDNM/exp/imagenet_val_1k.txt`.\n\nRun the following command. You may increase the batch_size to accelerate evaluation.\n```\nsh evaluation.sh\n```\n\n### High-Quality Results\nYou can try this [**Colab demo**](https://colab.research.google.com/drive/1SRSD6GXGqU0eO2CoTNY-2WykB9qRZHJv?usp=sharing) for High-Quality results. Note that the High-Quality results presented in the front figure are mostly generated by applying DDNM to the models in [RePaint](https://github.com/andreas128/RePaint).\n\n## 🔥Real-World Applications\n### Demo: Real-World Super-Resolution.\n![orig_62](https://user-images.githubusercontent.com/95485229/204471148-bf155c60-c7b3-4c3a-898c-859cb9d0d723.png)\n![00000](https://user-images.githubusercontent.com/95485229/204971948-7564b536-b562-4187-9d8c-d96db4c55f7c.png)\n\nRun the following command\n```\npython main.py --ni --simplified --config celeba_hq.yml --path_y solvay --eta 0.85 --deg \"sr_averagepooling\" --deg_scale 4.0 --sigma_y 0.1 -i demo\n```\n### Demo: Old Photo Restoration.\n![image](https://user-images.githubusercontent.com/95485229/204973149-4818426b-89af-410c-b1b7-f26b8f65358b.png)\n![image](https://user-images.githubusercontent.com/95485229/204973288-0f245e93-8980-4a32-a5e9-7f2bfe58d8eb.png)\n\nRun the following command\n```\npython main.py --ni --simplified --config oldphoto.yml --path_y oldphoto --eta 0.85 --deg \"mask_color_sr\" --deg_scale 2.0 --sigma_y 0.02 -i demo\n```\n### Try your own photos.\nYou may use DDNM to restore your own degraded images. DDNM provides full flexibility for you to define the degradation operator and the noise level. Note that these definitions are critical for a good results. You may reference the following guidance.\n1. If you are using CelebA pretrained models, try the tool in [GFPGAN](https://github.com/TencentARC/GFPGAN) to crop and align your photo.\n2. If there are local artifacts on your photo, try the tool in [LaMa](https://colab.research.google.com/github/advimman/lama/blob/master/colab/LaMa_inpainting.ipynb#scrollTo=-VZWySTMeGDM) to draw a mask to cover them, and save this mask to `DDNM/exp/inp_masks/mask.png`. Then run `DDNM/exp/inp_masks/get_mask.py` to generate `mask.npy`.\n3. If your photo is faded, you need a grayscale operator as part of the degradation.\n4. If your photo is blur, you need a downsampler operator as part of the degradation. Also, you need to set a proper SR scale `--deg_scale`.\n5. If your photo suffers global artifacts, e.g., jpeg-like artifacts or unkown noise, you need to set a proper `sigma_y` to remove these artifacts.\n6. Search `args.deg =='diy'` in `DDNM/guided_diffusion/diffusion.py` and change the definition of $\\mathbf{A}$ correspondingly.\nThen run\n```\npython main.py --ni --simplified --config celeba_hq.yml --path_y {YOUR_OWN_PATH} --eta 0.85 --deg \"diy\" --deg_scale {YOUR_OWN_SCALE} --sigma_y {YOUR_OWN_LEVEL} -i diy\n```\n\n## 🆕DDNM for Arbitrary Size\n![123456](https://user-images.githubusercontent.com/95485229/206181069-0f134804-63e3-4ba1-9ad1-0a4ccd9dd72e.png)![image](https://user-images.githubusercontent.com/95485229/206333186-c240fad9-3602-46f5-8bc7-a66e8c463196.png)\n\nAbove we show an example of using DDNM to SR a 64x256 input image into a 256x1024 result. The theory details can be found in this [paper](https://arxiv.org/abs/2303.00354), section 3.3.\n\nWe implement the **Mask-Shift Restoration** in the folder `hq_demo`, based on [RePaint](https://github.com/andreas128/RePaint). You can try this [**Colab demo**](https://colab.research.google.com/drive/1SRSD6GXGqU0eO2CoTNY-2WykB9qRZHJv?usp=sharing). \u003ca href=\"https://colab.research.google.com/drive/1SRSD6GXGqU0eO2CoTNY-2WykB9qRZHJv?usp=sharing\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e\n\nOr, you can try this function on your own device, you need to download the pre-trained models:\n```\nwget https://openaipublic.blob.core.windows.net/diffusion/jul-2021/256x256_classifier.pt\nwget https://openaipublic.blob.core.windows.net/diffusion/jul-2021/256x256_diffusion.pt\n```\nand put it to `hq_demo/data/pretrained`. Then run\n```\ncd hq_demo\nsh evaluation.sh\n```\nThis script contains SR results up to **2K** resolution. It may take hours to finish some demos in this script. Setting a smaller sampling step or time-travel parameters in hq_demo/confs/inet256.yml can speed up, but may compromise the generative quality.\n\n\n\n## 😊Applying DDNM to Your Own Diffusion Model\nIt is ***very easy*** to implement a basic DDNM on your own diffusion model! You may reference the following:\n1. Copy these operator implementations to the core diffusion sampling file, then define your task type, e.g., set `IR_mode=\"super resolution\"`.\n```python\ndef color2gray(x):\n    coef=1/3\n    x = x[:,0,:,:] * coef + x[:,1,:,:]*coef +  x[:,2,:,:]*coef\n    return x.repeat(1,3,1,1)\n\ndef gray2color(x):\n    x = x[:,0,:,:]\n    coef=1/3\n    base = coef**2 + coef**2 + coef**2\n    return th.stack((x*coef/base, x*coef/base, x*coef/base), 1)    \n    \ndef PatchUpsample(x, scale):\n    n, c, h, w = x.shape\n    x = torch.zeros(n,c,h,scale,w,scale) + x.view(n,c,h,1,w,1)\n    return x.view(n,c,scale*h,scale*w)\n\n# Implementation of A and its pseudo-inverse Ap    \n    \nif IR_mode==\"colorization\":\n    A = color2gray\n    Ap = gray2color\n    \nelif IR_mode==\"inpainting\":\n    A = lambda z: z*mask\n    Ap = A\n      \nelif IR_mode==\"super resolution\":\n    A = torch.nn.AdaptiveAvgPool2d((256//scale,256//scale))\n    Ap = lambda z: PatchUpsample(z, scale)\n\nelif IR_mode==\"old photo restoration\":\n    A1 = lambda z: z*mask\n    A1p = A1\n    \n    A2 = color2gray\n    A2p = gray2color\n    \n    A3 = torch.nn.AdaptiveAvgPool2d((256//scale,256//scale))\n    A3p = lambda z: PatchUpsample(z, scale)\n    \n    A = lambda z: A3(A2(A1(z)))\n    Ap = lambda z: A1p(A2p(A3p(z)))\n```\n2. Find the variant $\\mathbf{x}\\_{0|t}$ in the target codes, and use the result of this function to modify the sampling of $\\mathbf{x}\\_{t-1}$. Your may need to provide the input degraded image $\\mathbf{y}$ and the corresponding noise level $\\sigma_\\mathbf{y}$.\n```python\n# Core Implementation of DDNM+, simplified denoising solution (Section 3.3).\n# For more accurate denoising, please refer to the paper (Appendix I) and the source code.\n\ndef ddnm_plus_core(x0t, y, sigma_y=0, sigma_t, a_t):\n\n    #Eq 19\n    if sigma_t \u003e= a_t*sigma_y: \n        lambda_t = 1\n        gamma_t = sigma_t**2 - (a_t*lambda_t*sigma_y)**2\n    else:\n        lambda_t = sigma_t/(a_t*sigma_y)\n        gamma_t = 0\n        \n    #Eq 17    \n    x0t= x0t + lambda_t*Ap(y - A(x0t))\n    \n    return x0t, gamma_t\n```\n3. Actually, this repository contains the above simplified implementation: try search `arg.simplified` in `DDNM/guided_diffusion/diffusion.py` for related codes. \n\n# References\nIf you find this repository useful for your research, please cite the following work.\n```\n@article{wang2022zero,\n  title={Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model},\n  author={Wang, Yinhuai and Yu, Jiwen and Zhang, Jian},\n  journal={The Eleventh International Conference on Learning Representations},\n  year={2023}\n}\n```\nThis implementation is based on / inspired by:\n- https://github.com/wyhuai/RND (null-space learning)\n- https://github.com/andreas128/RePaint (time-travel trick)\n- https://github.com/bahjat-kawar/ddrm (code structure)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwyhuai%2Fddnm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwyhuai%2Fddnm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwyhuai%2Fddnm/lists"}