{"id":19786481,"url":"https://github.com/cszn/usrnet","last_synced_at":"2025-04-13T00:48:28.588Z","repository":{"id":37314475,"uuid":"243782399","full_name":"cszn/USRNet","owner":"cszn","description":"Deep Unfolding Network for Image Super-Resolution (CVPR, 2020) (PyTorch)","archived":false,"fork":false,"pushed_at":"2023-07-31T00:00:56.000Z","size":30989,"stargazers_count":881,"open_issues_count":11,"forks_count":115,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-04-13T00:48:20.190Z","etag":null,"topics":["bicubic-kernels","blur-kernels","deblurring","degradation-model","end-to-end-learning","gaussian-kernel","generalizability","motion-kernel","noise-levels","scale-factors","sisr","super-resolution","unfolding-algorithm"],"latest_commit_sha":null,"homepage":"https://cszn.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cszn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-28T14:34:53.000Z","updated_at":"2025-04-05T14:17:35.000Z","dependencies_parsed_at":"2022-07-13T15:59:18.267Z","dependency_job_id":"afcc276f-d832-43f7-9972-5ce5f61dec5e","html_url":"https://github.com/cszn/USRNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cszn%2FUSRNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cszn%2FUSRNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cszn%2FUSRNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cszn%2FUSRNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cszn","download_url":"https://codeload.github.com/cszn/USRNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650437,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bicubic-kernels","blur-kernels","deblurring","degradation-model","end-to-end-learning","gaussian-kernel","generalizability","motion-kernel","noise-levels","scale-factors","sisr","super-resolution","unfolding-algorithm"],"created_at":"2024-11-12T06:18:14.170Z","updated_at":"2025-04-13T00:48:28.568Z","avatar_url":"https://github.com/cszn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Deep unfolding network for image super-resolution](https://arxiv.org/pdf/2003.10428.pdf)\n![visitors](https://visitor-badge.glitch.me/badge?page_id=cszn/USRNet) \n\n\n[Kai Zhang](https://cszn.github.io/), [Luc Van Gool](https://vision.ee.ethz.ch/people-details.OTAyMzM=.TGlzdC8zMjQ4LC0xOTcxNDY1MTc4.html), [Radu Timofte](http://people.ee.ethz.ch/~timofter/)  \n_[Computer Vision Lab](https://vision.ee.ethz.ch/the-institute.html), ETH Zurich, Switzerland_\n\n[[Paper](http://openaccess.thecvf.com/content_CVPR_2020/papers/Zhang_Deep_Unfolding_Network_for_Image_Super-Resolution_CVPR_2020_paper.pdf)][[Code](https://github.com/cszn/USRNet/blob/9d1cb1c3244b40828c9058b594ed03d43e306bc3/models/network_usrnet.py#L309)]\n\n\n___________\n\n[[Training code --\u003e KAIR](https://github.com/cszn/KAIR)]\n\n`git clone https://github.com/cszn/KAIR.git`\n\n- Training with `DataParallel` - PSNR\n```python\npython main_train_psnr.py --opt options/train_usrnet.json\n```\n- Training with `DistributedDataParallel` - PSNR - 4 GPUs\n```python\npython -m torch.distributed.launch --nproc_per_node=4 --master_port=1234 main_train_psnr.py --opt options/train_usrnet.json  --dist True\n```\n\n___________\n\n* [Classical SISR degradation model](#classical-sisr-degradation-model)\n* [Motivation](#motivation)\n* [Unfolding algorithm](#unfolding-algorithm)\n* [Deep unfolding SR network](#deep-unfolding-sr-network)\n* [Models](#models)\n* [Codes](#codes)\n* [Blur kernels](#blur-kernels)\n* [Approximated bicubic kernel under classical SR degradation model assumption](#approximated-bicubic-kernel-under-classical-sr-degradation-model-assumption)\n* [PSNR results](#psnr-results)\n* [Visual results of USRNet](#visual-results-of-usrnet)\n* [Visual results of USRGAN](#visual-results-of-usrgan)\n* [Results for bicubic degradation](#results-for-bicubic-degradation)\n* [Results for deblurring](#results-for-deblurring)\n* [Generalizability](#generalizability)\n* [Real image SR](#real-image-sr)\n* [Citation](#citation)\n\n\n\nClassical SISR degradation model\n----------\nFor a scale factor of $\\mathbf{s}$, the classical (traditional) degradation model of SISR assumes the low-resolution (LR) image $\\mathbf{y}$ is a blurred, decimated, and noisy version of a high-resolution (HR) image $\\mathbf{x}$. Mathematically, it can be expressed by\n\n$$\\mathbf{y}=\\left(\\mathbf{x}\\otimes\\mathbf{k}\\right)\\downarrow_{\\mathrm{{s}}}+\\mathbf{n}$$\n\nwhere $\\otimes$ represents two-dimensional convolution of $\\mathbf{x}$ with blur kernel $\\mathbf{k}$, $\\downarrow_{\\mathrm{{s}}}$ denotes the standard $\\mathbf{s}$-fold downsampler, i.e., keeping the upper-left pixel for each distinct $\\mathbf{s}\\times \\mathbf{s}$ patch and discarding the others, and n is usually assumed to be additive, white Gaussian noise (AWGN) specified by standard deviation (or noise level) $\\mathbf{\\sigma}$. With a clear physical meaning, it can approximate a variety of LR images by setting proper blur kernels, scale factors and noises for underlying HR images. In particular, it has been extensively studied in model-based methods which solve a combination of a data term and a prior term under the MAP framework. Especially noteworthy is that it turns into a special case for deblurring when $\\mathbf{s} = 1$.\n\n\n\n\nMotivation\n----------\n\u003cimg src=\"figs/category.png\" width=\"536px\"/\u003e\n\nLearning-based single image super-resolution (SISR)\nmethods are continuously showing superior effectiveness\nand efficiency over traditional model-based methods, largely\ndue to the end-to-end training. However, different from\nmodel-based methods that can handle the SISR problem\nwith different scale factors, blur kernels and noise levels\nunder a unified MAP (maximum a posteriori) framework,\nlearning-based methods (e.g., SRMD [3]) generally lack such flexibility.\n\n```\n[1] \"Learning deep CNN denoiser prior for image restoration.\" CVPR, 2017.\n[2] \"Deep plug-and-play super-resolution for arbitrary blur kernels.\" CVPR, 2019.\n[3] \"Learning a single convolutional super-resolution network for multiple degradations.\" CVPR, 2018.\n```\n\n\u003cimg src=\"figs/fig1.png\" width=\"440px\"/\u003e\n\n\nWhile the classical degradation model can result in various LR images for an HR image, with different blur kernels, scale factors and noise, the study of learning *`a single end-to-end trained deep model`* to invert all such LR images to HR image is still lacking.\n\n\n**_This work focuses on `non-blind SISR` which assumes the LR image, scale factor, blur kernel and noise level are known beforehand. In fact, non-blind SISR is still an active research direction._**  \n* _First, the blur kernel and noise level can be estimated, or are known based on other information (e.g.,\ncamera setting)._\n* _Second, users can control the preference of sharpness and smoothness by tuning the blur kernel and\nnoise level._ \n* _Third, non-blind SISR can be an intermediate step towards solving blind SISR._\n\n\n\nUnfolding algorithm\n----------\nBy unfolding the MAP inference via a half-quadratic splitting\nalgorithm, a fixed number of iterations consisting of alternately solving a `data subproblem` and a `prior subproblem`\ncan be obtained.\n\n#TODO\n\n\n\nDeep unfolding SR network\n----------\nWe propose an end-to-end trainable unfolding network which leverages both learning-based\nmethods and model-based methods. \nUSRNet inherits the `flexibility of model-based methods` to super-resolve\nblurry, noisy images for different scale factors via `a single\nmodel`, while maintaining the `advantages of learning-based methods`.\n\n\u003cimg src=\"figs/architecture.png\" width=\"900px\"/\u003e \n\nThe overall architecture of the proposed USRNet with 8 iterations. USRNet can flexibly handle the classical degradation\nvia `a single model` as it takes the LR image, scale factor, blur kernel and noise level as input. Specifically, USRNet consists of three main modules, including the _**data module D**_ that makes HR estimation clearer, the _**prior module P**_ that makes HR estimation cleaner, and the _**hyper-parameter module H**_ that controls the outputs of _**D**_ and _**P**_.\n\n* **_Data module D:_** _closed-form solution for the data term; contains no trainable parameters_\n* **_Prior module P:_** _ResUNet denoiser for the prior term_\n* **_Hyper-parameter module H:_** _MLP for the hyper-parameter; acts as a slide bar to control the outputs of **D** and **P**_\n\n\nModels\n----------\n\n|Model|# iters|# params|ResUNet|\n|---|:--:|:---:|:---:|\n|[USRNet](https://drive.google.com/file/d/1qz8aaYOAMhoKn07VppFjRsDflYpxeVmz/view?usp=sharing)     | 8 | 17.02M |64-128-256-512|\n|[USRGAN](https://drive.google.com/file/d/1R5HKJzHJmou-3iUYd4cNLSyMeWXSOmD4/view?usp=sharing)     | 8 | 17.02M |64-128-256-512|\n|[USRNet-tiny](https://drive.google.com/file/d/1xPN26OW5YBN9-5QfK2BGnaovSutH7fLX/view?usp=sharing)| 6 | 0.59M  |16-32-64-64   |\n|[USRGAN-tiny](https://drive.google.com/file/d/1U4BV42Qf0dtBOVbhUNRay6di3j_ioZVM/view?usp=sharing)| 6 | 0.59M  |16-32-64-64   |\n\n\n\nCodes\n----------\n\n* [main_test_table1.py](main_test_table1.py):  _Code to produce the results in Table 1_\n* [main_test_bicubic.py](main_test_bicubic.py):  _Code to super-resolve LR images by bicubic degradation and produce the results in Table 2_\n* [main_test_realapplication.py](main_test_realapplication.py):  _Code to super-resolve real LR images_\n\n\nBlur kernels\n----------\n\n|\u003cimg src=\"figs/isotropic_gaussian.gif\" width=\"285px\"/\u003e|\u003cimg src=\"figs/anisotropic_gaussian.gif\" width=\"285px\"/\u003e|\u003cimg src=\"figs/motion.gif\" width=\"285px\"/\u003e|\n|:---:|:---:|:---:|\n|\u003ci\u003e(a) Isotropic Gaussian kernels\u003c/i\u003e|\u003ci\u003e(b) Anisotropic Gaussian kernels\u003c/i\u003e|\u003ci\u003e(c) Motion blur kernels\u003c/i\u003e|\n\nWhile it has been pointed out that anisotropic Gaussian kernels are enough for SISR task, the SISR method that\ncan handle more complex blur kernels would be a preferred choice in real applications.\n\n\nApproximated bicubic kernel under classical SR degradation model assumption\n----------\n\n|\u003cimg src=\"figs/bicubic_kernelx2.png\" width=\"285px\"/\u003e|\u003cimg src=\"figs/bicubic_kernelx3.png\" width=\"285px\"/\u003e|\u003cimg src=\"figs/bicubic_kernelx4.png\" width=\"285px\"/\u003e|\n|:---:|:---:|:---:|\n|\u003ci\u003e(a) [Bicubic kernel](kernels/kernels_bicubicx234.mat) (x2)\u003c/i\u003e|\u003ci\u003e(b) [Bicubic kernel](kernels/kernels_bicubicx234.mat) (x3)\u003c/i\u003e|\u003ci\u003e(c) [Bicubic kernel](kernels/kernels_bicubicx234.mat) (x4)\u003c/i\u003e|\n\nThe bicubic degradation can be approximated by setting a proper blur kernel for the classical degradation. Note that the bicubic kernels contain negative values.\n\n\nPSNR results\n-----------\n\nRun [main_test_table1.py](main_test_table1.py) to produce the following results.\n\n\u003cimg src=\"figs/psnr.png\" width=\"900px\"/\u003e  \nThe table shows the average PSNR(dB) results of different methods for different combinations of scale factors, blur kernels and noise levels.\n\n\nVisual results of USRNet\n----------\n\n\n\u003cimg align=\"left\" src=\"figs/butterfly_x2_k10_LR.png\" width=\"240px\"/\u003e  \u003cimg align=\"center\" src=\"figs/butterfly_x3_k2_LR.png\" width=\"240px\"/\u003e  \u003cimg align=\"right\" src=\"figs/butterfly_x4_k7_LR.png\" width=\"240px\"/\u003e\n\u003cp align=\"center\"\u003e\u003ci\u003e(a) LR images with scale factors 2, 3 and 4\u003c/i\u003e\u003c/p\u003e\n\n\n\u003cimg align=\"left\" src=\"figs/butterfly_x2_k10_usrnet.png\" width=\"240px\"/\u003e \u003cimg align=\"center\" src=\"figs/butterfly_x3_k2_usrnet.png\" width=\"240px\"/\u003e \u003cimg align=\"right\" src=\"figs/butterfly_x4_k7_usrnet.png\" width=\"240px\"/\u003e\n\u003cp align=\"center\"\u003e\u003ci\u003e(b) Results by the single USRNet model with s = 2, 3 and 4\u003c/i\u003e\u003c/p\u003e\n\n\nVisual results of USRGAN\n----------\n\n\u003cimg align=\"left\" src=\"figs/parrot_x4_k3_LR.png\" width=\"240px\"/\u003e  \u003cimg align=\"center\" src=\"figs/parrot_x4_k6_LR.png\" width=\"240px\"/\u003e  \u003cimg align=\"right\" src=\"figs/parrot_x4_k12_LR.png\" width=\"240px\"/\u003e\n\u003cp align=\"center\"\u003e\u003ci\u003e(a) LR images\u003c/i\u003e\u003c/p\u003e\n\n\n\u003cimg align=\"left\" src=\"figs/parrot_x4_k3_usrgan.png\" width=\"240px\"/\u003e \u003cimg align=\"center\" src=\"figs/parrot_x4_k6_usrgan.png\" width=\"240px\"/\u003e \u003cimg align=\"right\" src=\"figs/parrot_x4_k12_usrgan.png\" width=\"240px\"/\u003e\n\u003cp align=\"center\"\u003e\u003ci\u003e(b) Results by USRGAN(x4)\u003c/i\u003e\u003c/p\u003e\n\n\n|\u003cimg align=\"center\" src=\"figs/test_57_x4_k1_LR.png\" width=\"448px\"/\u003e | \u003cimg align=\"center\" src=\"figs/test_57_x4_k1_usrgan.png\" width=\"448px\"/\u003e |\n|:---:|:---:|\n|\u003cimg align=\"center\" src=\"figs/test_27_x4_k3_LR.png\" width=\"448px\"/\u003e | \u003cimg align=\"center\" src=\"figs/test_27_x4_k3_usrgan.png\" width=\"448px\"/\u003e |\n|\u003cimg align=\"center\" src=\"figs/test_14_x4_k5_LR.png\" width=\"448px\"/\u003e| \u003cimg align=\"center\" src=\"figs/test_14_x4_k5_usrgan.png\" width=\"448px\"/\u003e|\n|\u003cimg align=\"center\" src=\"figs/test_36_x4_k9_LR.png\" width=\"448px\"/\u003e| \u003cimg align=\"center\" src=\"figs/test_36_x4_k9_usrgan.png\" width=\"448px\"/\u003e|\n|\u003ci\u003e(a) LR images\u003c/i\u003e|\u003ci\u003e(b) Results by USRGAN(x4)\u003c/i\u003e|\n\n\n\nResults for bicubic degradation\n----------\n\nRun [main_test_bicubic.py](main_test_bicubic.py) to produce the following results.\n\nBy taking the approximated bicubic blur kernel as input, USRNet and USRGAN achieve very promising results for bicubic degradation. Note that the bicubic kernels are not adopted in training.\n\n* PSNR results of USRNet for bicubic degradation\n\n| Model |  Scale factor | Set5 | Set14 | BSD100 | Urban100 |\n|---|:---:|:---:|:---:|:---:|:---:|\n|       | x2 |  37.72  | 33.49  |  32.10  | 31.79  |\n|USRNet | x3 |  34.45  | 30.51  |  29.18  | 28.38  |\n|       | x4 |  32.45  | 28.83  |  27.69  | 26.44  |\n\n* Visual results of USRGAN for bicubic degradation\n\n\u003cimg src=\"figs/test_19_LR_x4.png\" width=\"120px\"/\u003e \u003cimg src=\"figs/test_19_x4.png\" width=\"480px\"/\u003e \n\n\u003cimg src=\"figs/test_35_LR_x4.png\" width=\"120px\"/\u003e \u003cimg src=\"figs/test_35_x4.png\" width=\"480px\"/\u003e \n\n\u003cimg src=\"figs/test_42_LR_x4.png\" width=\"120px\"/\u003e \u003cimg src=\"figs/test_42_x4.png\" width=\"480px\"/\u003e \n\n\u003ci\u003e(a) LR images via bicubic degradation\u003c/i\u003e; \u003ci\u003e(b) results by USRGAN(x4)\u003c/i\u003e\n\n\nResults for deblurring\n----------\n\nBy setting the scale factor to 1, USRNet can handle deblurring problem.\n\n|\u003cimg align=\"center\" src=\"figs/test_09_x1_k12_LR.png\" width=\"448px\"/\u003e | \u003cimg align=\"center\" src=\"figs/test_09_x1_k12_usrnet.png\" width=\"448px\"/\u003e |\n|:---:|:---:|\n|\u003cimg align=\"center\" src=\"figs/test_16_x1_k12_LR.png\" width=\"448px\"/\u003e | \u003cimg align=\"center\" src=\"figs/test_16_x1_k12_usrnet.png\" width=\"448px\"/\u003e |\n|\u003ci\u003e(a) Blurry images\u003c/i\u003e|\u003ci\u003e(b) Results by USRNet(x1)\u003c/i\u003e|\n\n\n\n\nGeneralizability\n----------\n\n\u003cimg src=\"figs/g1_LR.png\" width=\"150px\"/\u003e \u003cimg src=\"figs/g1_E.png\" width=\"450px\"/\u003e \n\n\u003ci\u003e(a) LR image with kernel size 67x67; (b) result by USRNet(x3)\u003c/i\u003e\n\n_**Even trained with kernel size 25x25, USRNet generalizes well to much larger kernel size.**_\n\n\u003cimg src=\"figs/g2_LR.png\" width=\"150px\"/\u003e \u003cimg src=\"figs/g2_E.png\" width=\"450px\"/\u003e \n\n\u003ci\u003e(a) LR image with kernel size 70x70; (b) result by USRGAN(x3)\u003c/i\u003e  \n\n_**Even trained with kernel size 25x25 and scale factor 4, USRGAN generalizes well to much larger kernel size and another scale factor 3.**_\n\n\n\nReal image SR\n----------\n\nRun [main_test_realapplication.py](main_test_realapplication.py) to produce the following results.\n\n| \u003cimg src=\"figs/chip.png\" width=\"75px\"/\u003e|\u003cimg src=\"figs/chip_x1.png\" width=\"75px\"/\u003e |\u003cimg src=\"figs/chip_x2.png\" width=\"150px\"/\u003e | \u003cimg src=\"figs/chip_x3.png\" width=\"225px\"/\u003e|\u003cimg src=\"figs/chip_x4.png\" width=\"300px\"/\u003e|\n|:---:|:---:|:---:|:---:|:---:|\n|\u003ci\u003eLR\u003c/i\u003e|\u003ci\u003eUSRNet(x1)\u003c/i\u003e|\u003ci\u003eUSRNet(x2)\u003c/i\u003e|\u003ci\u003eUSRNet(x3)\u003c/i\u003e|\u003ci\u003eUSRNet(x4)\u003c/i\u003e|\n\nThe above results are obtained via `a single USRNet model` by setting different scale factors (x1, x2, x3, x4) and Gaussian blur kernels (with width 0.6, 0.9, 1.7, 2.2).\n\n|\u003cimg src=\"figs/butterfly.bmp\" width=\"430x\"/\u003e |\u003cimg src=\"figs/butterfly_x2.bmp\" width=\"430x\"/\u003e|\n|:---:|:---:|\n|\u003ci\u003eZoomed real LR image Butterfly, 256x256\u003c/i\u003e|\u003ci\u003eResult by USRNet(x2), 512x512\u003c/i\u003e|\n\n|\u003cimg src=\"figs/comic.png\" width=\"430x\"/\u003e|\u003cimg src=\"figs/comic_x2.png\" width=\"430x\"/\u003e| \n|:---:|:---:|\n|\u003ci\u003eZoomed real LR image Comic, 250x361\u003c/i\u003e|\u003ci\u003eResult by USRNet(x2), 500x722\u003c/i\u003e|\n\n\n\nCitation\n----------\n```BibTex\n@inproceedings{zhang2020deep, % USRNet\n  title={Deep unfolding network for image super-resolution},\n  author={Zhang, Kai and Van Gool, Luc and Timofte, Radu},\n  booktitle={IEEE Conference on Computer Vision and Pattern Recognition},\n  pages={3217--3226},\n  year={2020}\n}\n```\n\nAcknowledgments\n----------\n```This work was partly supported by the ETH Zurich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Amazon AWS grant.```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcszn%2Fusrnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcszn%2Fusrnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcszn%2Fusrnet/lists"}