{"id":13657352,"url":"https://github.com/krasserm/super-resolution","last_synced_at":"2025-05-16T06:05:49.080Z","repository":{"id":38010180,"uuid":"153440744","full_name":"krasserm/super-resolution","owner":"krasserm","description":"Tensorflow 2.x based implementation of EDSR, WDSR and SRGAN for single image super-resolution","archived":false,"fork":false,"pushed_at":"2022-05-13T11:47:05.000Z","size":49318,"stargazers_count":1509,"open_issues_count":57,"forks_count":352,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-08T16:02:00.902Z","etag":null,"topics":["edsr","keras","single-image-super-resolution","srgan","super-resolution","tensorflow","tensorflow2","wdsr"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krasserm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-17T10:46:49.000Z","updated_at":"2025-03-15T20:41:19.000Z","dependencies_parsed_at":"2022-07-12T00:01:49.618Z","dependency_job_id":null,"html_url":"https://github.com/krasserm/super-resolution","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fsuper-resolution","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fsuper-resolution/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fsuper-resolution/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krasserm%2Fsuper-resolution/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krasserm","download_url":"https://codeload.github.com/krasserm/super-resolution/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478188,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["edsr","keras","single-image-super-resolution","srgan","super-resolution","tensorflow","tensorflow2","wdsr"],"created_at":"2024-08-02T05:00:41.384Z","updated_at":"2025-05-16T06:05:44.069Z","avatar_url":"https://github.com/krasserm.png","language":"Python","readme":"![Travis CI](https://travis-ci.com/krasserm/super-resolution.svg?branch=master)\n\n# Single Image Super-Resolution with EDSR, WDSR and SRGAN\n\nA [Tensorflow 2.x](https://www.tensorflow.org/beta) based implementation of\n\n- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/abs/1707.02921) (EDSR), winner \n  of the [NTIRE 2017](http://www.vision.ee.ethz.ch/ntire17/) super-resolution challenge.\n- [Wide Activation for Efficient and Accurate Image Super-Resolution](https://arxiv.org/abs/1808.08718) (WDSR), winner \n  of the [NTIRE 2018](http://www.vision.ee.ethz.ch/ntire18/) super-resolution challenge (realistic tracks).\n- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://arxiv.org/abs/1609.04802) (SRGAN).\n\nThis is a complete re-write of the old Keras/Tensorflow 1.x based implementation available [here](https://github.com/krasserm/super-resolution/tree/previous).\nSome parts are still work in progress but you can already train models as described in the papers via a high-level training \nAPI. Furthermore, you can also [fine-tune](#srgan-for-fine-tuning-edsr-and-wdsr-models) EDSR and WDSR models in an SRGAN \ncontext. [Training](#training) and [usage](#getting-started) examples are given in the notebooks\n\n- [example-edsr.ipynb](example-edsr.ipynb)\n- [example-wdsr.ipynb](example-wdsr.ipynb)\n- [example-srgan.ipynb](example-srgan.ipynb) \n\nA `DIV2K` [data provider](#div2k-dataset) automatically downloads [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/) \ntraining and validation images of given scale (2, 3, 4 or 8) and downgrade operator (\"bicubic\", \"unknown\", \"mild\" or \n\"difficult\"). \n\n**Important:** if you want to evaluate the pre-trained models with a dataset other than DIV2K please read \n[this comment](https://github.com/krasserm/super-resolution/issues/19#issuecomment-586114933) (and replies) first.  \n\n## Environment setup\n\nCreate a new [conda](https://conda.io) environment with\n\n    conda env create -f environment.yml\n    \nand activate it with\n\n    conda activate sisr\n\n## Introduction\n\nYou can find an introduction to single-image super-resolution in [this article](https://krasserm.github.io/2019/09/04/super-resolution/). \nIt also demonstrates how EDSR and WDSR models can be fine-tuned with SRGAN (see also [this section](#srgan-for-fine-tuning-edsr-and-wdsr-models)).\n\n## Getting started \n\nExamples in this section require following pre-trained weights for running (see also example notebooks):  \n\n### Pre-trained weights\n\n- [weights-edsr-16-x4.tar.gz](https://martin-krasser.de/sisr/weights-edsr-16-x4.tar.gz) \n    - EDSR x4 baseline as described in the EDSR paper: 16 residual blocks, 64 filters, 1.52M parameters. \n    - PSNR on DIV2K validation set = 28.89 dB (images 801 - 900, 6 + 4 pixel border included).\n- [weights-wdsr-b-32-x4.tar.gz](https://martin-krasser.de/sisr/weights-wdsr-b-32-x4.tar.gz) \n    - WDSR B x4 custom model: 32 residual blocks, 32 filters, expansion factor 6, 0.62M parameters. \n    - PSNR on DIV2K validation set = 28.91 dB (images 801 - 900, 6 + 4 pixel border included).\n- [weights-srgan.tar.gz](https://martin-krasser.de/sisr/weights-srgan.tar.gz) \n    - SRGAN as described in the SRGAN paper: 1.55M parameters, trained with VGG54 content loss.\n    \nAfter download, extract them in the root folder of the project with\n\n    tar xvfz weights-\u003c...\u003e.tar.gz\n\n### EDSR\n\n```python\nfrom model import resolve_single\nfrom model.edsr import edsr\n\nfrom utils import load_image, plot_sample\n\nmodel = edsr(scale=4, num_res_blocks=16)\nmodel.load_weights('weights/edsr-16-x4/weights.h5')\n\nlr = load_image('demo/0851x4-crop.png')\nsr = resolve_single(model, lr)\n\nplot_sample(lr, sr)\n```\n\n![result-edsr](docs/images/result-edsr.png)\n\n### WDSR\n\n```python\nfrom model.wdsr import wdsr_b\n\nmodel = wdsr_b(scale=4, num_res_blocks=32)\nmodel.load_weights('weights/wdsr-b-32-x4/weights.h5')\n\nlr = load_image('demo/0829x4-crop.png')\nsr = resolve_single(model, lr)\n\nplot_sample(lr, sr)\n```\n\n![result-wdsr](docs/images/result-wdsr.png)\n\nWeight normalization in WDSR models is implemented with the new `WeightNormalization` layer wrapper of \n[Tensorflow Addons](https://github.com/tensorflow/addons). In its latest version, this wrapper seems to \ncorrupt weights when running `model.predict(...)`. A workaround is to set `model.run_eagerly = True` or \ncompile the model with `model.compile(loss='mae')` in advance. This issue doesn't arise when calling the\nmodel directly with `model(...)` though. To be further investigated ... \n\n### SRGAN\n\n```python\nfrom model.srgan import generator\n\nmodel = generator()\nmodel.load_weights('weights/srgan/gan_generator.h5')\n\nlr = load_image('demo/0869x4-crop.png')\nsr = resolve_single(model, lr)\n\nplot_sample(lr, sr)\n```\n\n![result-srgan](docs/images/result-srgan.png)\n\n## DIV2K dataset\n\nFor training and validation on [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/) images, applications should use the \nprovided `DIV2K` data loader. It automatically downloads DIV2K images to `.div2k` directory and converts them to a \ndifferent format for faster loading.\n\n### Training dataset\n\n```python\nfrom data import DIV2K\n\ntrain_loader = DIV2K(scale=4,             # 2, 3, 4 or 8\n                     downgrade='bicubic', # 'bicubic', 'unknown', 'mild' or 'difficult' \n                     subset='train')      # Training dataset are images 001 - 800\n                     \n# Create a tf.data.Dataset          \ntrain_ds = train_loader.dataset(batch_size=16,         # batch size as described in the EDSR and WDSR papers\n                                random_transform=True, # random crop, flip, rotate as described in the EDSR paper\n                                repeat_count=None)     # repeat iterating over training images indefinitely\n\n# Iterate over LR/HR image pairs                                \nfor lr, hr in train_ds:\n    # .... \n```\n\nCrop size in HR images is 96x96. \n\n### Validation dataset\n\n```python\nfrom data import DIV2K\n\nvalid_loader = DIV2K(scale=4,             # 2, 3, 4 or 8\n                     downgrade='bicubic', # 'bicubic', 'unknown', 'mild' or 'difficult' \n                     subset='valid')      # Validation dataset are images 801 - 900\n                     \n# Create a tf.data.Dataset          \nvalid_ds = valid_loader.dataset(batch_size=1,           # use batch size of 1 as DIV2K images have different size\n                                random_transform=False, # use DIV2K images in original size \n                                repeat_count=1)         # 1 epoch\n                                \n# Iterate over LR/HR image pairs                                \nfor lr, hr in valid_ds:\n    # ....                                 \n```\n\n## Training \n\nThe following training examples use the [training and validation datasets](#div2k-dataset) described earlier. The high-level \ntraining API is designed around *steps* (= minibatch updates) rather than *epochs* to better match the descriptions in the \npapers.\n\n## EDSR\n\n```python\nfrom model.edsr import edsr\nfrom train import EdsrTrainer\n\n# Create a training context for an EDSR x4 model with 16 \n# residual blocks.\ntrainer = EdsrTrainer(model=edsr(scale=4, num_res_blocks=16), \n                      checkpoint_dir=f'.ckpt/edsr-16-x4')\n                      \n# Train EDSR model for 300,000 steps and evaluate model\n# every 1000 steps on the first 10 images of the DIV2K\n# validation set. Save a checkpoint only if evaluation\n# PSNR has improved.\ntrainer.train(train_ds,\n              valid_ds.take(10),\n              steps=300000, \n              evaluate_every=1000, \n              save_best_only=True)\n              \n# Restore from checkpoint with highest PSNR.\ntrainer.restore()\n\n# Evaluate model on full validation set.\npsnr = trainer.evaluate(valid_ds)\nprint(f'PSNR = {psnr.numpy():3f}')\n\n# Save weights to separate location.\ntrainer.model.save_weights('weights/edsr-16-x4/weights.h5')                                    \n```\n\nInterrupting training and restarting it again resumes from the latest saved checkpoint. The trained Keras model can be\naccessed with `trainer.model`.\n\n## WDSR\n\n```python\nfrom model.wdsr import wdsr_b\nfrom train import WdsrTrainer\n\n# Create a training context for a WDSR B x4 model with 32 \n# residual blocks.\ntrainer = WdsrTrainer(model=wdsr_b(scale=4, num_res_blocks=32), \n                      checkpoint_dir=f'.ckpt/wdsr-b-8-x4')\n\n# Train WDSR B model for 300,000 steps and evaluate model\n# every 1000 steps on the first 10 images of the DIV2K\n# validation set. Save a checkpoint only if evaluation\n# PSNR has improved.\ntrainer.train(train_ds,\n              valid_ds.take(10),\n              steps=300000, \n              evaluate_every=1000, \n              save_best_only=True)\n\n# Restore from checkpoint with highest PSNR.\ntrainer.restore()\n\n# Evaluate model on full validation set.\npsnr = trainer.evaluate(valid_ds)\nprint(f'PSNR = {psnr.numpy():3f}')\n\n# Save weights to separate location.\ntrainer.model.save_weights('weights/wdsr-b-32-x4/weights.h5')\n```\n\n## SRGAN\n\n### Generator pre-training\n\n```python\nfrom model.srgan import generator\nfrom train import SrganGeneratorTrainer\n\n# Create a training context for the generator (SRResNet) alone.\npre_trainer = SrganGeneratorTrainer(model=generator(), checkpoint_dir=f'.ckpt/pre_generator')\n\n# Pre-train the generator with 1,000,000 steps (100,000 works fine too). \npre_trainer.train(train_ds, valid_ds.take(10), steps=1000000, evaluate_every=1000)\n\n# Save weights of pre-trained generator (needed for fine-tuning with GAN).\npre_trainer.model.save_weights('weights/srgan/pre_generator.h5')\n```\n\n### Generator fine-tuning (GAN)\n\n```python\nfrom model.srgan import generator, discriminator\nfrom train import SrganTrainer\n\n# Create a new generator and init it with pre-trained weights.\ngan_generator = generator()\ngan_generator.load_weights('weights/srgan/pre_generator.h5')\n\n# Create a training context for the GAN (generator + discriminator).\ngan_trainer = SrganTrainer(generator=gan_generator, discriminator=discriminator())\n\n# Train the GAN with 200,000 steps.\ngan_trainer.train(train_ds, steps=200000)\n\n# Save weights of generator and discriminator.\ngan_trainer.generator.save_weights('weights/srgan/gan_generator.h5')\ngan_trainer.discriminator.save_weights('weights/srgan/gan_discriminator.h5')\n```\n\n## SRGAN for fine-tuning EDSR and WDSR models\n\nIt is also possible to fine-tune EDSR and WDSR x4 models with SRGAN. They can be used as drop-in replacement for the\noriginal SRGAN generator. More details in [this article](https://krasserm.github.io/2019/09/04/super-resolution/).\n\n```python\n# Create EDSR generator and init with pre-trained weights\ngenerator = edsr(scale=4, num_res_blocks=16)\ngenerator.load_weights('weights/edsr-16-x4/weights.h5')\n\n# Fine-tune EDSR model via SRGAN training.\ngan_trainer = SrganTrainer(generator=generator, discriminator=discriminator())\ngan_trainer.train(train_ds, steps=200000)\n```\n\n```python\n# Create WDSR B generator and init with pre-trained weights\ngenerator = wdsr_b(scale=4, num_res_blocks=32)\ngenerator.load_weights('weights/wdsr-b-16-32/weights.h5')\n\n# Fine-tune WDSR B  model via SRGAN training.\ngan_trainer = SrganTrainer(generator=generator, discriminator=discriminator())\ngan_trainer.train(train_ds, steps=200000)\n```\n","funding_links":[],"categories":["Model Implementations","Model 💛💛💛💛💛\u003ca name=\"Model\" /\u003e"],"sub_categories":["Super resolution 超分辨率"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrasserm%2Fsuper-resolution","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrasserm%2Fsuper-resolution","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrasserm%2Fsuper-resolution/lists"}