{"id":13730657,"url":"https://idealo.github.io/image-super-resolution/","last_synced_at":"2025-05-08T03:31:33.401Z","repository":{"id":37773939,"uuid":"159175746","full_name":"idealo/image-super-resolution","owner":"idealo","description":"🔎 Super-scale your images and run experiments with Residual Dense and Adversarial Networks.","archived":false,"fork":false,"pushed_at":"2024-03-12T11:21:52.000Z","size":15341,"stargazers_count":4655,"open_issues_count":107,"forks_count":766,"subscribers_count":102,"default_branch":"master","last_synced_at":"2024-11-11T12:03:04.503Z","etag":null,"topics":["aws","computer-vision","convolutional-neural-networks","deep-learning","docker","e-commerce","idealo","image-processing","image-super-resolution","keras","machine-learning","neural-network","nvidia-docker","super-resolution","tensorflow"],"latest_commit_sha":null,"homepage":"https://idealo.github.io/image-super-resolution/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idealo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-11-26T13:41:13.000Z","updated_at":"2024-11-11T08:27:32.000Z","dependencies_parsed_at":"2024-01-13T19:47:04.308Z","dependency_job_id":"9c90d975-caff-48fa-be77-2f06c09f07b6","html_url":"https://github.com/idealo/image-super-resolution","commit_stats":{"total_commits":126,"total_committers":9,"mean_commits":14.0,"dds":0.3571428571428571,"last_synced_commit":"3f6498cf1ac4dba162a52c5861aa9d90b7c2fe35"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealo%2Fimage-super-resolution","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealo%2Fimage-super-resolution/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealo%2Fimage-super-resolution/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idealo%2Fimage-super-resolution/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idealo","download_url":"https://codeload.github.com/idealo/image-super-resolution/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224695639,"owners_count":17354443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","computer-vision","convolutional-neural-networks","deep-learning","docker","e-commerce","idealo","image-processing","image-super-resolution","keras","machine-learning","neural-network","nvidia-docker","super-resolution","tensorflow"],"created_at":"2024-08-03T02:01:17.716Z","updated_at":"2025-05-08T03:31:33.388Z","avatar_url":"https://github.com/idealo.png","language":"Python","funding_links":[],"categories":["Image Upscaling"],"sub_categories":["Creative Uses of Generative AI Image Synthesis Tools"],"readme":"# Image Super-Resolution (ISR)\n\n\u003cimg src=\"figures/butterfly.png\"\u003e\n\n[![Build Status](https://travis-ci.org/idealo/image-super-resolution.svg?branch=master)](https://travis-ci.org/idealo/image-super-resolution)\n[![Docs](https://img.shields.io/badge/docs-online-brightgreen)](https://idealo.github.io/image-super-resolution/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-orange.svg)](https://github.com/idealo/image-super-resolution/blob/master/LICENSE)\n\nThe goal of this project is to upscale and improve the quality of low resolution images.\n\nSince the code is no longer actively maintained, it will be archived on 2025-01-03.\n\nThis project contains Keras implementations of different Residual Dense Networks for Single Image Super-Resolution (ISR) as well as scripts to train these networks using content and adversarial loss components.  \n\nThe implemented networks include:\n\n- The super-scaling Residual Dense Network described in [Residual Dense Network for Image Super-Resolution](https://arxiv.org/abs/1802.08797) (Zhang et al. 2018)\n- The super-scaling Residual in Residual Dense Network described in [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://arxiv.org/abs/1809.00219) (Wang et al. 2018)\n- A multi-output version of the Keras VGG19 network for deep features extraction used in the perceptual loss\n- A custom discriminator network based on the one described in [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://arxiv.org/abs/1609.04802) (SRGANS, Ledig et al. 2017)\n\nRead the full documentation at: [https://idealo.github.io/image-super-resolution/](https://idealo.github.io/image-super-resolution/).\n\n[Docker scripts](https://idealo.github.io/image-super-resolution/tutorials/docker/) and [Google Colab notebooks](https://github.com/idealo/image-super-resolution/tree/master/notebooks) are available to carry training and prediction. Also, we provide scripts to facilitate training on the cloud with AWS and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker) with only a few commands.\n\nISR is compatible with Python 3.6 and is distributed under the Apache 2.0 license. We welcome any kind of contribution. If you wish to contribute, please see the [Contribute](#contribute) section.\n\n## Contents\n- [Pre-trained networks](#pre-trained-networks)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Additional Information](#additional-information)\n- [Contribute](#contribute)\n- [Citation](#citation)\n- [Maintainers](#maintainers)\n- [License](#copyright)\n\n## Troubleshooting\n### Training not delivering good/patchy results\nWhen training your own model, start with only PSNR loss (50+ epochs, depending on the dataset) and only then introduce GANS and feature loss. This can be controlled by the loss weights argument.\n\nThis is just sample, you will need to tune these parameters.\n\nPSNR only:\n```\nloss_weights = {\n  'generator': 1.0,\n  'feature_extractor': 0.0,\n  'discriminator': 0.00\n}\n```\n\nLater:\n```\nloss_weights = {\n  'generator': 0.0,\n  'feature_extractor': 0.0833,\n  'discriminator': 0.01\n}\n```\n### Weights loading\nIf you are having trouble loading your own weights or the pre-trained weights (`AttributeError: 'str' object has no attribute 'decode'`), try:\n```bash\npip install 'h5py==2.10.0' --force-reinstall\n```\n[Issue](https://github.com/idealo/image-super-resolution/issues/197#issue-877826405)\n\n## Pre-trained networks\n\nThe weights used to produced these images are available directly when creating the model object. \n\nCurrently 4 models are available:\n  - RDN: psnr-large, psnr-small, noise-cancel\n  - RRDN: gans\n \nExample usage:\n\n```\nmodel = RRDN(weights='gans')\n```\n  \nThe network parameters will be automatically chosen.\n(see [Additional Information](#additional-information)).\n\n#### Basic model\nRDN model, PSNR driven, choose the option ```weights='psnr-large'``` or ```weights='psnr-small'``` when creating a RDN model.\n\n|![butterfly-sample](figures/butterfly_comparison_SR_baseline.png)|\n|:--:|\n| Low resolution image (left), ISR output (center), bicubic scaling (right). Click to zoom. |\n#### GANS model\nRRDN model, trained with Adversarial and VGG features losses, choose the option ```weights='gans'``` when creating a RRDN model.\n\n|![baboon-comparison](figures/baboon-compare.png)|\n|:--:|\n| RRDN GANS model (left), bicubic upscaling (right). |\n-\u003e [more detailed comparison](http://www.framecompare.com/screenshotcomparison/PGZPNNNX)\n\n#### Artefact Cancelling GANS model\nRDN model, trained with Adversarial and VGG features losses, choose the option ```weights='noise-cancel'``` when creating a RDN model.\n\n|![temple-comparison](figures/temple_comparison.png)|\n|:--:|\n| Standard vs GANS model. Click to zoom. |\n\n\n|![sandal-comparison](figures/sandal-compare.png)|\n|:--:|\n| RDN GANS artefact cancelling model (left), RDN standard PSNR driven model (right). |\n-\u003e [more detailed comparison](http://www.framecompare.com/screenshotcomparison/2ECCNNNU)\n\n\n## Installation\nThere are two ways to install the Image Super-Resolution package:\n\n- Install ISR from PyPI (recommended):\n```\npip install ISR\n```\n- Install ISR from the GitHub source:\n```\ngit clone https://github.com/idealo/image-super-resolution\ncd image-super-resolution\npython setup.py install\n```\n\n## Usage\n\n### Prediction\n\nLoad image and prepare it\n```python\nimport numpy as np\nfrom PIL import Image\n\nimg = Image.open('data/input/test_images/sample_image.jpg')\nlr_img = np.array(img)\n```\n\nLoad a pre-trained model and run prediction (check the prediction tutorial under notebooks for more details)\n```python\nfrom ISR.models import RDN\n\nrdn = RDN(weights='psnr-small')\nsr_img = rdn.predict(lr_img)\nImage.fromarray(sr_img)\n```\n\n#### Large image inference\nTo predict on large images and avoid memory allocation errors, use the `by_patch_of_size` option for the predict method, for instance\n```\nsr_img = model.predict(image, by_patch_of_size=50)\n```\nCheck the documentation of the `ImageModel` class for further details.\n\n### Training\n\nCreate the models\n```python\nfrom ISR.models import RRDN\nfrom ISR.models import Discriminator\nfrom ISR.models import Cut_VGG19\n\nlr_train_patch_size = 40\nlayers_to_extract = [5, 9]\nscale = 2\nhr_train_patch_size = lr_train_patch_size * scale\n\nrrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':64, 'G0':64, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)\nf_ext = Cut_VGG19(patch_size=hr_train_patch_size, layers_to_extract=layers_to_extract)\ndiscr = Discriminator(patch_size=hr_train_patch_size, kernel_size=3)\n```\n\nCreate a Trainer object using the desired settings and give it the models (`f_ext` and `discr` are optional)\n```python\nfrom ISR.train import Trainer\nloss_weights = {\n  'generator': 0.0,\n  'feature_extractor': 0.0833,\n  'discriminator': 0.01\n}\nlosses = {\n  'generator': 'mae',\n  'feature_extractor': 'mse',\n  'discriminator': 'binary_crossentropy'\n}\n\nlog_dirs = {'logs': './logs', 'weights': './weights'}\n\nlearning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}\n\nflatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}\n\ntrainer = Trainer(\n    generator=rrdn,\n    discriminator=discr,\n    feature_extractor=f_ext,\n    lr_train_dir='low_res/training/images',\n    hr_train_dir='high_res/training/images',\n    lr_valid_dir='low_res/validation/images',\n    hr_valid_dir='high_res/validation/images',\n    loss_weights=loss_weights,\n    learning_rate=learning_rate,\n    flatness=flatness,\n    dataname='image_dataset',\n    log_dirs=log_dirs,\n    weights_generator=None,\n    weights_discriminator=None,\n    n_validation=40,\n)\n```\n\nStart training\n```python\ntrainer.train(\n    epochs=80,\n    steps_per_epoch=500,\n    batch_size=16,\n    monitored_metrics={'val_PSNR_Y': 'max'}\n)\n```\n\n## Additional Information\nYou can read about how we trained these network weights in our Medium posts:\n- part 1: [A deep learning based magnifying glass](https://medium.com/idealo-tech-blog/a-deep-learning-based-magnifying-glass-dae1f565c359)\n- part 2: [Zoom in... enhance](https://medium.com/idealo-tech-blog/zoom-in-enhance-a-deep-learning-based-magnifying-glass-part-2-c021f98ebede\n)\n\n### RDN Pre-trained weights\nThe weights of the RDN network trained on the [DIV2K dataset](https://data.vision.ee.ethz.ch/cvl/DIV2K) are available in ```weights/sample_weights/rdn-C6-D20-G64-G064-x2/PSNR-driven/rdn-C6-D20-G64-G064-x2_PSNR_epoch086.hdf5```. \u003cbr\u003e\nThe model was trained using ```C=6, D=20, G=64, G0=64``` as parameters (see architecture for details) for 86 epochs of 1000 batches of 8 32x32 augmented patches taken from LR images.\n\nThe artefact can cancelling weights obtained with a combination of different training sessions using different datasets and perceptual loss with VGG19 and GAN can be found at `weights/sample_weights/rdn-C6-D20-G64-G064-x2/ArtefactCancelling/rdn-C6-D20-G64-G064-x2_ArtefactCancelling_epoch219.hdf5`\nWe recommend using these weights only when cancelling compression artefacts is a desirable effect.\n\n### RDN Network architecture\nThe main parameters of the architecture structure are:\n- D - number of Residual Dense Blocks (RDB)\n- C - number of convolutional layers stacked inside a RDB\n- G - number of feature maps of each convolutional layers inside the RDBs\n- G0 - number of feature maps for convolutions outside of RDBs and of each RBD output\n\n\u003cimg src=\"figures/RDN.png\" width=\"600\"\u003e\n\u003cbr\u003e\n\n\u003cimg src=\"figures/RDB.png\" width=\"600\"\u003e\n\nsource: [Residual Dense Network for Image Super-Resolution](https://arxiv.org/abs/1802.08797)\n\n### RRDN Network architecture\nThe main parameters of the architecture structure are:\n- T - number of Residual in Residual Dense Blocks (RRDB)\n- D - number of Residual Dense Blocks (RDB) insider each RRDB\n- C - number of convolutional layers stacked inside a RDB\n- G - number of feature maps of each convolutional layers inside the RDBs\n- G0 - number of feature maps for convolutions outside of RDBs and of each RBD output\n\n\u003cimg src=\"figures/RRDN.jpg\" width=\"600\"\u003e\n\u003cbr\u003e\n\n\u003cimg src=\"figures/RRDB.png\" width=\"600\"\u003e\n\nsource: [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://arxiv.org/abs/1809.00219)\n\n## Contribute\nWe welcome all kinds of contributions, models trained on different datasets, new model architectures and/or hyperparameters combinations that improve the performance of the currently published model.\n\nWill publish the performances of new models in this repository.\n\nSee the [Contribution](CONTRIBUTING.md) guide for more details.\n\n#### Bump version\nTo bump up the version, use\n```\nbumpversion {part} setup.py\n```\n\n## Citation\nPlease cite our work in your publications if it helps your research.\n\n```BibTeX\n@misc{cardinale2018isr,\n  title={ISR},\n  author={Francesco Cardinale et al.},\n  year={2018},\n  howpublished={\\url{https://github.com/idealo/image-super-resolution}},\n}\n```\n\n## Maintainers\n* Francesco Cardinale, github: [cfrancesco](https://github.com/cfrancesco)\n* Dat Tran, github: [datitran](https://github.com/datitran)\n\n## Copyright\n\nSee [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/idealo.github.io%2Fimage-super-resolution%2F","html_url":"https://awesome.ecosyste.ms/projects/idealo.github.io%2Fimage-super-resolution%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/idealo.github.io%2Fimage-super-resolution%2F/lists"}