Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/swz30/MIRNet
[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.
https://github.com/swz30/MIRNet
attention-mechanism computer-vision eccv2020 image-denoising image-enhancement image-restoration low-level-vision multi-resolution-streams pytorch super-resolution
Last synced: 10 days ago
JSON representation
[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.
- Host: GitHub
- URL: https://github.com/swz30/MIRNet
- Owner: swz30
- License: other
- Created: 2020-03-11T12:31:15.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-04-18T08:03:52.000Z (7 months ago)
- Last Synced: 2024-08-02T11:19:20.079Z (3 months ago)
- Topics: attention-mechanism, computer-vision, eccv2020, image-denoising, image-enhancement, image-restoration, low-level-vision, multi-resolution-streams, pytorch, super-resolution
- Language: Python
- Homepage:
- Size: 52.7 KB
- Stars: 648
- Watchers: 14
- Forks: 92
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Learning Enriched Features for Real Image Restoration and Enhancement (ECCV 2020)
[Syed Waqas Zamir](https://scholar.google.ae/citations?hl=en&user=POoai-QAAAAJ), [Aditya Arora](https://adityac8.github.io/), [Salman Khan](https://salman-h-khan.github.io/), [Munawar Hayat](https://scholar.google.com/citations?user=Mx8MbWYAAAAJ&hl=en), [Fahad Shahbaz Khan](https://scholar.google.es/citations?user=zvaeYnUAAAAJ&hl=en), [Ming-Hsuan Yang](https://scholar.google.com/citations?user=p9-ohHsAAAAJ&hl=en), [Ling Shao](https://scholar.google.com/citations?user=z84rLjoAAAAJ&hl=en)
[](https://arxiv.org/abs/2003.06792)
[](https://drive.google.com/file/d/1QIKp7h7Rd85odaS6bDoeDGXb0VLKo8I9/view?usp=sharing)
[](https://www.youtube.com/watch?v=6xSzRjAodv4)
[](https://drive.google.com/file/d/1hnhqSrjqQQiYn7XPAGpFgMBTfBlb1QAy/view?usp=sharing)
### News
- A lightweight, fast and extended version of MIRNet is accepted in **TPAMI**. [Paper](https://www.waqaszamir.com/publication/zamir-2022-mirnetv2/) | [Code](https://github.com/swz30/MIRNetv2)
- Keras Tutorial on MIRNet is available at https://keras.io/examples/vision/mirnet/
- Video on Tensorflow Youtube channel https://youtu.be/BMza5yrwZ9s
- Links to (unofficial) implementations are added [here](#other-implementations)
> **Abstract:** *With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network, and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In the nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for a variety of image processing tasks, including image denoising, super-resolution and image enhancement.*
Network Architecture (click to expand)
Overall Framework of MIRNet
Selective Kernel Feature Fusion (SKFF)
Downsampling Module
Dual Attention Unit (DAU)
Upsampling Module
## Installation
The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).
For installing, follow these intructions
```
sudo apt-get install cmake build-essential libjpeg-dev libpng-dev
conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image opencv-python yacs joblib natsort h5py tqdm
```
## Training
1. Download the SIDD-Medium dataset from [here](https://www.eecs.yorku.ca/~kamel/sidd/dataset.php)
2. Generate image patches
```
python generate_patches_SIDD.py --ps 256 --num_patches 300 --num_cores 10
```
3. Download validation images of SIDD and place them in `../SIDD_patches/val`
4. Install warmup scheduler
```
cd pytorch-gradual-warmup-lr; python setup.py install; cd ..
```
5. Train your model with default arguments by running
```
python train_denoising.py
```
**Note:** Our model is trained with 2 Nvidia Tesla-V100 GPUs. See [#5](https://github.com/swz30/MIRNet/issues/5) for changing the model parameters.
## Evaluation
You can download, at once, the complete repository of MIRNet (including pre-trained models, datasets, results, etc) from this Google Drive [link](https://drive.google.com/drive/folders/1C2XCufoxxckQ29EkxERFPxL8R3Kx68ZG?usp=sharing), or evaluate individual tasks with the following instructions:
### Image Denoising
- Download the [model](https://drive.google.com/file/d/13PGkg3yaFQCvz6ytN99Heh_yyvfxRCdG/view?usp=sharing) and place it in ./pretrained_models/denoising/
#### Testing on SIDD dataset
- Download sRGB [images](https://drive.google.com/drive/folders/1j5ESMU0HJGD-wU6qbEdnt569z7sM3479?usp=sharing) of SIDD and place them in ./datasets/sidd/
- Run
```
python test_sidd_rgb.py --save_images
```
#### Testing on DND dataset
- Download sRGB [images](https://drive.google.com/drive/folders/1-IBw_J0gdlM6AlqSm3Z7XWTXR-So4xzp?usp=sharing) of DND and place them in ./datasets/dnd/
- Run
```
python test_dnd_rgb.py --save_images
```
### Image Super-resolution
- Download the [models](https://drive.google.com/drive/folders/1yMtXbk6RXoFfmeRRGu1XfNFSHH6bSUoR?usp=sharing) and place them in ./pretrained_models/super_resolution/
- Download [images](https://drive.google.com/drive/folders/1mAr0YCqBJFXsnOnOp0WWxkAiGF9DQAe8?usp=sharing) of different scaling factor and place them in ./datasets/super_resolution/
- Run
```
python test_super_resolution.py --save_images --scale 3
python test_super_resolution.py --save_images --scale 4
```
### Image Enhancement
#### Testing on LOL dataset
- Download the LOL [model](https://drive.google.com/file/d/1t_FcBuMZD5th2KWVVNXYGJ7bMz5ZAWvF/view?usp=sharing) and place it in ./pretrained_models/enhancement/
- Download [images](https://drive.google.com/drive/folders/1LR6J4tkG6DLHqsipsMgHgU_p1xOZjdAA?usp=sharing) of LOL dataset and place them in ./datasets/lol/
- Run
```
python test_enhancement.py --save_images --input_dir ./datasets/lol/ --result_dir ./results/enhancement/lol/ --weights ./pretrained_models/enhancement/model_lol.pth
```
#### Testing on Adobe-MIT FiveK dataset
- Download the FiveK [model](https://drive.google.com/file/d/1BsXOvhMz2z80E_V93dgD6QaEspZE0w-u/view?usp=sharing) and place it in ./pretrained_models/enhancement/
- Download some sample [images](https://drive.google.com/drive/folders/1tyrELge59GdhZ18VR6yFwVb5Kenq2hSd?usp=sharing) of fiveK dataset and place them in ./datasets/fivek_sample_images/
- Run
```
python test_enhancement.py --save_images --input_dir ./datasets/fivek_sample_images/ --result_dir ./results/enhancement/fivek/ --weights ./pretrained_models/enhancement/model_fivek.pth
```
## Results
Experiments are performed on five real image datasets for different image processing tasks including, image denoising, super-resolution and image enhancement. Images produced by MIRNet can be downloaded from Google Drive [link](https://drive.google.com/drive/folders/1z6bFP7ydBaQOPmk8n1byYY0xcLx7aBHp?usp=sharing).
Image Denoising (click to expand)
Image Super-resolution (click to expand)
Image Enhancement (click to expand)
## Other Implementations
- [Tensorflow](https://github.com/soumik12345/MIRNet) (Soumik Rakshit)
- [Tensorflow-JS](https://github.com/Rishit-dagli/MIRNet-TFJS) (Rishit Dagli)
- [Tensorflow-TFLite](https://github.com/sayakpaul/MIRNet-TFLite-TRT) (Sayak Paul)
## Citation
If you use MIRNet, please consider citing:
@inproceedings{Zamir2020MIRNet,
title={Learning Enriched Features for Real Image Restoration and Enhancement},
author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
booktitle={ECCV},
year={2020}
}
## Contact
Should you have any question, please contact [email protected]
## Our Related Works
- Learning Enriched Features for Fast Image Restoration and Enhancement, TPAMI 2022. [Paper](https://www.waqaszamir.com/publication/zamir-2022-mirnetv2/) | [Code](https://github.com/swz30/MIRNetv2)
- Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. [Paper](https://arxiv.org/abs/2111.09881) | [Code](https://github.com/swz30/Restormer)
- Multi-Stage Progressive Image Restoration, CVPR 2021. [Paper](https://arxiv.org/abs/2102.02808) | [Code](https://github.com/swz30/MPRNet)
- CycleISP: Real Image Restoration via Improved Data Synthesis, CVPR 2020. [Paper](https://arxiv.org/abs/2003.07761) | [Code](https://github.com/swz30/CycleISP)