Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/google-research/maxim

[CVPR 2022 Oral] Official repository for "MAXIM: Multi-Axis MLP for Image Processing". SOTA for denoising, deblurring, deraining, dehazing, and enhancement.
https://github.com/google-research/maxim
architecture computer-vision deblurring dehazing denoising deraining enhancement image image-enhancement image-processing image-restoration low-level-vision mlp restoration retouching transformer
Last synced: about 2 months ago
JSON representation
[CVPR 2022 Oral] Official repository for "MAXIM: Multi-Axis MLP for Image Processing". SOTA for denoising, deblurring, deraining, dehazing, and enhancement.
Host: GitHub
URL: https://github.com/google-research/maxim
Owner: google-research
License: apache-2.0
Created: 2022-03-15T05:34:02.000Z (over 2 years ago)
Default Branch: main
Last Pushed: 2023-06-09T18:15:50.000Z (over 1 year ago)
Last Synced: 2024-05-09T17:12:02.724Z (4 months ago)
Topics: architecture, computer-vision, deblurring, dehazing, denoising, deraining, enhancement, image, image-enhancement, image-processing, image-restoration, low-level-vision, mlp, restoration, retouching, transformer
Language: Python
Homepage: https://arxiv.org/abs/2201.02973
Size: 10.5 MB
Stars: 953
Watchers: 14
Forks: 102
Open Issues: 17
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project

README

        [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-hide-trained-on-gopro)](https://paperswithcode.com/sota/deblurring-on-hide-trained-on-gopro?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-gopro)](https://paperswithcode.com/sota/deblurring-on-gopro?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-realblur-j-1)](https://paperswithcode.com/sota/deblurring-on-realblur-j-1?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-realblur-r)](https://paperswithcode.com/sota/deblurring-on-realblur-r?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-realblur-j-trained-on-gopro)](https://paperswithcode.com/sota/deblurring-on-realblur-j-trained-on-gopro?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/deblurring-on-realblur-r-trained-on-gopro)](https://paperswithcode.com/sota/deblurring-on-realblur-r-trained-on-gopro?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/low-light-image-enhancement-on-lol)](https://paperswithcode.com/sota/low-light-image-enhancement-on-lol?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/photo-retouching-on-mit-adobe-5k)](https://paperswithcode.com/sota/photo-retouching-on-mit-adobe-5k?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/single-image-deraining-on-rain100h)](https://paperswithcode.com/sota/single-image-deraining-on-rain100h?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/single-image-deraining-on-rain100l)](https://paperswithcode.com/sota/single-image-deraining-on-rain100l?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/single-image-deraining-on-test100)](https://paperswithcode.com/sota/single-image-deraining-on-test100?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/single-image-deraining-on-test2800)](https://paperswithcode.com/sota/single-image-deraining-on-test2800?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/single-image-deraining-on-test1200)](https://paperswithcode.com/sota/single-image-deraining-on-test1200?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/image-denoising-on-sidd)](https://paperswithcode.com/sota/image-denoising-on-sidd?p=maxim-multi-axis-mlp-for-image-processing)

[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/maxim-multi-axis-mlp-for-image-processing/image-denoising-on-dnd)](https://paperswithcode.com/sota/image-denoising-on-dnd?p=maxim-multi-axis-mlp-for-image-processing)

# MAXIM: Multi-Axis MLP for Image Processing (CVPR 2022 Oral, Best Paper Nomination)

[![Paper](https://img.shields.io/badge/arXiv-Paper-.svg)](https://arxiv.org/abs/2201.02973)

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google-research/maxim/blob/master/colab_inference_demo.ipynb)

[![slides](https://img.shields.io/badge/Presentation-Slides-B762C1)](https://docs.google.com/presentation/d/1NKT0PZrpmsCZTdgvsZztfNUJJ9Bvlr1r/edit?usp=sharing&ouid=103274492054041370194&rtpof=true&sd=true)

[![Poster](https://img.shields.io/badge/Poster-Slide-87CEEB)](https://docs.google.com/presentation/d/1fd73qn_8Ymc5okFttQ3vzQm1SABbIeoI/edit?usp=sharing&ouid=103274492054041370194&rtpof=true&sd=true)

This repo hosts the official implementation of the MAXIM models: 

["MAXIM: Multi-Axis MLP for Image Processing"](https://arxiv.org/abs/2201.02973). CVPR 2022 Oral.\

[Zhengzhong Tu](https://www.linkedin.com/in/vztu/), [Hossein Talebi](https://scholar.google.com/citations?hl=en&user=UOX9BigAAAAJ), [Han Zhang](https://sites.google.com/view/hanzhang), [Feng Yang](https://sites.google.com/view/feng-yang), [Peyman Milanfar](https://sites.google.com/view/milanfarhome/), [Alan Bovik](https://www.ece.utexas.edu/people/faculty/alan-bovik), and [Yinxiao Li](https://scholar.google.com/citations?user=kZsIU74AAAAJ&hl=en)\

Google Research, University of Texas at Austin

*Disclaimer: This is not an officially supported Google product.*

**News**:

- Jan 8, 2023: Released a pytorch implementation. Check it out here: [maxim-pytorch](https://github.com/vztu/maxim-pytorch/tree/main/maxim_pytorch).

- Oct 21, 2022: MAXIM models have been ported to TensorFlow by [@sayakpaul](https://github.com/sayakpaul). Check it out here: [maxim-tf](https://github.com/sayakpaul/maxim-tf). He also created a couple of Hugging Face Spaces to allow users to quickly try out the different models:

  * [Denoising](https://huggingface.co/spaces/sayakpaul/sidd-denoising-maxim)

  * [Low-light enhancement](https://huggingface.co/spaces/sayakpaul/lol-enhancement-maxim)

  * [Image retouching](https://huggingface.co/spaces/sayakpaul/fivek-retouching-maxim)

  * [Dehazing indoors](https://huggingface.co/spaces/sayakpaul/sots-indoor-dehazing-maxim)

  * [Dehazing outdoors](https://huggingface.co/spaces/sayakpaul/sots-outdoor-dehazing-maxim)

  * [Image deraining](https://huggingface.co/spaces/sayakpaul/rain13k-deraining-maxim)

  * [Image deblurring](https://huggingface.co/spaces/sayakpaul/gopro-deblurring-maxim)

- Sep 8, 2022: our Google AI blog covering both [MaxViT](https://arxiv.org/abs/2204.01697) and [MAXIM](https://github.com/google-research/maxim) is [live](https://ai.googleblog.com/2022/09/a-multi-axis-approach-for-vision.html).

- Apr 25, 2022: Added demos.

  - Colab demo by [@deshwalmahesh](https://github.com/deshwalmahesh) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google-research/maxim/blob/master/colab_inference_demo.ipynb)

  - Replicate web demo [![Replicate](https://replicate.com/google-research/maxim/badge)](https://replicate.com/google-research/maxim).

- Jun 22, 2022: MAXIM selected as 1 of the best paper nomination!

- Mar 29, 2022: MAXIM selected for an oral presentation at CVPR 2022!

- Mar 28, 2022: initial push to Github.

- Mar 3, 2022: paper accepted to CVPR 2022!

- Jan 9, 2022: initial uploads to [Arxiv](https://arxiv.org/abs/2201.02973)

## Quick Demos

Try the web demo for Image Denoising, Deblurring, Deraining, Dehazing and Enhancement with customized input image here [![Replicate](https://replicate.com/google-research/maxim/badge)](https://replicate.com/google-research/maxim)

Try the Colab here using [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google-research/maxim/blob/master/colab_inference_demo.ipynb)

## Architecture

![Model overview](maxim/images/overview.png)

## Installation

Install dependencies:

```

pip install -r requirements.txt

```

Setup project:

```

pip install .

```

## Results and Pre-trained models

We provide the pre-trained models and visual results.

Please contact us if you have any questions or requests.

| Task | Dataset | PSNR | SSIM | Model | #params | FLOPs | ckpt | outputs |

|:---:|:---:|:---:|:---:| :---:|:---:|:---:|:---:|:---:|

| Denoising | SIDD | 39.96 | 0.960 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Denoising/SIDD/) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Denoising/SIDD/) |

| Denoising | DND  | 39.84 | 0.954 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Denoising/SIDD/) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Denoising/DND/) |

| Deblurring | GoPro | 32.86 | 0.961 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deblurring/GoPro) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Deblurring/GoPro/) |

| Deblurring | HIDE  | 32.83 | 0.956 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deblurring/GoPro) | images  |

| Deblurring | REDS  | 28.93 | 0.865 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deblurring/REDS) | images  |

| Deblurring | RealBlur-R | 39.45 | 0.962 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deblurring/RealBlur_R) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Deblurring/RealBlur/) |

| Deblurring | RealBlur-J | 32.84 | 0.935 | MAXIM-3S | 22.2M | 339G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deblurring/RealBlur_J) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Deblurring/RealBlur/) |

| Deraining | Rain13k | 33.24 | 0.933 | MAXIM-2S | 14.1M | 216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deraining/Rain13k) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Deraining/Rain13k/) |

| Deraining | Raindrop | 31.87 | 0.935 | MAXIM-2S | 14.1M | 216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Deraining/Raindrop) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Deraining/Raindrop/) |

| Dehazing | RESIDE-Indoor | 38.11 | 0.991 | MAXIM-2S | 14.1M | 216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Dehazing/SOTS-Indoor) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Dehazing/RESIDE-Indoor/) |

| Dehazing | RESIDE-Outdoor | 34.19 | 0.985 | MAXIM-2S | 14.1M | 216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Dehazing/SOTS-Outdoor) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Dehazing/RESIDE-Outdoor/) |

| Enhancement | LOL | 23.43 | 0.863 | MAXIM-2S | 14.1M | 216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Enhancement/LOL) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Enhancement/LOL/) |

| Enhancement | FiveK | 26.15 | 0.945 | MAXIM-2S | 14.1M  |  216G | [ckpt](https://console.cloud.google.com/storage/browser/gresearch/maxim/ckpt/Enhancement/FiveK) | [images](https://console.cloud.google.com/storage/browser/gresearch/maxim/results/Enhancement/FiveK/) |

## Demo

First download corresponding checkpoints and then go ahead and run:

  Image Denoising (click to expand) 

```

python3 maxim/run_eval.py --task Denoising --ckpt_path ${SIDD_CKPT_PATH} \

  --input_dir maxim/images/Denoising --output_dir maxim/images/Results --has_target=False

```

  Image Deblurring (click to expand) 

```

python3 maxim/run_eval.py --task Deblurring --ckpt_path ${GOPRO_CKPT_PATH} \

  --input_dir maxim/images/Deblurring --output_dir maxim/images/Results --has_target=False

```

  Image Deraining (click to expand) 

Rain streak:

```

python3 maxim/run_eval.py --task Deraining --ckpt_path ${RAIN13K_CKPT_PATH} \

  --input_dir maxim/images/Deraining --output_dir maxim/images/Results --has_target=False

```

Rain drop:

```

python3 maxim/run_eval.py --task Deraining --ckpt_path ${RAINDROP_CKPT_PATH} \

  --input_dir maxim/images/Deraining --output_dir maxim/images/Results --has_target=False

```

  Image Dehazing (click to expand) 

Indoor:

```

python3 maxim/run_eval.py --task Dehazing --ckpt_path ${REDISE_INDOOR_CKPT_PATH} \

  --input_dir maxim/images/Dehazing --output_dir maxim/images/Results --has_target=False

```

Outdoor:

```

python3 maxim/run_eval.py --task Dehazing --ckpt_path ${REDISE_OUTDOOR_CKPT_PATH} \

  --input_dir maxim/images/Dehazing --output_dir maxim/images/Results --has_target=False

```

  Image Enhancement (click to expand) 

Low-light enhancement:

```

python3 maxim/run_eval.py --task Enhancement --ckpt_path ${LOL_CKPT_PATH} \

  --input_dir maxim/images/Enhancement --output_dir maxim/images/Results --has_target=False

```

Retouching:

```

python3 maxim/run_eval.py --task Enhancement --ckpt_path ${FIVEK_CKPT_PATH} \

  --input_dir maxim/images/Enhancement --output_dir maxim/images/Results --has_target=False

```

## Results

  Image Denoising (click to expand) 



Image Deblurring (click to expand) 

  

      

      

  

  

    
Synthetic blur

    Realistic blur

  

Image Deraining (click to expand) 

  

      

     
  

  

  

    Rain streak

    Rain drop

  

Image Dehazing (click to expand) 



Image Enhancement (click to expand) 



## Citation

Should you find this repository useful, please consider citing:

```

@article{tu2022maxim,

  title={MAXIM: Multi-Axis MLP for Image Processing},

  author={Tu, Zhengzhong and Talebi, Hossein and Zhang, Han and Yang, Feng and Milanfar, Peyman and Bovik, Alan and Li, Yinxiao},

  journal={CVPR},

  year={2022},

}

```

## Acknowledgement

This repository is built on the [vision_transformer](https://github.com/google-research/vision_transformer) and [musiq](https://github.com/google-research/google-research/tree/master/musiq) repositories. Our work is also inspired by [HiT](https://github.com/google-research/hit-gan), [MPRNet](https://github.com/swz30/MPRNet), and [HINet](https://github.com/megvii-model/HINet).