Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/MingTian99/ESDNet

Learning A Spiking Neural Network for Efficient Image Deraining (IJCAI 2024)
https://github.com/MingTian99/ESDNet

Last synced: 2 months ago
JSON representation

Learning A Spiking Neural Network for Efficient Image Deraining (IJCAI 2024)

Host: GitHub
URL: https://github.com/MingTian99/ESDNet
Owner: MingTian99
Created: 2024-05-09T20:12:23.000Z (8 months ago)
Default Branch: master
Last Pushed: 2024-05-11T03:55:54.000Z (8 months ago)
Last Synced: 2024-05-11T21:43:33.695Z (8 months ago)
Language: Python
Size: 380 KB
Stars: 5
Watchers: 2
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

Awesome-Spiking-Neural-Networks - [code

README

# 【IJCAI'2024🔥】Learning A Spiking Neural Network for Efficient Image Deraining

> Learning A Spiking Neural Network for Efficient Image Deraining
>
> [Tianyu Song](https://scholar.google.com/citations?user=wA3Op6cAAAAJ&hl=zh-CN), Guiyue Jin, [Pengpeng Li](https://github.com/halamadrid-lpp),[Kui Jiang](https://homepage.hit.edu.cn/jiangkui), [Xiang Chen](https://cschenxiang.github.io/), Jiyu Jin
>
> Dalian Polytechnic University, Nanjing University of Science and Technology and Harbin Institute of Technology
>
> Primary contact: Tianyu Song([email protected])

> **Abstract:**
Recently, spiking neural networks (SNNs) have demonstrated substantial potential in computer vision tasks.
In this paper, we present an Efficient Spiking Deraining Network, called ESDNet.
Our work is motivated by the observation that rain pixel values will lead to a more pronounced intensity of spike signals in SNNs. However, directly applying deep SNNs to image deraining task still remains a significant challenge.
This is attributed to the information loss and training difficulties that arise from discrete binary activation and complex spatio-temporal dynamics.
To this end, we develop a spiking residual block to convert the input into spike signals, then adaptively optimize the membrane potential by introducing attention weights to adjust spike responses in a data-driven manner, alleviating information loss caused by discrete binary activation.
By this way, our ESDNet can effectively detect and analyze the characteristics of rain streaks by learning their fluctuations. This also enables better guidance for the deraining process and facilitates high-quality image reconstruction.
Instead of relying on the ANN-SNN conversion strategy, we introduce a gradient proxy strategy to directly train the model for overcoming the challenge of training.
Experimental results show that our approach gains comparable performance against ANN-based methods while reducing energy consumption by 54\%.

![ESDNet](arch.png)

## Preparation

## Datasets

Dataset
Rain12
Rain200L
Rain200H
Rain1200
RW_Rain

Baidu Cloud
Download
Download
Download
Download
Download

## 🤖 Pre-trained Models

Dataset
Rain200L
Rain200H
Rain1200

Baidu Cloud
Download
Download
Download

### Install

We test the code on PyTorch 1.9.1 + CUDA 11.1 + cuDNN 8.0.5.

1. Create a new conda environment
```
conda create -n ESDNet python=3.8
conda activate ESDNet
```

2. Install dependencies
```
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

pip install matplotlib scikit-image opencv-python numpy einops math natsort tqdm lpips time tensorboardX
```

### Download

You can download the pre-trained models and datasets on BaiduPan.

The final file path should be the same as the following:

```
┬─ pretrained_models
│ ├─ Rain12.pth
│ ├─ Rain200H.pth
│ ├─ ... (model name)
│ └─ ... (exp name)
└─ data
├─ Rain200L
├─ Rain200H
│├─ train
││ ├─ input
││ │ └─ ... (image filename)
││ │ └─ target
││ │ └─ ... (corresponds to the former)
│└─ test
│ └─ ...
└─ ... (dataset name)

```
### 🛠️ Training, Testing and Evaluation

### Train
The training code will be released after the paper is accepted.
You should change the path to yours in the `Train.py` file. Then run the following script to test the trained model:

```sh
python train.py
```

### Test
You should change the path to yours in the `Test.py` file. Then run the following script to test the trained model:

```sh
python test.py
```

### Evaluation
You should change the path to yours in the `dataset_load.py` file. Then run the following script to test the trained model:

```sh
python evaluation.py
```
It is recommended that you can download the visual deraining results and retest the quantitative results on your own device and environment.

### 🚀 Visual Results

Dataset
Rain12
Rain200L
Rain200H
Rain1200
RW_Rain

Baidu Cloud
Download
Download
Download
Download
Download

## 🚨 Notes

1. Send e-mail to [email protected] if you have critical issues to be addressed.
2. Please note that there exists the slight gap in the final version due to errors caused by different testing devices and environments.
3. Because the synthetic dataset is not realistic enough, the trained models may not work well on real world datasets.

## 👍 Acknowledgment

This code is based on the [Restormer](https://github.com/swz30/Restormer), [spikingjelly](https://github.com/fangwei123456/spikingjelly) and [C2PNet](https://github.com/YuZheng9/C2PNet). Thanks for their awesome work.