Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/vita-group/insp
[NeurIPS 2022] "Signal Processing for Implicit Neural Representations" by Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang
https://github.com/vita-group/insp
audio-processing geometry-processing image-processing implicit-neural-representation signal-processing
Last synced: about 2 months ago
JSON representation
[NeurIPS 2022] "Signal Processing for Implicit Neural Representations" by Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang
- Host: GitHub
- URL: https://github.com/vita-group/insp
- Owner: VITA-Group
- License: mit
- Created: 2022-10-09T20:59:49.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2022-12-01T16:25:03.000Z (about 2 years ago)
- Last Synced: 2023-03-04T14:58:45.308Z (almost 2 years ago)
- Topics: audio-processing, geometry-processing, image-processing, implicit-neural-representation, signal-processing
- Language: Python
- Homepage: https://vita-group.github.io/INSP
- Size: 48.9 MB
- Stars: 36
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Signal Processing for Implicit Neural Representations
[](https://opensource.org/licenses/MIT)
The official implementation of NeurIPS 2022 paper ["Signal Processing for Implicit Neural Representations"]().
Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang (Atlas) Wang
[[Paper]](https://arxiv.org/abs/2210.08772) [[Website]](https://vita-group.github.io/INSP)
## Method Overview
## Environment
You can then set up a conda environment with all dependencies like so:
```
conda env create -f environment.yml
conda activate INSP
```
## High-Level structure
- Fit INR
- Export gradients for INR
- Train INSP-Net
- Inference INSP-Net
## Image Processing
For image processing, we experiment on div2k dataset.
```bash
wget http://data.vision.ee.ethz.ch/cvl/DIV2K/DIV2K_train_HR.zip &
unzip DIV2K_train_HR.zip
```
- Fit multiple INR
Use `--type ` to specify the type of images you want to train on.
```python gen_div2k.py | zsh```
- Export gradients for INR
`--load` is used for `glob` to filter out corresponding INRs.
```
python export_colorray.py --save_dir grad/train_color_noise/ --load 'div2k*.png_color_noise_'
```
Then, manually divide `grad/train_color_noise` and put a few of them into `grad/test_color_noise` (in our case we used first 100 images in DIV2K for training and the following 100 images for testing)
- Train INSP-Net
`--img_num` changes the number of images that are used for training.
The training should converge in a couple of minutes.
```
python experiment_scripts/train_img_grad_offline.py --model_type=sine --experiment_name denoise --noise_level 0 --target denoise --img_num 100 --overwrite --sigma 7 --sz 256 --batch_size 10240 --lr 1e-4
```
- Inference INSP-Net
```
python eval_insp.py --save_path output/denoise --target denoise --ckpt_path logs/denoise/checkpoints/model_current.pth
```
The INRs used in our experiments can be found [here](https://drive.google.com/drive/folders/1VaEgKiWIGpQhIw5uxPJGWL0OdTTM-cuo?usp=sharing).
## Audio Denoising
- Fit INR
```
python experiment_scripts/train_audio.py --model_type=sine --wav_path=data/gt_bach.wav --experiment_name audio_noisy_10
```
- Export gradients for INR
```
python export_audio.py
```
- Train INSP-Net
```
python experiment_scripts/train_audio_insp.py --experiment_name audio_denoise --batch_size 10240
```
- Inference INSP-Net
```
python eval_audio_insp.py
```
## SDF Smoothing
- Fit INR
```
```
- Export gradients for INR
```
python export_sdf_ray.py
```
- Train INSP-Net
```
python experiment_scripts/train_sdf_insp.py --experiment_name smooth_armadillo --sz 256 --ti 10 --batch_size 1
```
- Inference INSP-Net
```
python eval_sdf_insp.py
```
## Image Classification
Due to the large size of MNIST and CIFAR INRs, we can't provide all of the checkpoints. However, we share the scripts to generate the INRs.
## Citation
```
@inproceedings{Xu_2022_INSP,
title={Signal Processing for Implicit Neural Representations},
author={Xu, Dejia and Wang, Peihao and Jiang, Yifan and Fan, Zhiwen and Wang, Zhangyang},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2022}
}
```