https://akshatdave.github.io/pandora/

Official Pytorch implementation of PANDORA: Polarization-Aided Neural Decomposition of Radiance
https://akshatdave.github.io/pandora/

3d-reconstruction computer-vision inverse-rendering nerf neural-rendering polarization polarization-imaging

Last synced: 3 months ago
JSON representation

Official Pytorch implementation of PANDORA: Polarization-Aided Neural Decomposition of Radiance

• Host: GitHub
• URL: https://akshatdave.github.io/pandora/
• Owner: akshatdave
• Created: 2022-07-18T20:09:19.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2022-07-19T20:36:27.000Z (over 2 years ago)
• Last Synced: 2024-10-28T18:29:43.758Z (4 months ago)
• Topics: 3d-reconstruction, computer-vision, inverse-rendering, nerf, neural-rendering, polarization, polarization-imaging
• Language: Python
• Homepage: https://akshatdave.github.io/pandora
• Size: 7.49 MB
• Stars: 62
• Watchers: 3
• Forks: 5
• Open Issues: 4
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• Awesome-Polarization - PANDORA: Polarization-Aided Neural Decomposition Of Radiance

README

        
# PANDORA: Polarization-Aided Neural Decomposition Of Radiance

### [Project Page](https://akshatdave.github.io/pandora) | [Paper](https://arxiv.org/abs/2203.13458) | [Data](https://drive.google.com/file/d/1FvOi_2wfSUnASHulQdBhHQcQCxOuJ8zz/view?usp=sharing)




Official PyTorch implementation of Pandora: an inverse rendering technique that exploits neural implicit representations and polarization cues. 




[PANDORA: Polarization-Aided Neural Decomposition Of Radiance](https://akshatdave.github.io/pandora)

 [Akshat Dave](https://akshadave.github.io),

 [Yongyi Zhao](https://yongyizhao.com/),

 [Ashok Veeraraghavan](https://computationalimaging.rice.edu/team/ashok-veeraraghavan/) 

 [Computational Imaging Lab, Rice University](https://computationalimaging.rice.edu)

accepted for ECCV 2022

![Teaser Animation](media/teaser_animation.gif)

## Setting up

### Loading conda environment

Create a new Anaconda environment using the supplied `environment.yml` 

```

conda env create -f environment.yml

```

### Downloading datasets

Unzip [this](https://drive.google.com/file/d/1FvOi_2wfSUnASHulQdBhHQcQCxOuJ8zz/view?usp=sharing) zip file (4.5 GB) into the `data` folder of the repo directory. The zip file contains real and rendered multi-view polarimetric datasets shown in the paper. 

Refer to `dataio/Ours.py` and `data/Mitsuba2.py` for pre-processing of real and rendered data respectively.

## Training

Run the following command to train geometry and radiance neural representations from multi-view polarimetric images.

```

python -m train --config configs/real_ceramic_owl.yaml

```

Config files input through `--config` describe the parameters required for training. As an example the parameters for real ceramic owl dataset are described in `real_ceramic_owl.yaml`

Tensorboard logs, checkpoints, arguments and images are saved in the corresponding experiment folder in `logs/`.

## Rendering Trained Representations

Using the saved arguments from `config.yaml` and the saved checkpoint such as `latest.pt` in `logs/`, novel views can be rendered using the following command.

```

python -m tools.render_view 

    --config  logs/our_ceramic_owl_v2/config.yaml

    --load_pt logs/our_ceramic_owl_v2/ckpts/latest.pt

```

Refer to [this](https://github.com/ventusff/neurecon/blob/main/docs/usage.md#evaluation-free-viewport-rendering) documentation in  `neurecon` repo for possible camera trajectories. By default first three views used for training are rendered. 

Outputs are saved in the corresponding experiment folder in `out/`. By default,the outputs include surface normal, diffuse radiance, specular radiance and combined radiance for each view along with the estimated roughness.

## Acknowledgements

This repository adapts code or draws inspiration from

- https://github.com/ventusff/neurecon

- https://github.com/yenchenlin/nerf-pytorch

- https://github.com/Fyusion/LLFF

- https://github.com/elerac/polanalyser

- https://github.com/sxyu/svox2

## Citation

```

@article{dave2022pandora,

  title={PANDORA: Polarization-Aided Neural Decomposition Of Radiance},

  author={Dave, Akshat and Zhao, Yongyi and Veeraraghavan, Ashok},

  journal={arXiv preprint arXiv:2203.13458},

  year={2022}

}

```