https://github.com/trickygo/pano2room
Pytorch implementation of Pano2Room (SIGGRAPH Asia 2024)
https://github.com/trickygo/pano2room
Last synced: about 1 year ago
JSON representation
Pytorch implementation of Pano2Room (SIGGRAPH Asia 2024)
- Host: GitHub
- URL: https://github.com/trickygo/pano2room
- Owner: TrickyGo
- License: mit
- Created: 2024-08-06T05:40:03.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-23T04:35:59.000Z (almost 2 years ago)
- Last Synced: 2025-03-26T08:22:26.102Z (about 1 year ago)
- Language: Python
- Size: 16.1 MB
- Stars: 100
- Watchers: 6
- Forks: 7
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Welcome to Pano2Room!
[Pano2Room: Novel View Synthesis from a Single Indoor Panorama (SIGGRAPH Asia 2024)](https://arxiv.org/abs/2408.11413).
## Overview
#### In short, Pano2Room converts an input panorama into 3DGS.
## Demo
In this demo, specify input panorama as:
Then, Pano2Room generates the corresponding 3DGS and renders novel views:
And the corresponding rendered depth:
## Quick Run
### 0. Setup the environment
(1) Create a new conda environment and install [Pytorch3D](https://github.com/facebookresearch/pytorch3d) (for mesh rendering) and [diff-gaussian-rasterization-w-depth](https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth) (for 3DGS rendering with depth) accordingly. Other requirements are specified in \.
(2) Download pretrained weights in \ (for image inpainting and depth estimation). See \ for instructions.
### 1. Run Demo
```
sh scripts/run_Pano2Room.sh
```
This demo converts \ to 3DGS and renders novel views as in \.
### (Optional) 0.5. Fine-tune Inpainter (SDFT)
Before running step 1., you can also fine-tune SD Inpainter model for better inpainting performance for a specific panorama. To do this:
(1) Create self-supervised training pairs:
```
sh scripts/create_SDFT_pairs.sh
```
The pairs are then stored at \.
(2) Training:
```
sh scripts/train_SDFT.sh
```
The SDFT weights are then stored at \.
Then by running step 1., the SDFT weights will be automatically loaded.
Notice this step needs to be performed for each new panorama. If you don't want to train SDFT for a new panorama, delete previous \ if exists.
## Try on your own panorama
Simply replace \ with your own panorama and run the previous steps!
#### Camera Trajectory
We provide a camera trajectory at \ as in the above demo. Each file consists of [R|T] 4*4 matrix of a frame. Feel free to use more camera trajectories.
## Cite our paper
If you find our work helpful, please cite our paper. Thank you!
ACM Reference Format:
```
Guo Pu, Yiming Zhao, and Zhouhui Lian. 2024. Pano2Room: Novel View Synthesis from a Single Indoor Panorama. In SIGGRAPH Asia 2024 Conference Papers (SA Conference Papers '24), December 3--6, 2024, Tokyo, Japan. ACM, New York, NY, USA, 10 pages.
https://doi.org/10.1145/3680528.3687616
```