{"id":13474702,"url":"https://zvict.github.io/papr/","last_synced_at":"2025-03-26T22:31:08.039Z","repository":{"id":211543880,"uuid":"729281018","full_name":"zvict/papr","owner":"zvict","description":"Original reference implementation of \"PAPR: Proximity Attention Point Rendering\"","archived":false,"fork":false,"pushed_at":"2024-07-25T03:26:43.000Z","size":62193,"stargazers_count":35,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-10-30T08:48:47.525Z","etag":null,"topics":["3d-reconstruction","computer-graphics","computer-vision","novel-view-synthesis"],"latest_commit_sha":null,"homepage":"https://zvict.github.io/papr/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zvict.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-08T19:51:42.000Z","updated_at":"2024-10-28T22:40:45.000Z","dependencies_parsed_at":"2024-01-13T18:24:57.218Z","dependency_job_id":"6b4a29f5-2029-48f7-b97e-70b2875cb6c0","html_url":"https://github.com/zvict/papr","commit_stats":null,"previous_names":["zvict/papr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zvict%2Fpapr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zvict%2Fpapr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zvict%2Fpapr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zvict%2Fpapr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zvict","download_url":"https://codeload.github.com/zvict/papr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245747584,"owners_count":20665813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-graphics","computer-vision","novel-view-synthesis"],"created_at":"2024-07-31T16:01:14.185Z","updated_at":"2025-03-26T22:31:07.496Z","avatar_url":"https://github.com/zvict.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# PAPR: Proximity Attention Point Rendering (NeurIPS 2023 Spotlight 🤩)\n[Yanshu Zhang*](https://zvict.github.io/), [Shichong Peng*](https://sites.google.com/view/niopeng/home), [Alireza Moazeni](https://amoazeni75.github.io/), [Ke Li](https://www.sfu.ca/~keli/) (* denotes equal contribution)\u003cbr\u003e\n\n\u003cimg src=\"./images/SFU_AI.png\" height=100px /\u003e\u003cimg src=\"images/APEX_lab.png\" height=120px /\u003e\n\n[Project Sites](https://zvict.github.io/papr)\n | [Paper](https://arxiv.org/abs/2307.11086) |\nPrimary contact: [Yanshu Zhang](https://zvict.github.io/)\n\nProximity Attention Point Rendering (PAPR) is a new method for joint novel view synthesis and 3D reconstruction. It simultaneously learns from scratch an accurate point cloud representation of the scene surface, and an attention-based neural network that renders the point cloud from novel views.\n\n\u003c!-- \u003cimg src=\"./images/pipeline.png\" /\u003e --\u003e\n\n[![NeurIPS 2023 Presentation](https://github.com/zvict/papr/blob/main/images/papr_video_cover.png)](https://youtu.be/1atBGH_pDHY)\n\n## BibTeX\n \u003cstrong\u003ePAPR: Proximity Attention Point Rendering\u003c/strong\u003e.  \u0026nbsp;\u0026nbsp;\u0026nbsp; \n```\n@inproceedings{zhang2023papr,\n    title={PAPR: Proximity Attention Point Rendering},\n    author={Yanshu Zhang and Shichong Peng and Seyed Alireza Moazenipourasil and Ke Li},\n    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},\n    year={2023}\n}\n```\n\n## Installation\n```\ngit clone git@github.com:zvict/papr.git   # or 'git clone https://github.com/zvict/papr'\ncd papr\nconda env create -f papr.yml\nconda activate papr\n```\nOr use virtual environment with `python=3.9`\n```\npython -m venv path/to/\u003cenv_name\u003e\nsource path/to/\u003cenv_name\u003e/bin/activate\npip install -r requirements.txt\n```\n\n## Data Preparation\n\nExpected dataset structure in the source path location:\n```\npapr\n├── data\n│   ├── nerf_synthetic\n│   │   ├── chair\n│   │   │   ├── train\n│   │   │   ├── val\n│   │   │   ├── test\n│   │   │   ├── transforms_train.json\n│   │   │   ├── transforms_val.json\n│   │   │   ├── transforms_test.json\n│   │   ├── ...\n│   ├── tanks_temples\n│   │   ├── Barn\n│   │   │   ├── pose\n│   │   │   ├── rgb\n│   │   │   ├── intrinsics.txt\n│   │   ├── ...\n```\n### NeRF Synthetic\nDownload NeRF Synthetic Dataset from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) and put it under `data/nerf_synthetic/`\n\n\n### Tanks \u0026 Temples\nDownload [Tanks\u0026Temples](https://www.tanksandtemples.org/) from [here](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip) and put it under:\n`data/tanks_temples/`\n\n### Use your own data\nYou can refer to this [issue](https://github.com/zvict/papr/issues/3#issuecomment-1907260683) for the instructions on how to prepare the dataset.\n\nYou need to create a new configuration file for your own dataset, and put it under `configs`. The parameter `dataset.type` in the configuration file specifies the type of the dataset. If your dataset is in the same format as the NeRF Synthetic dataset, you can directly set `dataset.type` to `\"synthetic\"`. Otherwise, you need to implement your own python script to load the dataset under the `dataset` folder, and add it in the function `load_meta_data` in `dataset/utils.py`.\n\nMost default parameters in `configs/default.yml` are general and can be used for your own dataset. You can specify the parameters that are specific to your dataset in the configuration file you created, similar to the configuration files for the NeRF Synthetic dataset and the Tanks and Temples dataset.\n\n## Overview\n\nThe codebase has two main components: data loading part in `dataset/` and models in `models/`. Class `PAPR` in `models/model.py` defines our main model. All the configurations are in `configs/`, and `configs/demo.yml` is a demo configuration with comments of important arguments.\n\nWe provide a notebook `demo.ipynb` to demonstrate how to train and test the model with the demo configuration file, as well as how to use exposure control to improve the rendering quality of real-world scenes captured with auto-exposure turned on.\n\n## Training\n```\npython train.py --opt configs/nerfsyn/chair.yml\n```\n\n## Finetuning with [cIMLE](https://arxiv.org/abs/2004.03590) (Optional)\n\nFor real-world scenes where exposure can change between views, we can introduce an additional latent code input into our model and finetune the model using a technique called [conditional Implicit Maximum Likelihood Estimation (cIMLE)](https://arxiv.org/abs/2004.03590) to control the exposure level of the rendered image, as described in Section 4.4 and Appendix A.8 in the paper. A pre-trained model is required to finetune with exposure control, by running `train.py` with default configurations. We provide a demo configuration file for the Caterpillar scene from the Tanks and Temples dataset at `configs/t2/Caterpillar_exposure_control.yml`.\n\nTo finetune a pre-trained model with exposure control, run:\n```\npython exposure_control_finetune.py --opt configs/t2/Caterpillar_exposure_control.yml\n```\n\n## Evaluation\nTo evaluate your trained model without the finetuning for exposure control, run:\n```\npython test.py --opt configs/nerfsyn/chair.yml\n```\nWhich gives you rendered images and metrics on the test set.\n\nWith a finetuned model, you can render all the test views with a single random exposure level, by runing:\n```\npython test.py --opt configs/t2/Caterpillar_exposure_control.yml --exp\n```\nTo generate images with different random exposure levels for a single view, run:\n```\npython test.py --opt configs/t2/Caterpillar_exposure_control.yml --exp --random --view 0\n```\nNote that during testing, the scale of the latent codes should be increased to generate images with more diverse exposures, for example,\n```\npython test.py --opt configs/t2/Caterpillar_exposure_control.yml --exp --random --view 0 --scale 8\n```\nOnce you generate images with different exposure levels, you can interpolate two picked exposure levels by specifiying their index, for example,\n```\npython test.py --opt configs/t2/Caterpillar_exposure_control.yml --exp --intrp --view 0 --start_index 0 --end_index 1\n```\n\n## Pretrained Models\n\nWe provide pretrained models on NeRF Synthetic and Tanks\u0026Temples datasets here (without finetuning): [Google Drive](https://drive.google.com/drive/folders/1HSNlMu6Uup9o5hqi7T0hgDf63yR9W90s?usp=sharing). We also provide a pre-trained model with exposure control on the Caterpillar scene in the Google Drive. To load the pretrained models, please put them under `checkpoints/`, and change the `test.load_path` in the config file.\n\n## Acknowledgement\nThis research was enabled in part by support provided by NSERC, the BC DRI Group and the Digital Research Alliance of Canada.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/zvict.github.io%2Fpapr%2F","html_url":"https://awesome.ecosyste.ms/projects/zvict.github.io%2Fpapr%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/zvict.github.io%2Fpapr%2F/lists"}