{"id":13443320,"url":"https://github.com/POSTECH-CVLab/PeRFception","last_synced_at":"2025-03-20T16:30:59.375Z","repository":{"id":52789512,"uuid":"436146387","full_name":"POSTECH-CVLab/PeRFception","owner":"POSTECH-CVLab","description":"[NeurIPS2022] Official implementation of PeRFception: Perception using Radiance Fields. ","archived":false,"fork":false,"pushed_at":"2023-01-21T15:34:02.000Z","size":11076,"stargazers_count":332,"open_issues_count":9,"forks_count":17,"subscribers_count":13,"default_branch":"release","last_synced_at":"2025-03-16T04:08:44.139Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/POSTECH-CVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-08T06:52:30.000Z","updated_at":"2025-02-18T12:03:41.000Z","dependencies_parsed_at":"2023-01-22T21:45:40.767Z","dependency_job_id":null,"html_url":"https://github.com/POSTECH-CVLab/PeRFception","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FPeRFception","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FPeRFception/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FPeRFception/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FPeRFception/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/POSTECH-CVLab","download_url":"https://codeload.github.com/POSTECH-CVLab/PeRFception/tar.gz/refs/heads/release","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244649793,"owners_count":20487493,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:59.142Z","updated_at":"2025-03-20T16:30:58.758Z","avatar_url":"https://github.com/POSTECH-CVLab.png","language":"Python","funding_links":[],"categories":["Python","其他_机器视觉"],"sub_categories":["网络服务_其他"],"readme":"# PeRFception - Perception using Radiance Fields\n\n\u003cdiv style=\"text-align:center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/33657821/183290143-7fbf49ce-2f6f-4bfc-8496-9d24e5d8ab61.gif\" alt=\"teaser\"/\u003e\n\u003c/div\u003e\n\n### Quick Access\n\n[Project Page](https://postech-cvlab.github.io/PeRFception) | [Paper](https://arxiv.org/abs/2208.11537) | [Supplementary Materials](https://openreview.net/attachment?id=MzaPEKHv-0J\u0026name=supplementary_material)\n### Author Info\n\n- [Yoonwoo Jeong](https://yoonwooinfo.notion.site) [[Google Scholar](https://scholar.google.com/citations?user=HQ1PMggAAAAJ\u0026hl=en)]\n- [Seungjoo Shin](https://seungjooshin.github.io/) [[Google Scholar](https://scholar.google.com/citations?user=io7PSDIAAAAJ\u0026hl=en)]\n- [Junha Lee](https://junha-l.github.io/) [[Google Scholar](https://scholar.google.com/citations?user=RB7qMm4AAAAJ\u0026hl=en)]\n- [Chris Choy](https://chrischoy.org) [[Google Scholar](https://scholar.google.com/citations?user=2u8G5ksAAAAJ\u0026hl=en\u0026oi=ao)]\n- [Animashree Anandkumar](http://tensorlab.cms.caltech.edu/users/anima/) [[Google Scholar](https://scholar.google.com/citations?user=bEcLezcAAAAJ\u0026hl=en\u0026oi=ao)]\n- [Minsu Cho](http://cvlab.postech.ac.kr/~mcho/) [[Google Scholar](https://scholar.google.com/citations?user=5TyoF5QAAAAJ\u0026hl=en\u0026oi=ao)]\n- [Jaesik Park](http://jaesik.info/) [[Google Scholar](https://scholar.google.com/citations?user=_3q6KBIAAAAJ\u0026hl=en\u0026oi=ao)]\n\n### News\n- 22.10.27: Added HuggingFace repo for ScanNet.\n- 22.10.25: Moved PeRFception datasets from OneDrive to HuggingFace.\n- 22.08.04: The first version of PeRFception has been released. \n\n\u003cdiv style=\"text-align:center\"\u003e\n\u003cimg src=\"assets/teaser.png\" alt=\"teaser\"/\u003e\n\u003c/div\u003e\n\n## Abstract\n\nThe recent progress in implicit 3D representation, i.e., Neural Radiance Fields (NeRFs), has made accurate and photorealistic 3D reconstruction possible in a differentiable manner. This new representation can effectively convey the information of hundreds of high-resolution images in one compact format and allows photorealistic synthesis of novel views. In this work, using the variant of NeRF called Plenoxels, we create the first large-scale implicit representation datasets  for perception tasks, called PeRFception, which consists of two parts that incorporate both object-centric and scene-centric scans for classification and segmentation. It shows a significant memory compression rate (96.4%) from the original dataset, while containing both 2D and 3D information in a unified form. We construct the  classification and segmentation models that directly take as input this implicit format and also propose a novel augmentation technique to avoid overfitting on backgrounds of images. The code and data will be publicly available. \n\n## Downloading PeRFception-Datastes [[CO3D-link1](https://huggingface.co/datasets/YWjimmy/PeRFception-v1-1)] [[CO3D-link2](https://huggingface.co/datasets/YWjimmy/PeRFception-v1-2)] [[CO3D-link3](https://huggingface.co/datasets/YWjimmy/PeRFception-v1-3)] [[ScanNet]( https://huggingface.co/datasets/YWjimmy/PeRFception-ScanNet)]\n\n```\n# Link1 - PeRFception-CO3D-v1\ngit clone https://huggingface.co/datasets/YWjimmy/PeRFception-v1-1\n# Link2 - PeRFception-CO3D-v1\ngit clone https://huggingface.co/datasets/YWjimmy/PeRFception-v1-2\n# Link3 - PeRFception-CO3D-v1\ngit clone https://huggingface.co/datasets/YWjimmy/PeRFception-v1-3\n# Link1 - PeRFception-ScanNet\ngit clone https://huggingface.co/datasets/YWjimmy/PeRFception-ScanNet\n```\n### Downloading specific chunks\n```\nmkdir \u003crepo\u003e\ncd \u003crepo\u003e\ngit init\ngit remote add -f origin [link] \ngit config core.sparseCheckout true\necho \"some/dir/\" \u003e\u003e .git/info/sparse-checkout\necho \"another/sub/tree\" \u003e\u003e .git/info/sparse-checkout\ngit pull origin main\n\n# ex) If you want to download data only from 288_30460_58530\necho \"30/plenoxel_co3d_288_30460_58530\" \u003e\u003e .git/info/sparse-checkout\n```\n### PeRFception-CO3D\n\n|Dataset| # Scenes | # Frames | 3D Shape | Features | 3D-BKGD | Memory | Memoery(Rel)\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CO3D| 18.6K | 1.5M | pcd | C | X | 1.44TB | $$\\pm0.00\\%$$\n|PeRFception-CO3D| 18.6K | $$\\infty$$ | voxel | SH + D | O | 1.33TB | $$-6.94\\%$$\n\n### PeRFception-ScanNet \n\n|Dataset| # Scenes | # Frames | 3D Shape | Features | 3D-BKGD | Memory | Memoery(Rel)\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ScanNet| 1.5K | 2.5M | pcd | C | X | 966GB | $$\\pm0.00\\%$$\n|PeRFception-ScanNet| 1.5K | $$\\infty$$ | voxel | SH + D | O | 35GB | $$-96.4\\%$$\n\n\n## Get Ready (Installation)\n\nOur code is verified on Ubuntu 20.04 with a CUDA version 11.1.  \n\n```\nconda create -n perfception -c anaconda python=3.8 -y\nconda activate perfception\nconda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge -y\npip3 install imageio tqdm requests configargparse scikit-image imageio-ffmpeg piqa wandb pytorch_lightning==1.5.5 opencv-python gin-config gdown plyfile\npip3 install .\n```\n\n\n## Demo \nWe provide a short demo for rendering a scene on CO3D or ScanNet. After installing the requirements, you could run the demo with the codes below:\n```\n# CO3D demo\npython3 -m run --ginc configs/co3d.gin\n# ScanNet demo\npython3 -m run --ginc configs/scannet.gin\n```\n\n## Rendering CO3D and ScanNet \nWe deliver the full code to reproduce the performance reported in the main paper. To run the code, you should first put the dataset on a proper location. \n```\ndata\n  |\n  |--- co3d\n         -- apple \n         -- banana\n         ... \n  |\n  |--- scannet\n         -- scene000_00\n         -- scene000_01\n         ...\n```\nScanNet-v2 can be downloaded in [here](http://www.scan-net.org/) and CO3D-v1 can be downloaded in [here](https://github.com/facebookresearch/co3d). Thanks to great functions in `wandb`, we could manage tremendous scripts. You can download the `sweep` file [here](https://1drv.ms/u/s!As9A9EbDsoWcj6toSOfdeWMaHhqF3Q?e=1INfNg). \n\n\n## Downstream Tasks\n\n### Codes for downstream tasks: https://github.com/POSTECH-CVLab/NeRF-Downstream\n\n### 2D object classification (PeRFception-CO3D)\n\nWe benchmark several 2D classification models on rendered PeRFception-CO3D. For faster reproducing, we also provide the rendered images from PeRFception-CO3D on the link [link](https://1drv.ms/u/s!AgY2evoYo6FggthVfVngtHinq3czqQ?e=crnTlu). Before running the code, be sure that you had put the  downloaded dataset on `data/perfcepton_2d`. You can easily reproduce the scores using the scripts of `scripts/downstream/2d_cls/[model].sh`. Details for the training pipeline and models are elaborated in the main paper. \n\nThe pretrained models can be reached with the links below: \n\u003cdiv style=\"text-align:center\"\u003e\n\u003cimg src=\"assets/2D_score.png\" alt=\"2D score\"/\u003e\n\u003c/div\u003e\n\n\n### 3D object classification (PeRFception-CO3D)\n\nWe also benchmark several 3D classification models on PeRFception-CO3D. We provide the full code on the link [](). You can downloa\n\n\u003cdiv style=\"text-align:center\"\u003e\n\u003cimg src=\"assets/3D_score.png\" alt=\"3D score\" height=300/\u003e\n\u003c/div\u003e\n\n\n### 3D semantic segmentation (PeRFception-ScanNet)\nIn PeRFception-ScanNet, we have evaluated several 3D semantic segmentation models with depth-supervised labels. \n\n## Plans for v2\n\nAccording to the official CO3D repository[[link](https://github.com/facebookresearch/co3d)], authors provided an improved version, v2, of CO3D, which would result in better rendering quality and more accurate geometries in our model. We are planning to extend this work to PeRFception-CO3D-v2 from the CO3D-v2. \n\n## Citation\n```bib\n@article{jeong2022perfception,\n  title   = {PeRFception: Perception using Radiance Fields},\n  author  = {Jeong, Yoonwoo and Shin, Seungjoo and Lee, Junha and Choy, Chris and Anandkumar, Anima and Cho, Minsu and Park, Jaesik}\n  year    = {2022}\n}\n```\n\n## Acknowledgement\nWe appreciate for the reviewers for their constructive comments and suggestions. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPOSTECH-CVLab%2FPeRFception","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPOSTECH-CVLab%2FPeRFception","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPOSTECH-CVLab%2FPeRFception/lists"}