{"id":13443509,"url":"https://github.com/Xharlie/pointnerf","last_synced_at":"2025-03-20T16:31:39.629Z","repository":{"id":37283214,"uuid":"458356607","full_name":"Xharlie/pointnerf","owner":"Xharlie","description":"Point-NeRF: Point-based Neural Radiance Fields","archived":false,"fork":false,"pushed_at":"2024-03-08T20:57:52.000Z","size":2039,"stargazers_count":1081,"open_issues_count":70,"forks_count":124,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-08-01T03:43:51.066Z","etag":null,"topics":["3d","3d-reconstruction","cvpr2022","differentiable-rendering","graphics","implicit-functions","multiview-stereo","mvs","nerf","neural-radiance-fields","neural-renderer","neural-rendering","point-based-graphics","point-cloud","radiance-field","reconstruction","scannet","scene-reconstruction","tanks-and-temples","volume-rendering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Xharlie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-02-11T22:19:49.000Z","updated_at":"2024-08-01T03:43:51.067Z","dependencies_parsed_at":"2024-01-18T14:44:07.983Z","dependency_job_id":"b68f012e-a688-4a44-a17d-bc5e0b5660c2","html_url":"https://github.com/Xharlie/pointnerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xharlie%2Fpointnerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xharlie%2Fpointnerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xharlie%2Fpointnerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xharlie%2Fpointnerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Xharlie","download_url":"https://codeload.github.com/Xharlie/pointnerf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221780030,"owners_count":16879040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-reconstruction","cvpr2022","differentiable-rendering","graphics","implicit-functions","multiview-stereo","mvs","nerf","neural-radiance-fields","neural-renderer","neural-rendering","point-based-graphics","point-cloud","radiance-field","reconstruction","scannet","scene-reconstruction","tanks-and-temples","volume-rendering"],"created_at":"2024-07-31T03:02:02.489Z","updated_at":"2024-10-28T04:31:12.822Z","avatar_url":"https://github.com/Xharlie.png","language":"Python","funding_links":[],"categories":["Python","Papers"],"sub_categories":["NeRF"],"readme":"# Point-NeRF: Point-based Neural Radiance Fields (CVPR 2022 Oral 🤩)\n\u003cimg src=\"./images/Adobe-Logos.png\" width=120px /\u003e\u003cimg src=\"images/USC-Logos.png\" width=120px /\u003e\n\n[Project Sites](https://xharlie.github.io/projects/project_sites/pointnerf/index.html)\n | [Paper](https://arxiv.org/pdf/2201.08845.pdf) |\nPrimary contact: [Qiangeng Xu](https://xharlie.github.io/)\n\nPoint-NeRF uses neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.\n\n\u003c!-- \u003cimg src=\"./images/pipeline.png\" /\u003e --\u003e\n\n[![CVPR 2022 Oral Presentation](https://github.com/Xharlie/pointnerf/blob/master/images/youtube.png)](https://youtu.be/zmR9j-4AebA)\n\n## Reference\nPlease cite our paper if you are interested   \n \u003cstrong\u003ePoint-NeRF: Point-based Neural Radiance Fields\u003c/strong\u003e.  \u0026nbsp;\u0026nbsp;\u0026nbsp; \n```\n@inproceedings{xu2022point,\n  title={Point-nerf: Point-based neural radiance fields},\n  author={Xu, Qiangeng and Xu, Zexiang and Philip, Julien and Bi, Sai and Shu, Zhixin and Sunkavalli, Kalyan and Neumann, Ulrich},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={5438--5448},\n  year={2022}\n}\n```\n\n## Updates ##\n1. To replace pycuda, we have implemented the pytorch cuda functions when using world coordinates to group neural points. Simply set wcoord_query=-1\nin your configuration file if the original setting is wcoord_query=1 (see dev_scripts/w_n360/chair_cuda.sh).\n2. We have received constructive feedbacks that when Point-NeRF use MVSNet to reconstruct point cloud, the point fusion after depth estimation by MVSNet will use the alpha channel information in the NeRF-Synthetic Dataset. It is due to the fact that MVSNet cannot handle background very well. To improve the fairness, we include new training scripts and results of PointNeRF + MVSNet when using background color for filtering.  The results (see below) are similar to the ones that are previously reported.\n\n| | Chair |\tDrums\t| Lego\t| Mic\t| Materials\t| Ship\t| Hotdog\t| Ficus | Avg |\n| ---- | ---- | ---- | --- | ---- | ---- | ---- | ------- | ------- |------- |\n| PSNR | 35.60 | 26.04\t| 35.27\t| 35.91\t| 29.65\t| 30.61 |\t37.34 | 35.61\t| 33.25 |\n| SSIM  | 0.991\t| 0.954\t| 0.989\t| 0.994\t| 0.971\t| 0.938\t| 0.991\t| 0.992\t| 0.978 |\n| LPIPSVgg | 0.023\t| 0.078\t| 0.021\t| 0.014\t| 0.071\t| 0.129 | 0.036\t| 0.025 | 0.050 |\n| LPIPSAlex | 0.010\t| 0.055\t| 0.010\t| 0.007\t| 0.041\t| 0.076\t| 0.016\t| 0.011\t| 0.028 |\n\nThis issue only affacts situations when Point-NeRF uses MVSNet on NeRF-Synthetic Dataset. The Colmap results and results on other datasets are not impacted.    \nAn even more reasonable reconstruction approach should exclude using the knowledge of background color or other point filtering. Therefore, we suggest users to combine PointNeRF with more powerful MVS models, such as [TransMVS](https://github.com/megvii-research/TransMVSNet).\n\n\n## Overal Instruction\n1. Please first install the libraries as below and download/prepare the datasets as instructed.\n2. Point Initialization: Download pre-trained MVSNet as below and train the feature extraction from scratch or directly download the pre-trained models. (Obtain 'MVSNet' and 'init' folder in checkpoints folder)\n3.  Per-scene Optimization: Download pre-trained models or optimize from scratch as instructed.\n\nFor nerfsynthetic, colmap_nerfsynthetic, tanks\u0026temples, scannet and dtu,  \nWe provide all the checkpoint_files [google drive](https://drive.google.com/drive/folders/1xk1GhDhgPk1MrlX8ncfBz5hNMvSa9vS6?usp=sharing) | [baidu wangpan](https://pan.baidu.com/s/1doJHI03Tgl_qIquGZuW5bw?pwd=p8bs); all the images and scores of the test results [google drive](https://drive.google.com/drive/folders/1KAYs7XuBJNMTHVBuOCtpLNv9P8UMoayw?usp=sharing) | [baidu wangpan](https://pan.baidu.com/s/1BMewWRSIkNFlp7DKYmx9vQ?pwd=3yse); and video results [google drive](https://drive.google.com/drive/folders/1dutZEZO9vfeIbfWwplbIIam7YBeyZ0dY?usp=sharing) | [baidu wangpan](https://pan.baidu.com/s/1kC1qSL5dkT8cDdE3dHTc2A?pwd=j46j);\n\nWe also share the visual results of [npbg](https://github.com/alievk/npbg), [nsvf](https://github.com/facebookresearch/NSVF) and [ibrnet](https://github.com/googleinterns/IBRNet) on the Nerf Synthetic dataset generated by our machine [google drive](https://drive.google.com/drive/folders/1KHhljnqLvIJkRkaqQ8TaeBZirMsnDAhf?usp=sharing);  Please cite their papers accordingly if interested.\n\n## Installation\n\n### Requirements\nAll the codes are tested in the following environment:\n* Linux (tested on Ubuntu 16.04, 18.04, 20.04)\n* Python 3.6+\n* PyTorch 1.7 or higher (tested on PyTorch 1.7, 1.8.1, 1.9, 1.10)\n* CUDA 10.2 or higher \n\n### Install\nInstall the dependent libraries as follows:\n\n* Install the dependent python libraries:\n```\npip install torch==1.8.1+cu102 h5py\npip install imageio scikit-image\n```\n* Install pycuda (crucial) following:\nhttps://documen.tician.de/pycuda/install.html \n* Install torch_scatter following:\nhttps://github.com/rusty1s/pytorch_scatter\n\nWe develope our code with pytorch1.8.1, pycuda2021.1, and torch_scatter 2.0.8\n\n## Data Preparation\n\nThe layout should looks like this, we provide all data folder here: [google_drive](https://drive.google.com/drive/folders/1-Fn5g-NgHC0RcyWapHdfsqbSQksTgB4N?usp=sharing), except for scannet (We take it down on Mar 8th 2023, to respect Scannet's policy, please go to Scannet's official website for data.)\n```\npointnerf\n├── data_src\n│   ├── dtu\n    │   │   │──Cameras\n    │   │   │──Depths\n    │   │   │──Depths_raw\n    │   │   │──Rectified\n    ├── nerf\n    │   │   │──nerf_synthetic\n    │   │   │──nerf_synthetic_colmap\n    ├── TanksAndTemple\n    ├── scannet\n    │   │   │──scans \n    |   │   │   │──scene0101_04\n    |   │   │   │──scene0241_01\n```\nOr you can download using the official links as follows:\n\n## DTU:\nDownload the preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view)\nand [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repo](https://github.com/YoYo000/MVSNet)\nand unzip. \n\n\n## NeRF Synthetic\nDownload `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) under ``data_src/nerf/''\n\n\n## Tanks \u0026 Temples\nFollow Neural Sparse Voxel Fields and download [Tanks\u0026Temples](https://www.tanksandtemples.org/) | [download (.zip)](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip) | 0_\\* (training) 1_\\* (testing) as:\n``data_src/TanksAndTemple/''\n\n## ScanNet\nDownload and extract ScanNet by following the instructions provided at http://www.scan-net.org/. The detailed steps including:\n* Go to http://www.scan-net.org and fill \u0026 sent the request form.\n* You will get a email that has command instruction and a download-scannet.py file, this file is for python 2, you can use our download-scannet.py in the ``data'' directory for python 3.\n* clone the official repo:\n    ```\n    git clone https://github.com/ScanNet/ScanNet.git\n    ```\n* Download specific scenes (used by NSVF):\n  ```\n   python data/download-scannet.py -o ../data_src/scannet/ id scene0101_04 \n   python data/download-scannet.py -o ../data_src/scannet/ id scene0241_01\n  ```\n* Process the sens files:\n  ```\n    python ScanNet/SensReader/python/reader.py --filename data_src/nrData/scannet/scans/scene0101_04/scene0101_04.sens  --output_path data_src/nrData/scannet/scans/scene0101_04/exported/ --export_depth_images --export_color_images --export_poses --export_intrinsics\n    \n    python ScanNet/SensReader/python/reader.py --filename data_src/nrData/scannet/scans/scene0241_01/scene0241_01.sens  --output_path data_src/nrData/scannet/scans/scene0241_01/exported/ --export_depth_images --export_color_images --export_poses --export_intrinsics\n  ```\n\n## Point Initialization / Generalization:\n### \u0026nbsp; Download pre-trained MVSNet checkpoints:\nWe trained [MVSNet](https://github.com/xy-guo/MVSNet_pytorch) on DTU. You can Download ''MVSNet'' directory from \n[google drive](https://drive.google.com/drive/folders/1xk1GhDhgPk1MrlX8ncfBz5hNMvSa9vS6?usp=sharing)\nand place them under '''checkpoints/'''\n\n### \u0026nbsp;  Train 2D feature extraction and point representation\n#####  \u0026nbsp; Directly use our trained checkpoints files:\nDownload ''init'' directory from \n[google drive](https://drive.google.com/drive/folders/1xk1GhDhgPk1MrlX8ncfBz5hNMvSa9vS6?usp=sharing).\nand place them under '''checkpoints/'''\n\n##### \u0026nbsp; Or train from scratch:\nTrain for point features of 63 channels (as in paper) \n```\nbash dev_scripts/ete/dtu_dgt_d012_img0123_conf_color_dir_agg2.sh\n```\nTrain for point features of 32 channels (better for per-scene optimization)\n```\nbash dev_scripts/ete/dtu_dgt_d012_img0123_conf_agg2_32_dirclr20.sh\n```\nAfter the training, you should pick a checkpoint and rename it to best checkpoint, e.g.:\n```\ncp checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/250000_net_ray_marching.pth  checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/best_net_ray_marching.pth\n\ncp checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/250000_net_mvs.pth  checkpoints/dtu_dgt_d012_img0123_conf_color_dir_agg2/best_net_mvs.pth\n```\n### \u0026nbsp; Test feed forward inference on dtu scenes \nThese scenes that are selected by MVSNeRF, please also refer their code to understand the metrics calculation.\n```\nbash dev_scripts/dtu_test_inf/inftest_scan1.sh\nbash dev_scripts/dtu_test_inf/inftest_scan8.sh\nbash dev_scripts/dtu_test_inf/inftest_scan21.sh\nbash dev_scripts/dtu_test_inf/inftest_scan103.sh\nbash dev_scripts/dtu_test_inf/inftest_scan114.sh\n```\n\n## Per-scene Optimization:\n\u003cimg src=\"https://github.com/Xharlie/xharlie.github.io/raw/master/projects/project_sites/pointnerf/vid/ficus.gif\" width=\"45%\" /\u003e\u003cimg src=\"https://github.com/Xharlie/xharlie.github.io/raw/master/projects/project_sites/pointnerf/vid/scene101.gif\" width=\"50%\" /\u003e\n\u003cimg src=\"https://github.com/Xharlie/xharlie.github.io/raw/master/projects/project_sites/pointnerf/vid/truck.gif\" width=\"70%\" /\u003e\n\n(Please visit the project sites to see the original videos of above scenes, which have quality loss when being converted to gif files here.)\n### Download per-scene optimized Point-NeRFs\n You can skip training and download the folders of ''nerfsynth'', ''tanksntemples'' and ''scannet'' here [google drive](https://drive.google.com/drive/folders/1xk1GhDhgPk1MrlX8ncfBz5hNMvSa9vS6?usp=sharing), and place them in ''checkpoints/''.\n\n```\npointnerf\n├── checkpoints\n│   ├── init\n    ├── MVSNet\n    ├── nerfsynth\n    ├── col_nerfsynth\n    ├── scannet\n    ├── tanksntemples\n```\n\nIn each scene, we provide initialized point features and network weights ''0_net_ray_marching.pth'', points and weights at 20K steps ''20000_net_ray_marching.pth'' and 200K steps ''200000_net_ray_marching.pth''\n\n### Test the per-scene optimized Point-NeRFs\n#### NeRF Synthetics\n\u003cdetails\u003e\n  \u003csummary\u003etest scripts\u003c/summary\u003e\n  \n```\n    bash dev_scripts/w_n360/chair_test.sh\n    bash dev_scripts/w_n360/drums_test.sh\n    bash dev_scripts/w_n360/ficus_test.sh\n    bash dev_scripts/w_n360/hotdog_test.sh\n    bash dev_scripts/w_n360/lego_test.sh\n    bash dev_scripts/w_n360/materials_test.sh\n    bash dev_scripts/w_n360/mic_test.sh\n    bash dev_scripts/w_n360/ship_test.sh\n```\n\u003c/details\u003e\n\n\n#### ScanNet\n\u003cdetails\u003e\n  \u003csummary\u003etest scripts\u003c/summary\u003e\n  \n```\n    bash dev_scripts/w_scannet_etf/scane101_test.sh\n    bash dev_scripts/w_scannet_etf/scane241_test.sh\n```\n\u003c/details\u003e\n\n#### Tanks \u0026 Temples\n\u003cdetails\u003e\n  \u003csummary\u003etest scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_tt_ft/barn_test.sh\n    bash dev_scripts/w_tt_ft/caterpillar_test.sh\n    bash dev_scripts/w_tt_ft/family_test.sh\n    bash dev_scripts/w_tt_ft/ignatius_test.sh\n    bash dev_scripts/w_tt_ft/truck_test.sh\n```\n\u003c/details\u003e\n\n### Per-scene optimize from scatch \nMake sure the ''checkpoints'' folder has ''init'' and ''MVSNet''.\nThe training scripts will start to do initialization if there is no ''.pth'' files in a scene folder. It will start from the last ''.pth'' files until reach the iteration of ''maximum_step''.\n\n#### NeRF Synthetics using MVSNet (w/ alpha channel filtering during point cloud reconstruction and pycuda)\n\u003cdetails\u003e\n  \u003csummary\u003etrain scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_n360/chair.sh\n    bash dev_scripts/w_n360/drums.sh\n    bash dev_scripts/w_n360/ficus.sh\n    bash dev_scripts/w_n360/hotdog.sh\n    bash dev_scripts/w_n360/lego.sh\n    bash dev_scripts/w_n360/materials.sh\n    bash dev_scripts/w_n360/mic.sh\n    bash dev_scripts/w_n360/ship.sh\n```\n\u003c/details\u003e\n\n\n#### NeRF Synthetics using MVSNet (w/ background color filtering during point cloud reconstruction and pytorch cuda)\n\u003cdetails\u003e\n  \u003csummary\u003etrain scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_n360/chair_cuda.sh\n    bash dev_scripts/w_n360/drums_cuda.sh\n    bash dev_scripts/w_n360/ficus_cuda.sh\n    bash dev_scripts/w_n360/hotdog_cuda.sh\n    bash dev_scripts/w_n360/lego_cuda.sh\n    bash dev_scripts/w_n360/materials_cuda.sh\n    bash dev_scripts/w_n360/mic_cuda.sh\n    bash dev_scripts/w_n360/ship_cuda.sh\n```\n\u003c/details\u003e\n\n#### NeRF Synthetics using COLMAP points\nPlease download the COLMAP data (see above). If there is {maximum_step}.pth checkpoint files in the path, the scripts below will also run test.\n\u003cdetails\u003e\n  \u003csummary\u003etrain scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_colmap_n360/col_chair.sh\n    bash dev_scripts/w_colmap_n360/col_drums.sh\n    bash dev_scripts/w_colmap_n360/col_ficus.sh\n    bash dev_scripts/w_colmap_n360/col_hotdog.sh\n    bash dev_scripts/w_colmap_n360/col_lego.sh\n    bash dev_scripts/w_colmap_n360/col_materials.sh\n    bash dev_scripts/w_colmap_n360/col_mic.sh\n    bash dev_scripts/w_colmap_n360/col_ship.sh\n```\n\u003c/details\u003e\n\n#### ScanNet\n\u003cdetails\u003e\n  \u003csummary\u003etrain scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_scannet_etf/scene101.sh\n    bash dev_scripts/w_scannet_etf/scene241.sh\n```\n\u003c/details\u003e\n\n#### Tanks \u0026 Temples\n\u003cdetails\u003e\n  \u003csummary\u003etrain scripts\u003c/summary\u003e\n\n```\n    bash dev_scripts/w_tt_ft/barn.sh\n    bash dev_scripts/w_tt_ft/caterpillar.sh\n    bash dev_scripts/w_tt_ft/family.sh\n    bash dev_scripts/w_tt_ft/ignatius.sh\n    bash dev_scripts/w_tt_ft/truck.sh\n```\n\u003c/details\u003e\n\n\n\n## Acknowledgement\nOur repo is developed based on [MVSNet](https://github.com/YoYo000/MVSNet),  [NeRF](https://github.com/bmild/nerf), [MVSNeRF](https://github.com/apchenstu/mvsnerf), and [NSVF](https://github.com/facebookresearch/NSVF).\n\nPlease also consider citing the corresponding papers. \n\nThe project is conducted collaboratively between Adobe Research and University of Southern California. \n\n## LICENSE\nThe repo is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 2.0, and is restricted to academic use only.\nSee [LICENSE](https://github.com/Xharlie/pointnerf/blob/master/LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXharlie%2Fpointnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FXharlie%2Fpointnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FXharlie%2Fpointnerf/lists"}