{"id":13444138,"url":"https://github.com/mit-han-lab/pvcnn","last_synced_at":"2025-03-20T18:30:37.208Z","repository":{"id":36547895,"uuid":"226997592","full_name":"mit-han-lab/pvcnn","owner":"mit-han-lab","description":"[NeurIPS 2019, Spotlight] Point-Voxel CNN for Efficient 3D Deep Learning","archived":true,"fork":false,"pushed_at":"2022-04-12T12:07:21.000Z","size":136,"stargazers_count":642,"open_issues_count":0,"forks_count":131,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-10-28T07:42:58.388Z","etag":null,"topics":["frustum-pointnet","frustum-pointnets","kitti","point-cloud","point-voxel-cnn","pointnet","pointnet2","pvcnn","pytorch","s3dis","shapenet"],"latest_commit_sha":null,"homepage":"https://pvcnn.mit.edu/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-han-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-10T00:52:20.000Z","updated_at":"2024-10-24T03:09:21.000Z","dependencies_parsed_at":"2022-08-09T07:34:56.142Z","dependency_job_id":null,"html_url":"https://github.com/mit-han-lab/pvcnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fpvcnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fpvcnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fpvcnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fpvcnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-han-lab","download_url":"https://codeload.github.com/mit-han-lab/pvcnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244669865,"owners_count":20490872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["frustum-pointnet","frustum-pointnets","kitti","point-cloud","point-voxel-cnn","pointnet","pointnet2","pvcnn","pytorch","s3dis","shapenet"],"created_at":"2024-07-31T03:02:20.032Z","updated_at":"2025-03-20T18:30:37.192Z","avatar_url":"https://github.com/mit-han-lab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# PVCNN: Point-Voxel CNN for Efficient 3D Deep Learning\n\n[NVIDIA Jetson Community Project Spotlight](https://news.developer.nvidia.com/point-voxel-cnn-3d/?ncid=so-twit-99540#cid=em02_so-twit_en-us)\n\n```\n@inproceedings{liu2019pvcnn,\n  title={Point-Voxel CNN for Efficient 3D Deep Learning},\n  author={Liu, Zhijian and Tang, Haotian and Lin, Yujun and Han, Song},\n  booktitle={Advances in Neural Information Processing Systems},\n  year={2019}\n}\n```\n\n## Prerequisites\n\nThe code is built with following libraries (see [requirements.txt](requirements.txt)):\n- Python \u003e= 3.7\n- [PyTorch](https://github.com/pytorch/pytorch) \u003e= 1.3\n- [numba](https://github.com/numba/numba)\n- [numpy](https://github.com/numpy/numpy)\n- [scipy](https://github.com/scipy/scipy)\n- [six](https://github.com/benjaminp/six)\n- [tensorboardX](https://github.com/lanpa/tensorboardX) \u003e= 1.2\n- [tqdm](https://github.com/tqdm/tqdm)\n- [plyfile](https://github.com/dranjan/python-plyfile)\n- [h5py](https://github.com/h5py/h5py)\n\n## Data Preparation\n\n### S3DIS\n\nWe follow the data pre-processing in [PointCNN](https://github.com/yangyanli/PointCNN).\nThe code for preprocessing the S3DIS dataset is located in [`data/s3dis/`](data/s3dis/prepare_data.py).\nOne should first download the dataset from [here](http://buildingparser.stanford.edu/dataset.html), then run \n```bash\npython data/s3dis/prepare_data.py -d [path to unzipped dataset dir]\n```\n\n### ShapeNet\n\nWe follow the data pre-processing in [PointNet2](https://github.com/charlesq34/pointnet2). Please run the following\ncommand to down the dataset\n```bash\n./data/shapenet/download.sh\n```\n\n### KITTI\n\nFor Frustum-PointNet backbone, we follow the data pre-processing in [Frustum-Pointnets](https://github.com/charlesq34/frustum-pointnets).\nOne should first download the ground truth labels from [here](http://www.cvlibs.net/download.php?file=data_object_label_2.zip), then run\n```bash\nunzip data_object_label_2.zip\nmv training/label_2 data/kitti/ground_truth\n./data/kitti/frustum/download.sh\n```\n\n## Code\n\nThe core code to implement PVConv is [modules/pvconv.py](modules/pvconv.py). Its key idea costs only a few lines of code:\n\n```python\n    voxel_features, voxel_coords = voxelize(features, coords)\n    voxel_features = voxel_layers(voxel_features)\n    voxel_features = trilinear_devoxelize(voxel_features, voxel_coords, resolution)\n    fused_features = voxel_features + point_layers(features)\n```\n\n## Pretrained Models\n\nHere we provide some of the pretrained models. The accuracy might vary a little bit compared to the paper,\nsince we re-train some of the models for reproducibility.\n\n### S3DIS\n\nWe compare PVCNN against the PointNet, 3D-UNet and PointCNN performance as reported in the following table.\nThe accuracy is tested following [PointCNN](https://github.com/yangyanli/PointCNN). The list is keeping updated.\n\n|                                                  Models                                                     | Overall Acc |     mIoU     | \n| :---------------------------------------------------------------------------------------------------------: | :---------: | :----------: |\n|  PointNet                                                                                                   |    82.54    |     42.97    |\n|  [PointNet (Reproduce)](https://hanlab.mit.edu/files/pvcnn/s3dis.pointnet.area5.pth.tar)    |    80.46    |     44.03    |\n|  [PVCNN (0.125 x C)](https://hanlab.mit.edu/files/pvcnn/s3dis.pvcnn.area5.c0p125.pth.tar)   |    82.79    |   **48.75**  |\n|  [PVCNN (0.25  x C)](https://hanlab.mit.edu/files/pvcnn/s3dis.pvcnn.area5.c0p25.pth.tar)    |    85.00    |   **53.08**  |\n|  3D-UNet                                                                                                    |    85.12    |     54.93    |\n|  [PVCNN](https://hanlab.mit.edu/files/pvcnn/s3dis.pvcnn.area5.c1.pth.tar)                   |    86.47    |   **56.64**  |\n|  PointCNN                                                                                                   |    85.91    |     57.26    |\n|  [PVCNN++ (0.5 x C)](https://hanlab.mit.edu/files/pvcnn/s3dis.pvcnn2.area5.c0p5.pth.tar)    |    86.88    |   **58.30**  |\n|  [PVCNN++](https://hanlab.mit.edu/files/pvcnn/s3dis.pvcnn2.area5.c1.pth.tar)                |    87.48    |   **59.02**  |\n\n### ShapeNet\nWe compare PVCNN against the PointNet, PointNet++, 3D-UNet, Spider CNN and PointCNN performance as reported in the following table.\nThe accuracy is tested following [PointNet](https://github.com/charlesq34/pointnet2). The list is keeping updated.\n\n|                                                  Models                                                         |     mIoU     | \n| :-------------------------------------------------------------------------------------------------------------: | :----------: |\n|  [PointNet (Reproduce)](https://hanlab.mit.edu/files/pvcnn/shapenet.pointnet.pth.tar)           |     83.5     |\n|  PointNet                                                                                                       |     83.7     |\n|  3D-UNet                                                                                                        |     84.6     |\n|  [PVCNN (0.25 x C)](https://hanlab.mit.edu/files/pvcnn/shapenet.pvcnn.c0p25.pth.tar)            |   **84.9**   |\n|  [PointNet++ SSG (Reproduce)](https://hanlab.mit.edu/files/pvcnn/shapenet.pointnet2ssg.pth.tar) |     85.1     |\n|  PointNet++ MSG                                                                                                 |     85.1     |\n|  [PVCNN (0.25 x C, DML)](https://hanlab.mit.edu/files/pvcnn/shapenet.pvcnn.c0p25.dml.pth.tar)   |   **85.1**   |\n|  SpiderCNN                                                                                                      |     85.3     |\n|  [PointNet++ MSG (Reproduce)](https://hanlab.mit.edu/files/pvcnn/shapenet.pointnet2msg.pth.tar) |     85.3     |\n|  [PVCNN (0.5  x C)](https://hanlab.mit.edu/files/pvcnn/shapenet.pvcnn.c0p5.pth.tar)             |   **85.5**   |\n|  [PVCNN](https://hanlab.mit.edu/files/pvcnn/shapenet.pvcnn.c1.pth.tar)                          |   **85.8**   |\n|  PointCNN                                                                                                       |     86.1     |\n|  [PVCNN (DML)](https://hanlab.mit.edu/files/pvcnn/shapenet.pvcnn.c1.dml.pth.tar)                |   **86.1**   |\n\n\n### KITTI\nWe compare PVCNN (Efficient Version in the paper) against PointNets performance as reported in the following table.\nThe accuracy is tested on **val** set following [Frustum PointNets](https://github.com/charlesq34/frustum-pointnets) using \nmodified code from [kitti-object-eval-python](https://github.com/traveller59/kitti-object-eval-python).\nSince there is random sampling in Frustum Pointnets, random seed will influence the evaluation. All results provided by us\nare the average of 20 measurements with different seeds, and the best one of 20 measurements is shown in the parentheses.\nThe list is keeping updated.\n\n|                                                   Models                                                             |        Car        |        Car        |        Car        |     Pedestrian    |     Pedestrian    |     Pedestrian    |      Cyclist      |      Cyclist      |      Cyclist      |\n|:--------------------------------------------------------------------------------------------------------------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|\n|                                                                                                                      |        Easy       |      Moderate     |        Hard       |        Easy       |      Moderate     |        Hard       |        Easy       |      Moderate     |        Hard       |\n| Frustum PointNet                                                                                                     |       83.26       |       69.28       |       62.56       |         -         |         -         |         -         |         -         |         -         |         -         |\n| [Frustum PointNet (Reproduce)](https://hanlab.mit.edu/files/pvcnn/kitti.frustum.pointnet.pth.tar)    |   85.24 (85.17)   |   71.63 (71.56)   |   63.79 (63.78)   |   66.44 (66.83)   |   56.90 (57.20)   |   50.43 (50.54)   |   77.14 (78.16)   |   56.46 (57.41)   |   52.79 (53.66)   |\n| Frustum PointNet++                                                                                                   |       83.76       |       70.92       |       63.65       |       70.00       |       61.32       |       53.59       |       77.15       |       56.49       |       53.37       |\n| [Frustum PointNet++ (Reproduce)](https://hanlab.mit.edu/files/pvcnn/kitti.frustum.pointnet2.pth.tar) |   84.72 (84.46)   |   71.99 (71.95)   |   64.20 (64.13)   |   68.40 (69.27)   |   60.03 (60.80)   |   52.61 (53.19)   |   75.56 (79.41)   |   56.74 (58.65)   | 53.33 (**54.82**) |\n| [Frustum PVCNN (Efficient)](https://hanlab.mit.edu/files/pvcnn/kitti.frustum.pvcnne.pth.tar)         | **85.25 (85.30)** | **72.12 (72.22)** | **64.24 (64.36)** | **70.60 (70.60)** | **61.24 (61.35)** | **56.25 (56.38)** | **78.10 (79.79)** | **57.45 (58.72)** | **53.65 (54.81)** |\n\n\n## Testing Pretrained Models\n\nFor example, to test the downloaded pretrained models on S3DIS, one can run\n\n```\npython train.py [config-file] --devices [gpu-ids] --evaluate --configs.evaluate.best_checkpoint_path [path to the model checkpoint]\n```\n\nFor instance, to evaluate PVCNN on GPU 0,1 (with 4096 points on Area 5 of S3DIS), one can run\n\n```\npython train.py configs/s3dis/pvcnn/area5.py --devices 0,1 --evaluate --configs.evaluate.best_checkpoint_path s3dis.pvcnn.area5.c1.pth.tar\n```\n\nSpecially, for Frustum KITTI evaluation, one can specify the number of measurements to eliminate the random seed effects,\n\n```\npython train.py configs/kitti/frustum/pvcnne.py --devices 0 --evaluate --configs.evaluate.best_checkpoint_path kitti.frustum.pvcnne.pth.tar --configs.evaluate.num_tests [#measurements]\n```\n\n## Training\n\nWe provided several examples to train PVCNN with this repo:\n\n- To train PVCNN on S3DIS holding out Area 5, one can run\n \n```\npython train.py configs/s3dis/pvcnn/area5/c1.py --devices 0,1\n```\n\nIn general, to train a model, one can run\n \n```\npython train.py [config-file] --devices [gpu-ids]\n```\n\n**NOTE**: During training, the meters will provide accuracies and IoUs.\nHowever, these are just rough estimations.\nOne have to run the following command to get accurate evaluation.\n\nTo evaluate trained models, one can do inference by running:\n\n```\npython train.py [config-file] --devices [gpu-ids] --evaluate\n```\n\n## License\n\nThis repository is released under the MIT license. See [LICENSE](LICENSE) for additional details.\n\n\n## Acknowledgement\n\n- The code for data [pre-processing](data/s3dis/prepare_data.py) and [evaluation](evaluate/s3dis/eval.py) of S3DIS dataset is modified from [PointCNN](https://github.com/yangyanli/PointCNN/) (MIT License).\n\n- The code for PointNet and PointNet++ [primitive](modules/functional/src) is modified from [PointNet2](https://github.com/charlesq34/pointnet2) (MIT License) and [Pointnet2_PyTorch](https://github.com/erikwijmans/Pointnet2_PyTorch).\nWe modified the data layout and merged kernels to speed up and meet with PyTorch style.\n\n- The code for data [pre-processing](datasets/kitti/frustum.py) and [evaluation](meters/kitti/utils.py) of KITTI dataset is modified from [Frustum-Pointnets](https://github.com/charlesq34/frustum-pointnets) (Apache 2.0 License).\n\n- The code for [evaluation](evaluate/kitti/utils) of KITTI dataset is modified from [kitti-object-eval-python](https://github.com/traveller59/kitti-object-eval-python) (MIT License).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fpvcnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-han-lab%2Fpvcnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fpvcnn/lists"}