{"id":13443535,"url":"https://github.com/happinesslz/EPNet","last_synced_at":"2025-03-20T16:31:40.960Z","repository":{"id":64496187,"uuid":"276774529","full_name":"happinesslz/EPNet","owner":"happinesslz","description":"EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection(ECCV 2020)","archived":false,"fork":false,"pushed_at":"2024-07-25T10:57:06.000Z","size":453,"stargazers_count":235,"open_issues_count":23,"forks_count":36,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-10-28T06:58:30.015Z","etag":null,"topics":["3d-object-detection","kitti-3d","multimodal"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/happinesslz.png","metadata":{"files":{"readme":"README.PointRCNN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-03T00:55:03.000Z","updated_at":"2024-10-15T04:14:55.000Z","dependencies_parsed_at":"2024-10-28T04:53:37.154Z","dependency_job_id":"1b8966d8-68db-467d-b6e0-5135f92e9d2e","html_url":"https://github.com/happinesslz/EPNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FEPNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FEPNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FEPNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FEPNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/happinesslz","download_url":"https://codeload.github.com/happinesslz/EPNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244649899,"owners_count":20487517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-object-detection","kitti-3d","multimodal"],"created_at":"2024-07-31T03:02:03.160Z","updated_at":"2025-03-20T16:31:40.952Z","avatar_url":"https://github.com/happinesslz.png","language":"Python","funding_links":[],"categories":["Python","Code list"],"sub_categories":[],"readme":"# PointRCNN\n\n## PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud\n![teaser](https://github.com/sshaoshuai/PointRCNN/blob/master/doc/teaser.png)\n\nCode release for the paper **PointRCNN:3D Object Proposal Generation and Detection from Point Cloud**, CVPR 2019.\n\n**Authors**: [Shaoshuai Shi](https://sshaoshuai.github.io/), [Xiaogang Wang](http://www.ee.cuhk.edu.hk/~xgwang/), [Hongsheng Li](http://www.ee.cuhk.edu.hk/~hsli/).\n\n[[arXiv]](https://arxiv.org/abs/1812.04244)\u0026nbsp;  [[Project Page]](#)\u0026nbsp;\n\n\n**New:** We have provided another implementation of PointRCNN for joint training with multi-class in a general 3D object detection toolbox [[OpenPCDet]](https://github.com/open-mmlab/OpenPCDet). \n\n\n## Introduction\nIn this work, we propose the PointRCNN 3D object detector to directly generated accurate 3D box proposals from raw point cloud in a bottom-up manner, which are then refined in the canonical coordinate by the proposed bin-based 3D box regression loss. \nTo the best of our knowledge, PointRCNN is **the first two-stage 3D object detector** for 3D object detection by using only the raw point cloud as input. PointRCNN is evaluated on the KITTI dataset and achieves state-of-the-art performance on the KITTI 3D object detection [leaderboard](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) among all published works at the time of submission.\n\nFor more details of PointRCNN, please refer to [our paper](https://arxiv.org/abs/1812.04244) or [project page](#).\n\n### Supported features and ToDo list\n- [x] Multiple GPUs for training\n- [x] GPU version rotated NMS\n- [x] Faster PointNet++ inference and training supported by [Pointnet2.PyTorch](https://github.com/sshaoshuai/Pointnet2.PyTorch)\n- [x] PyTorch 1.0\n- [x] TensorboardX\n- [ ] Still in progress\n\n## Installation\n### Requirements\nAll the codes are tested in the following environment:\n* Linux (tested on Ubuntu 14.04/16.04)\n* Python 3.6+\n* PyTorch 1.0\n\n### Install PointRCNN \n\na. Clone the PointRCNN repository.\n```shell\ngit clone --recursive https://github.com/sshaoshuai/PointRCNN.git\n```\nIf you forget to add the `--recursive` parameter, just run the following command to clone the `Pointnet2.PyTorch` submodule.\n```shell\ngit submodule update --init --recursive\n```\n\nb. Install the dependent python libraries like `easydict`,`tqdm`, `tensorboardX ` etc.\n\nc. Build and install the `pointnet2_lib`, `iou3d`, `roipool3d` libraries by executing the following command:\n```shell\nsh build_and_install.sh\n```\n\n## Dataset preparation\nPlease download the official [KITTI 3D object detection](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) dataset and organize the downloaded files as follows: \n```\nPointRCNN\n├── data\n│   ├── KITTI\n│   │   ├── ImageSets\n│   │   ├── object\n│   │   │   ├──training\n│   │   │      ├──calib \u0026 velodyne \u0026 label_2 \u0026 image_2 \u0026 (optional: planes)\n│   │   │   ├──testing\n│   │   │      ├──calib \u0026 velodyne \u0026 image_2\n├── lib\n├── pointnet2_lib\n├── tools\n```\nHere the images are only used for visualization and the [road planes](https://drive.google.com/file/d/1d5mq0RXRnvHPVeKx6Q612z0YRO1t2wAp/view?usp=sharing) are optional for data augmentation in the training. \n\n\n## Pretrained model\nYou could download the pretrained model(Car) of PointRCNN from [here(~15MB)](https://drive.google.com/file/d/1aapMXBkSn5c5hNTDdRNI74Ptxfny7PuC/view?usp=sharing), which is trained on the *train* split (3712 samples) and evaluated on the *val* split (3769 samples) and *test* split (7518 samples). The performance on validation set is as follows:\n```\nCar AP@0.70, 0.70, 0.70:\nbbox AP:96.91, 89.53, 88.74\nbev  AP:90.21, 87.89, 85.51\n3d   AP:89.19, 78.85, 77.91\naos  AP:96.90, 89.41, 88.54\n```\n### Quick demo\nYou could run the following command to evaluate the pretrained model (set `RPN.LOC_XZ_FINE=False` since it is a little different with the default configuration): \n```\npython eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt PointRCNN.pth --batch_size 1 --eval_mode rcnn --set RPN.LOC_XZ_FINE False\n```\n\n## Inference\n* To evaluate a single checkpoint, run the following command with `--ckpt` to specify the checkpoint to be evaluated:\n```\npython eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt ../output/rpn/ckpt/checkpoint_epoch_200.pth --batch_size 4 --eval_mode rcnn \n```\n\n* To evaluate all the checkpoints of a specific training config file, add the `--eval_all` argument, and run the command as follows:\n```\npython eval_rcnn.py --cfg_file cfgs/default.yaml --eval_mode rcnn --eval_all\n```\n\n* To generate the results on the *test* split, please modify the `TEST.SPLIT=TEST` and add the `--test` argument. \n\nHere you could specify a bigger `--batch_size` for faster inference based on your GPU memory. Note that the `--eval_mode` argument should be consistent with the `--train_mode` used in the training process. If you are using `--eval_mode=rcnn_offline`, then you should use `--rcnn_eval_roi_dir` and `--rcnn_eval_feature_dir` to specify the saved features and proposals of the validation set. Please refer to the training section for more details. \n\n## Training\nCurrently, the two stages of PointRCNN are trained separately. Firstly, to use the ground truth sampling data augmentation for training, we should generate the ground truth database as follows:\n```\npython generate_gt_database.py --class_name 'Car' --split train\n```\n\n### Training of RPN stage\n* To train the first proposal generation stage of PointRCNN with a single GPU, run the following command:\n```\npython train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 16 --train_mode rpn --epochs 200\n```\n\n* To use **mutiple GPUs for training**, simply add the `--mgpus` argument as follows:\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 16 --train_mode rpn --epochs 200 --mgpus\n```\n\nAfter training, the checkpoints and training logs will be saved to the corresponding directory according to the name of your configuration file. Such as for the `default.yaml`, you could find the checkpoints and logs in the following directory:\n```\nPointRCNN/output/rpn/default/\n```\nwhich will be used for the training of RCNN stage. \n\n### Training of RCNN stage\nSuppose you have a well-trained RPN model saved at `output/rpn/default/ckpt/checkpoint_epoch_200.pth`, \nthen there are two strategies to train the second stage of PointRCNN. \n\n(a) Train RCNN network with fixed RPN network to use online GT augmentation: Use `--rpn_ckpt` to specify the path of a well-trained RPN model and run the command as follows:\n```\npython train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --train_mode rcnn --epochs 70  --ckpt_save_interval 2 --rpn_ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth\n```\n(b) Train RCNN network with offline GT augmentation: \n1. Generate the augmented offline scenes by running the following command:\n```\npython generate_aug_scene.py --class_name Car --split train --aug_times 4\n```\n2. Save the RPN features and proposals by adding `--save_rpn_feature`:\n\n* To save features and proposals for the training, we set `TEST.RPN_POST_NMS_TOP_N=300` and `TEST.RPN_NMS_THRESH=0.85` as follows:\n```\npython eval_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --eval_mode rpn --ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth --save_rpn_feature --set TEST.SPLIT train_aug TEST.RPN_POST_NMS_TOP_N 300 TEST.RPN_NMS_THRESH 0.85\n```\n\n* To save features and proposals for the evaluation, we keep `TEST.RPN_POST_NMS_TOP_N=100` and `TEST.RPN_NMS_THRESH=0.8` as default:\n```\npython eval_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --eval_mode rpn --ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth --save_rpn_feature\n```\n3. Now we could train our RCNN network. Note that you should modify `TRAIN.SPLIT=train_aug` to use the augmented scenes for the training, and use `--rcnn_training_roi_dir` and `--rcnn_training_feature_dir` to specify the saved features and proposals in the above step:\n```\npython train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --train_mode rcnn_offline --epochs 30  --ckpt_save_interval 1 --rcnn_training_roi_dir ../output/rpn/default/eval/epoch_200/train_aug/detections/data --rcnn_training_feature_dir ../output/rpn/default/eval/epoch_200/train_aug/features\n```\nFor the offline GT sampling augmentation, the default setting to train the RCNN network is `RCNN.ROI_SAMPLE_JIT=True`, which means that we sample the RoIs and calculate their GTs in the GPU. I also provide the CPU version proposal sampling, which is implemented in the dataloader, and you could enable this feature by setting `RCNN.ROI_SAMPLE_JIT=False`. Typically the CPU version is faster but costs more CPU resources since they use mutiple workers.  \n\nAll the codes supported **mutiple GPUs**, simply add the `--mgpus` argument as above. And you could also increase the `--batch_size` by using multiple GPUs for training.\n\n**Note**: \n* The strategy (a), online augmentation, is more elegant and easy to train.\n* The best model is trained by the offline augmentation strategy with CPU proposal sampling (set `RCNN.ROI_SAMPLE_JIT=False`). \n* Theoretically, the online augmentation should be better, but currently the online augmentation is a bit lower than the offline augmentation, and I still didn't know why. All discussions are welcomed.\n* I am still working on this codes to make it more stable. \n\n## Citation\nIf you find this work useful in your research, please consider cite:\n```\n@InProceedings{Shi_2019_CVPR,\n    author = {Shi, Shaoshuai and Wang, Xiaogang and Li, Hongsheng},\n    title = {PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud},\n    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month = {June},\n    year = {2019}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhappinesslz%2FEPNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhappinesslz%2FEPNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhappinesslz%2FEPNet/lists"}