{"id":13443358,"url":"https://github.com/dvlab-research/FocalsConv","last_synced_at":"2025-03-20T16:31:07.705Z","repository":{"id":37352787,"uuid":"469622370","full_name":"dvlab-research/FocalsConv","owner":"dvlab-research","description":"Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)","archived":false,"fork":false,"pushed_at":"2023-06-01T03:23:11.000Z","size":7011,"stargazers_count":370,"open_issues_count":5,"forks_count":35,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-28T06:58:05.324Z","etag":null,"topics":["3d-object-detection","autonomous-driving","kitti","nuscenes","sparse-convolution"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2204.12463","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dvlab-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-14T07:22:29.000Z","updated_at":"2024-10-16T02:31:18.000Z","dependencies_parsed_at":"2024-01-16T02:56:07.173Z","dependency_job_id":null,"html_url":"https://github.com/dvlab-research/FocalsConv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FFocalsConv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FFocalsConv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FFocalsConv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvlab-research%2FFocalsConv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dvlab-research","download_url":"https://codeload.github.com/dvlab-research/FocalsConv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244649811,"owners_count":20487496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-object-detection","autonomous-driving","kitti","nuscenes","sparse-convolution"],"created_at":"2024-07-31T03:01:59.637Z","updated_at":"2025-03-20T16:31:06.941Z","avatar_url":"https://github.com/dvlab-research.png","language":"Python","funding_links":[],"categories":["Python","3. Perception"],"sub_categories":["3.1.1 Vision based"],"readme":"[![arXiv](https://img.shields.io/badge/arXiv-Paper-\u003cCOLOR\u003e.svg)](https://arxiv.org/abs/2204.12463)\n![visitors](https://visitor-badge.glitch.me/badge?page_id=dvlab-research/FocalsConv)\n\n\n# Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)\n\nThis is the official implementation of ***Focals Conv*** (CVPR 2022), a new sparse convolution design for 3D object detection (feasible for both lidar-only and multi-modal settings). For more details, please refer to:\n\n**Focal Sparse Convolutional Networks for 3D Object Detection [[Paper](https://arxiv.org/abs/2204.12463)]** \u003cbr /\u003e\nYukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, Jiaya Jia\u003cbr /\u003e\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"docs/imgs/FocalSparseConv23D.png\" width=\"100%\"\u003e \u003c/p\u003e\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"docs/imgs/FocalSparseConv_Pipeline.png\" width=\"100%\"\u003e \u003c/p\u003e\n\n## News\n- [2023-01-05] The ***CUDA version*** of Focals Conv is released in [spconv-plus](https://github.com/dvlab-research/spconv-plus), including some other sparse operators. The example for using it can be found [here](https://github.com/dvlab-research/FocalsConv/blob/master/OpenPCDet/pcdet/models/backbones_3d/focal_sparse_conv/focal_sparse_conv_cuda.py).\n- [2022-08-24] The code and example for ***test-time augmentations*** have been released [here](https://github.com/dvlab-research/FocalsConv/tree/master/CenterPoint/test_aug_examples).\n- [2022-07-05] The code for ***Focals Conv*** has been marged into the official codebase [OpenPCDet](https://github.com/open-mmlab/OpenPCDet).\n- [2022-06-21] The other 3D backbone network design is presented ***LargeKernel3D*** [[Paper](https://arxiv.org/abs/2206.10555) \\| [Github](https://github.com/dvlab-research/LargeKernel3D)]. \n\n\n### Experimental results\n\n#### KITTI dataset\n|                                             | Car@R11 | Car@R40  |download | \n|---------------------------------------------|-------:|:-------:|:---------:|\n| [PV-RCNN + Focals Conv](OpenPCDet/tools/cfgs/kitti_models/pv_rcnn_focal_lidar.yaml) | 83.91 | 85.20 | [Google](https://drive.google.com/file/d/1XOpIzHKtkEj9BNrQR6VYADO_T5yaOiJq/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1t1Gk8bDv8Q_Dd5vB4VtChA) (key: m15b) |\n| [PV-RCNN + Focals Conv (multimodal)](OpenPCDet/tools/cfgs/kitti_models/pv_rcnn_focal_multimodal.yaml) | 84.58 | 85.34 | [Google](https://drive.google.com/file/d/183araPcEmYSlruife2nszKeJv1KH2spg/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/10XodrSazMFDFnTRdKIfbKA) (key: ie6n) |\n| [Voxel R-CNN (Car) + Focals Conv (multimodal)](OpenPCDet/tools/cfgs/kitti_models/voxel_rcnn_car_focal_multimodal.yaml) | 85.68 | 86.00 | [Google](https://drive.google.com/file/d/1M7IUosz4q4qHKEZeRLIIBQ6Wj1-0Wjdg/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1bIN3zDmPXrURMOPg7pukzA) (key: tnw9) |\n\n\n#### nuScenes dataset\n\n|                                             | mAP | NDS | download | \n|---------------------------------------------|----------:|:-------:|:---------:|\n| [CenterPoint + Focals Conv (multi-modal)](CenterPoint/configs/nusc/voxelnet/nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_focal_multimodal.py) | 63.86\t| 69.41\t | [Google](https://drive.google.com/file/d/12VXMl6RQcz87OWPxXJsB_Nb0MdimsTiG/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1ZXn-fhmeL6AsveV2G3n5Jg) (key: 01jh) | \n| [CenterPoint + Focals Conv (multi-modal) - 1/4 data](CenterPoint/configs/nusc/voxelnet/nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_focal_multimodal_1_4_data.py) | 62.15\t| 67.45\t | [Google](https://drive.google.com/file/d/1HC3nTEE8GVhInquwRd9hRJPSsZZylR58/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1tKlO4GgzjXojzjzpoJY_Ng) (key: 6qsc) | \n\nVisualization of voxel distribution of Focals Conv on KITTI val dataset:\n\u003cp align=\"center\"\u003e \u003cimg src=\"docs/imgs/Sparsity_comparison_3pairs.png\" width=\"100%\"\u003e \u003c/p\u003e\n\n\n\n## Getting Started\n### Installation\n\n#### a. Clone this repository\n```shell\nhttps://github.com/dvlab-research/FocalsConv \u0026\u0026 cd FocalsConv\n```\n#### b. Install the environment\n\nFollowing the install documents for [OpenPCdet](OpenPCDet/docs/INSTALL.md) and [CenterPoint](CenterPoint/docs/INSTALL.md) codebases respectively, based on your preference.\n\n*spconv 2.x is highly recommended instead of spconv 1.x version.\n\n#### c. Prepare the datasets. \n\nDownload and organize the official [KITTI](OpenPCDet/docs/GETTING_STARTED.md) and [Waymo](OpenPCDet/docs/GETTING_STARTED.md) following the document in OpenPCdet, and [nuScenes](CenterPoint/docs/NUSC.md) from the CenterPoint codebase.\n\n*Note that for nuScenes dataset, we use image-level gt-sampling (copy-paste) in the multi-modal training.\nPlease download this [dbinfos_train_10sweeps_withvelo.pkl](https://drive.google.com/file/d/1ypJKpZifM-NsGdUSLMFpBo-KaXlfpplR/view?usp=sharing) to replace the original one. ([Google](https://drive.google.com/file/d/1ypJKpZifM-NsGdUSLMFpBo-KaXlfpplR/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1iz1KWthc1XhXG3du3QG__w) (key: b466))\n\n*Note that for nuScenes dataset, we conduct ablation studies on a 1/4 data training split. \nPlease download [infos_train_mini_1_4_10sweeps_withvelo_filter_True.pkl](https://drive.google.com/file/d/19-Zo8o0OWZYed0UpnOfDqTY5oLXKJV9Q/view?usp=sharing) if you needed for training. ([Google](https://drive.google.com/file/d/19-Zo8o0OWZYed0UpnOfDqTY5oLXKJV9Q/view?usp=sharing) \\| [Baidu](https://pan.baidu.com/s/1VbkNBs155JyJLhNtSlbEGQ) (key: 769e))\n\n#### d. Download pre-trained models.\nIf you want to directly evaluate the trained models we provide, please download them first.\n\nIf you want to train by yourselvef, for multi-modal settings, please download this resnet pre-train model first,\n[torchvision-res50-deeplabv3](https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth).\n\n\n### Evaluation\nWe provide the trained weight file so you can just run with that. You can also use the model you trained.\n\nFor models in OpenPCdet, \n```shell\nNUM_GPUS=8\ncd tools \nbash scripts/dist_test.sh ${NUM_GPUS} --cfg_file cfgs/kitti_models/voxel_rcnn_car_focal_multimodal.yaml --ckpt path/to/voxelrcnn_focal_multimodal.pth\n\nbash scripts/dist_test.sh ${NUM_GPUS} --cfg_file cfgs/kitti_models/pv_rcnn_focal_multimodal.yaml --ckpt ../pvrcnn_focal_multimodal.pth\n\nbash scripts/dist_test.sh ${NUM_GPUS} --cfg_file cfgs/kitti_models/pv_rcnn_focal_lidar.yaml --ckpt path/to/pvrcnn_focal_lidar.pth\n```\n\nFor models in CenterPoint, \n```shell\nCONFIG=\"nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_focal_multimodal\"\npython -m torch.distributed.launch --nproc_per_node=${NUM_GPUS} ./tools/dist_test.py configs/nusc/voxelnet/$CONFIG.py --work_dir ./work_dirs/$CONFIG --checkpoint centerpoint_focal_multimodal.pth\n```\n\n\n### Training\n\nFor configures in OpenPCdet,\n```shell\nbash scripts/dist_train.sh ${NUM_GPUS} --cfg_file cfgs/kitti_models/CONFIG.yaml\n```\n\nFor configures in CenterPoint,\n```shell\npython -m torch.distributed.launch --nproc_per_node=${NUM_GPUS} ./tools/train.py configs/nusc/voxelnet/$CONFIG.py --work_dir ./work_dirs/CONFIG\n```\n\n* Note that we use 8 GPUs to train OpenPCdet models and 4 GPUs to train CenterPoint models.\n\n* Note that for model size counting of multi-modal model, please refer to this [issue](https://github.com/dvlab-research/FocalsConv/issues/9).\n\n## Citation \nIf you find this project useful in your research, please consider citing:\n\n```\n@inproceedings{focalsconv-chen,\n  title={Focal Sparse Convolutional Networks for 3D Object Detection},\n  author={Chen, Yukang and Li, Yanwei and Zhang, Xiangyu and Sun, Jian and Jia, Jiaya},\n  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},\n  year={2022}\n}\n```\n\n## Acknowledgement\n-  This work is built upon the `OpenPCDet` and `CenterPoint`. Please refer to the official github repositories, [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) and [CenterPoint](https://github.com/tianweiy/CenterPoint) for more information.\n\n-  This README follows the style of [IA-SSD](https://github.com/yifanzhang713/IA-SSD).\n\n\n\n## License\n\nThis project is released under the [Apache 2.0 license](LICENSE).\n\n\n## Related Repos\n1. [spconv](https://github.com/traveller59/spconv) ![GitHub stars](https://img.shields.io/github/stars/traveller59/spconv.svg?style=flat\u0026label=Star)\n2. [Deformable Conv](https://github.com/msracver/Deformable-ConvNets) ![GitHub stars](https://img.shields.io/github/stars/msracver/Deformable-ConvNets.svg?style=flat\u0026label=Star)\n3. [Submanifold Sparse Conv](https://github.com/facebookresearch/SparseConvNet) ![GitHub stars](https://img.shields.io/github/stars/facebookresearch/SparseConvNet.svg?style=flat\u0026label=Star)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2FFocalsConv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdvlab-research%2FFocalsConv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvlab-research%2FFocalsConv/lists"}