{"id":13443878,"url":"https://github.com/AutoVision-cloud/SA-Det3D","last_synced_at":"2025-03-20T17:32:08.171Z","repository":{"id":38372042,"uuid":"317713434","full_name":"AutoVision-cloud/SA-Det3D","owner":"AutoVision-cloud","description":"[ICCVW-2021] SA-Det3D: Self-attention based Context-Aware 3D Object Detection","archived":false,"fork":false,"pushed_at":"2022-05-05T22:54:36.000Z","size":1777,"stargazers_count":166,"open_issues_count":7,"forks_count":32,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-28T07:41:20.244Z","etag":null,"topics":["3d-deep-learning","3d-detection","autonomous-driving","object-detection","point-cloud","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AutoVision-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-02T01:20:47.000Z","updated_at":"2024-10-11T06:18:54.000Z","dependencies_parsed_at":"2022-08-25T01:40:49.299Z","dependency_job_id":null,"html_url":"https://github.com/AutoVision-cloud/SA-Det3D","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoVision-cloud%2FSA-Det3D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoVision-cloud%2FSA-Det3D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoVision-cloud%2FSA-Det3D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AutoVision-cloud%2FSA-Det3D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AutoVision-cloud","download_url":"https://codeload.github.com/AutoVision-cloud/SA-Det3D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244660626,"owners_count":20489367,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-deep-learning","3d-detection","autonomous-driving","object-detection","point-cloud","pytorch"],"created_at":"2024-07-31T03:02:12.738Z","updated_at":"2025-03-20T17:32:03.159Z","avatar_url":"https://github.com/AutoVision-cloud.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection\n\nBy [Prarthana Bhattacharyya](https://scholar.google.com/citations?user=v6pGkNQAAAAJ\u0026hl=en), [Chengjie Huang](https://scholar.google.com/citations?user=O6gvGZgAAAAJ\u0026hl=en) and [Krzysztof Czarnecki](https://scholar.google.com/citations?hl=en\u0026user=ZzCpumQAAAAJ).\n\nWe provide code support and configuration files to reproduce the results in the paper:\n[SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection](https://arxiv.org/abs/2101.02672). \n\u003cbr/\u003e Our code is based on [OpenPCDet](https://github.com/open-mmlab/OpenPCDet), which is a clean open-sourced project for benchmarking 3D object detection methods. \n\n\n## Overview\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/overview.png\" width=\"600px\" height=\"280px\" /\u003e\n  \u003cp\u003eFig.1. Self-Attention augmented global-context aware backbone networks. \u003c/p\u003e\n\u003c/div\u003e\n\nIn this paper, we explore variations of \nself-attention for contextual modeling in 3D object \ndetection by augmenting convolutional features with \nself-attention features. \n\u003cbr/\u003e\nWe first incorporate the pairwise self-attention mechanism into the  current \nstate-of-the-art BEV, voxel, point and point-voxel based detectors and show \nconsistent improvement over strong baseline models while simultaneously \nsignificantly reducing their parameter footprint and computational cost. \nWe call this variant *full self-attention* (*FSA*).\n\u003cbr/\u003e\nWe also propose a self-attention variant that \nsamples a subset of the most representative features by \nlearning deformations over randomly sampled locations.\nThis not only allows us to scale explicit global contextual \nmodeling to larger point-clouds, \nbut also leads to more discriminative and informative feature \ndescriptors. We call this variant *deformable self-attention* (*DSA*).\n\n\n## Results\n- Self-attention (SA) systematically \nimproves 3D object detection across  state-of-the-art  3D  detectors: PointPillars, SECOND and \nPoint-RCNN. In this figure, we show 3D AP on moderate Car class of KITTI val split (R40) vs. the \nnumber of parameters (Top) and \nGFLOPs (Bottom) for baseline models and proposed baseline extensions with \nDeformable SA (DSA) and Full SA (FSA).\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/demo_params_flops.png\" width=\"300px\" /\u003e\n  \u003cp\u003eFig.2. 3D Car AP with respect to params and FLOPs of baseline and proposed \nself-attention variants. \u003c/p\u003e\n\u003c/div\u003e\n\u003cbr/\u003e\n\n- We also illustrate qualitative performance on KITTI val split. We show that our method \nidentifies missed detections and removes false positives. Red bounding box \nrepresents ground truth and green represents detector outputs. \nFrom left to right: (a) RGB image of challenging scenes. \n(b) Result of the state-of-the-art methods: PointPillars, \nSECOND, Point-RCNN and PV-RCNN. (c) Result of our full self-attention (FSA) \naugmented baselines, which uses significantly fewer \nparameters and FLOPs.\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/demo_qual.png\" width=\"600px\" /\u003e\n  \u003cp\u003eFig.3. Visualizing qualitative results between baseline and\nour proposed self-attention module.\u003c/p\u003e\n\u003c/div\u003e\n\n## Model Zoo\nWe provide our proposed detection models in this section. The 3D \nAP results (R-40) on KITTI 3D Object Detection validation of the \nCar moderate category are shown in the table below.\n\n**Notes:**\n- For inference, our models have been tested with 1 Tesla V-100 GPU and Pytorch 1.3.\n- We use the checkpoints released by [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) as our baseline for evaluation.\n- Our models are trained with 4 Tesla V-100 GPUs and Pytorch 1.3.\n\u003cbr/\u003e\n\n|                                                     | Car 3D AP | Params (M) | G-FLOPs | download | \n|-----------------------------------------------------|:-------:|:-------:|:-------:|:---------:|\n| [PointPillar_baseline](https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/pointpillar.yaml) | 78.39 | 4.8 | 63.4 | [PointPillar](https://drive.google.com/file/d/1tXU14qouG6EWo5QkjwP9KK22NSyWq1Bf/view?usp=sharing) |\n| [PointPillar_red](configs/pointpillar_red.yaml) | 78.07 | 1.5 | 31.5 | [PointPillar-red](https://drive.google.com/file/d/1xQFhCEuiAWr8oLxL-xGYSqqCKVcrx9au/view?usp=sharing) |\n| [PointPillar_DSA](configs/pointpillar_dsa.yaml) | 78.94 | 1.1 | 32.4 | [PointPillar-DSA](https://drive.google.com/file/d/1pPH5woN9IBQeWxdVdlcO03k3zRCtEjoJ/view?usp=sharing) |\n| [PointPillar_FSA](configs/pointpillar_fsa.yaml) | **79.04** | 1.0 | 31.7 | [PointPillar-FSA](https://drive.google.com/file/d/1pVkS_zf7XMnwkWtyaxfgZ0a7_kF_ghmV/view?usp=sharing) |\n| [SECOND_baseline](https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/second.yaml) | 81.61 | 4.6 | 76.7 | [SECOND](https://drive.google.com/file/d/16N3zqCTk6j973XY26mdx2KwudsxRZ2r0/view?usp=sharing) |\n| [SECOND_red](configs/second_red.yaml) | 81.11 | 2.5 | 51.2 | [SECOND-red](https://drive.google.com/file/d/1HhUtyWu8JVSDPImyAe2NBVyIqyroTEXG/view?usp=sharing) |\n| [SECOND_DSA](configs/second_dsa.yaml) | **82.03** | 2.2 | 52.6 | [SECOND-DSA](https://drive.google.com/file/d/1lbgI5ZOB8WhMiTgK0vTw26pw_BdZ9J87/view?usp=sharing) |\n| [SECOND_FSA](configs/second_fsa.yaml) | 81.86 | 2.2 | 51.9 | [SECOND-FSA](https://drive.google.com/file/d/1hyIBqFdql_FtEgavq6rEzUbIbPztQT7r/view?usp=sharing) |\n| [Point-RCNN_baseline](https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/pointrcnn_iou.yaml) | 80.52 | 4.0 | 27.4 | [Point-RCNN](https://drive.google.com/file/d/1Hab5lfwmqnRf928jIMJur5xt4PzspoOY/view?usp=sharing) |\n| [Point-RCNN_red](configs/pointrcnn_red.yaml) | 80.40 | 2.2 | 24 | [Point-RCNN-red](https://drive.google.com/file/d/1jD-TzJwKYfEYzSVTmErU9AzO9d0uiCos/view?usp=sharing) |\n| [Point-RCNN_DSA](configs/pointrcnn_dsa.yaml) | 81.80 | 2.3 | 19.3 | [Point-RCNN-DSA](https://drive.google.com/file/d/1buJs9bNwl0YzP7uLlEjdfkHcqbvVMKAN/view?usp=sharing) |\n| [Point-RCNN_FSA](configs/pointrcnn_fsa.yaml) | **82.10** | 2.5 | 19.8 | [Point-RCNN-FSA](https://drive.google.com/file/d/1i_My1kHZNt6n5QAgO-AoiNRxjOwEG4R0/view?usp=sharing) |\n| [PV-RCNN_baseline](https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/pv_rcnn.yaml) | 84.83 | 12 | 89 | [PV-RCNN](https://drive.google.com/file/d/11twTR4wN0ZmEQE2Bh6Cb2gO7qc9z1b8p/view?usp=sharing) |\n| [PV-RCNN_DSA](configs/pvrcnn_dsa.yaml) | 84.71 | 10 | 64 | [PV-RCNN-DSA](https://drive.google.com/file/d/1-iLeNQ4XdbN7ncVBuHXd-e-L3pwpoAyB/view?usp=sharing) |\n| [PV-RCNN_FSA](configs/pvrcnn_fsa.yaml) | **84.95** | 10 | 64.3 | [PV-RCNN-FSA](https://drive.google.com/file/d/1hsy3nCO6Xk6cW3VP0SYLojhUpmqzYRab/view?usp=sharing) |\n\n\n## Usage\na. Clone the repo:\n```\ngit clone --recursive https://github.com/AutoVision-cloud/SA-Det3D\n```\nb. Copy SA-Det3D src into OpenPCDet: \n```\nsh ./init.sh\n```\n\nc. Install OpenPCDet and prepare KITTI data:\n\nPlease refer to [INSTALL.md](docs/INSTALL.md) for installation and dataset preparation.\n\nd. Run experiments with a specific configuration file:\n\nPlease refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md) to learn more about how to train and run inference on this detector.\n\n\n\n## Citation\nIf you find this project useful in your research, please consider citing:\n```\n@misc{bhattacharyya2021sadet3d,\n      title={SA-Det3D: Self-Attention Based Context-Aware 3D Object Detection}, \n      author={Prarthana Bhattacharyya and Chengjie Huang and Krzysztof Czarnecki},\n      year={2021},\n      eprint={2101.02672},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\n\n## Acknowledgement\n* [second.pytorch](https://github.com/traveller59/second.pytorch)\n* [PointRCNN](https://github.com/sshaoshuai/PointRCNN)\n* [PV-RCNN](https://github.com/open-mmlab/OpenPCDet)\n* [MLCVNet](https://github.com/NUAAXQ/MLCVNet)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAutoVision-cloud%2FSA-Det3D","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAutoVision-cloud%2FSA-Det3D","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAutoVision-cloud%2FSA-Det3D/lists"}