{"id":13443641,"url":"https://github.com/happinesslz/TANet","last_synced_at":"2025-03-20T17:30:47.470Z","repository":{"id":43114133,"uuid":"229737382","full_name":"happinesslz/TANet","owner":"happinesslz","description":"The  AAAI-2020 Paper(Oral):\"TANet: Robust 3D Object Detection from Point Clouds with Triple Attention\"","archived":false,"fork":false,"pushed_at":"2020-11-23T10:57:10.000Z","size":4484,"stargazers_count":253,"open_issues_count":6,"forks_count":49,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-10-28T07:39:32.144Z","etag":null,"topics":["3d-object-detection"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/happinesslz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-23T11:18:09.000Z","updated_at":"2024-10-22T07:59:28.000Z","dependencies_parsed_at":"2022-08-30T05:32:03.576Z","dependency_job_id":null,"html_url":"https://github.com/happinesslz/TANet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FTANet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FTANet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FTANet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/happinesslz%2FTANet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/happinesslz","download_url":"https://codeload.github.com/happinesslz/TANet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244660194,"owners_count":20489296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-object-detection"],"created_at":"2024-07-31T03:02:05.975Z","updated_at":"2025-03-20T17:30:46.218Z","avatar_url":"https://github.com/happinesslz.png","language":"Python","funding_links":[],"categories":["Python","Code list"],"sub_categories":[],"readme":"# TANet\n\nOur code is mainly based on [PointPillars](https://github.com/nutonomy/second.pytorch) and [SECOND](https://github.com/traveller59/second.pytorch), thanks for them! We also plan to introduce TensorRT to further improve the inference speed. For more information please refer the [Paper](https://arxiv.org/pdf/1912.05163.pdf).  \n### NEWS UPDATE!  \n**Add TANet to the newest [second.pytorch](https://github.com/traveller59/second.pytorch).** \nTANet in second.pytorch package achieves the same performance with pointpillars_with_TANet, so I suggest you use second.pytorch_with_TANet instead. In addition, only using TA moudle in Nuscenes achieves an obvious  improvement than pointpillars in Nuscenes dataset.  (Note: On Nuscenes dataset, don't use PSA or other refinemnets, just use the TA module.  )\n\n\n## The network architecture of [TANet](https://arxiv.org/pdf/1912.05163.pdf)\n![image](imgs/TANet.png)\n    First, we equally divide the point clouds into a voxel grid consisting of a set of voxels. Then, the stacked triple attention separately process each voxel to obtain a more discriminative representation. Subsequently, a compact feature representation for each voxel is extracted by aggregating the points inside it in a max-pooling manner. And we arrange the voxel feature according to its original spatial position in the grid, and thus lead to a feature representation for the voxel grid in the shape of C' × H × W . Finally, the coarse-to-fine regression is employed to generate the final 3D bounding boxes.\n\n\n## Visualization\n\nThe visualization of learned feature map and predicted confidence score for PointPillars and Ours:\n![image](imgs/Result_Vis.png)\n\n## Discussion for TANet\n\n\n\n## TANet for [PointPillars](https://github.com/nutonomy/second.pytorch) \n### Start Up\nExperiment: Tested in Ubuntu 16.04 with python 3.6+, pytorch 1.0.0+, CUDA9.0 in Titan XP/Titan V.\n\n### Install:\nyou can refer the following steps or directly refer  [PointPillars](https://github.com/nutonomy/second.pytorch) \n\n#### 1. Clone code\n\tgit clone https://github.com/happinesslz/TANet.git\n\n#### 2. Install dependence python packages\n\n\tconda install scikit-image scipy numba pillow matplotlib\n\tpip install fire tensorboardX protobuf opencv-python shapely easydict pybind11\n\n\n#### 3. Setup cuda for numba\n\nAdd following environment variables for numba to ~/.bashrc (**Optional**):\n\n```\nexport NUMBAPRO_CUDA_DRIVER=/usr/lib/x86_64-linux-gnu/libcuda.so\nexport NUMBAPRO_NVVM=/usr/local/cuda-9.0/nvvm/lib64/libnvvm.so\nexport NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.0/nvvm/libdevice\n```\n\tsource ~/.bashrc\n\n#### 4. PYTHONPATH\n\nAdd TANet/pointpillars_with_TANet/ to your PYTHONPATH.\n\n### Prepare dataset\n#### 1. Dataset preparation\n\nDownload KITTI dataset and create some directories first:\n\n```plain\n└── KITTI_DATASET_ROOT\n       ├── training    \u003c-- 7481 train data\n       |   ├── image_2 \u003c-- for visualization\n       |   ├── calib\n       |   ├── label_2\n       |   ├── velodyne\n       |   └── velodyne_reduced \u003c-- empty directory\n       └── testing     \u003c-- 7580 test data\n           ├── image_2 \u003c-- for visualization\n           ├── calib\n           ├── velodyne\n           └── velodyne_reduced \u003c-- empty directory\n```\n\nNote: PointPillar's protos use ```KITTI_DATASET_ROOT=/data/sets/kitti_second/```.\n\n#### 2. Create kitti infos:\n\n```bash\npython create_data.py create_kitti_info_file --data_path=KITTI_DATASET_ROOT\n```\n\n#### 3. Create reduced point cloud:\n\n```bash\npython create_data.py create_reduced_point_cloud --data_path=KITTI_DATASET_ROOT\n```\n\n#### 4. Create groundtruth-database infos:\n\n```bash\npython create_data.py create_groundtruth_database --data_path=KITTI_DATASET_ROOT\n```\n\n#### 5. Modify config file\n\nThe config file needs to be edited to point to the above datasets:\n\n```bash\ntrain_input_reader: {\n  ...\n  database_sampler {\n    database_info_path: \"/path/to/kitti_dbinfos_train.pkl\"\n    ...\n  }\n  kitti_info_path: \"/path/to/kitti_infos_train.pkl\"\n  kitti_root_path: \"KITTI_DATASET_ROOT\"\n}\n...\neval_input_reader: {\n  ...\n  kitti_info_path: \"/path/to/kitti_infos_val.pkl\"\n  kitti_root_path: \"KITTI_DATASET_ROOT\"\n}\n```\n\n### Run the Code \nThe common can be found in pointpillars_with_TANet/second/*sh\n#### 1. Train TANet for Car \n\tbash run_car_16_tanet.sh\n#### 2. Train TANet for Ped\u0026Cyc\n\tbash run_ped_cycle_16_tanet.sh\nNote:  you need specify the absolute  path of Model_Path\n\n### Results on TANet\n**Trained model is available (Note: Please use the newest code!):**\n1. **Google Driver:** https://drive.google.com/drive/folders/1jSuQy5svfkDBw_4JszSzmIp9tpH1Ox7f\n2. **Baiduyunpan:** https://pan.baidu.com/s/1TfeYpkyRORPFLktS9REc2g   \n **Password:**  5fe1\n\n#### 1. Car\n```\nCar AP@0.70, 0.70, 0.70:\nbbox AP:90.81, 89.56, 87.91\nbev  AP:89.90, 86.94, 86.44\n3d   AP:88.17, 77.75, 75.31\naos  AP:90.75, 89.18, 87.20\nCar AP@0.70, 0.50, 0.50:\nbbox AP:90.81, 89.56, 87.91\nbev  AP:90.86, 90.25, 89.43\n3d   AP:90.85, 90.16, 89.19\naos  AP:90.75, 89.18, 87.20\n```\n\n#### 2.Ped\u0026Cyc (Only one model) \n```\nCyclist AP@0.50, 0.50, 0.50:\nbbox AP:86.94, 70.20, 66.22\nbev  AP:86.20, 67.70, 63.42\n3d   AP:85.21, 65.29, 61.57\naos  AP:86.75, 69.88, 65.89\nCyclist AP@0.50, 0.25, 0.25:\nbbox AP:86.94, 70.20, 66.22\nbev  AP:86.82, 71.54, 66.99\n3d   AP:86.72, 70.49, 65.43\naos  AP:86.75, 69.88, 65.89\nPedestrian AP@0.50, 0.50, 0.50:\nbbox AP:70.74, 67.94, 63.76\nbev  AP:78.55, 71.41, 66.03\n3d   AP:71.04, 64.20, 59.11\naos  AP:52.15, 50.10, 47.10\nPedestrian AP@0.50, 0.25, 0.25:\nbbox AP:70.74, 67.94, 63.76\nbev  AP:86.90, 82.47, 78.98\n3d   AP:86.89, 82.47, 78.96\naos  AP:52.15, 50.10, 47.10\n```\n\n\n## TANet for [Second.Pytorch](https://github.com/traveller59/second.pytorch) \n### Start Up\nRefer the README in  [second.pytorch](https://github.com/traveller59/second.pytorch) . \n\nNote:\n**you need to specify the \" Dataset: Kitti\" or \"Dataset: Nuscenes\" in \"tanet.yaml\"** \n\n### On KITTI\n1. Prepare dataset:\n```bash\npython create_data.py kitti_data_prep --root_path=/mnt/data2/Kitti_for_TANet_2/object\n```\n2. Training\n```bash\nCUDA_VISIBLE_DEVICES=5 python ./pytorch/train.py train --config_path=./configs/tanet/car/xyres_16.config --model_dir=/mnt/data2/TANet_2/second.pytorch/second/train_car_tanet_weight_2  --refine_weight=2\n```\n3. Testing\n```bash\nCUDA_VISIBLE_DEVICES=5 python ./pytorch/train.py evaluate --config_path=./configs/tanet/car/xyres_16.config --model_dir=/mnt/data2/TANet_2/second.pytorch/second/train_car_tanet_weight_2 --measure_time=True --batch_size=1\n```\n### On Nuscenes\nNuScenes dataset:\n```plain\n└── NUSCENES_TRAINVAL_DATASET_ROOT\n       ├── samples       \u003c-- key frames\n       ├── sweeps        \u003c-- frames without annotation\n       ├── maps          \u003c-- unused\n       └── v1.0-trainval \u003c-- metadata and annotations\n└── NUSCENES_TEST_DATASET_ROOT\n       ├── samples       \u003c-- key frames\n       ├── sweeps        \u003c-- frames without annotation\n       ├── maps          \u003c-- unused\n       └── v1.0-test     \u003c-- metadata\n```\n\n1. Prepare dataset:\n```bash\npython create_data.py nuscenes_data_prep --root_path=/mnt/data4/NuScenes/train --version=\"v1.0-trainval\" --max_sweeps=10 --dataset_name=\"NuScenesDataset\"\n\npython create_data.py nuscenes_data_prep --root_path=/mnt/data4/NuScenes/test --version=\"v1.0-test\" --max_sweeps=10 --dataset_name=\"NuScenesDataset\"\n```\n\n2. Training\n```bash\nCUDA_VISIBLE_DEVICES=4 python ./pytorch/train.py train --config_path=./configs/nuscenes/all.pp.lowa_only_add_ta_model.config --model_dir=/mnt/data2/TANet_2/second.pytorch/second/train_all_lowa_nuscenes_only_ta_module \n```\n\n3. Testing\n```bash\nCUDA_VISIBLE_DEVICES=4 python ./pytorch/train.py train --config_path=./configs/nuscenes/all.pp.lowa_only_add_ta_model.config --model_dir=/mnt/data2/TANet_2/second.pytorch/second/train_all_lowa_nuscenes_only_ta_module \n```\n\n\n## Citation\nIf you find our work useful in your research, please consider citing:\n``` \n         @article{liu2019tanet,\n         title={TANet: Robust 3D Object Detection from Point Clouds with Triple Attention},\n         author={Zhe Liu and Xin Zhao and Tengteng Huang and Ruolan Hu and Yu Zhou and Xiang Bai},\n         year={2020},\n         journal={AAAI},\n         url={https://arxiv.org/pdf/1912.05163.pdf},\n         eprint={1912.05163},\n         archivePrefix={arXiv},\n         primaryClass={cs.CV}\n         }\n```\n\n```\n         @article{lang2018pointpillars,\n         title={PointPillars: Fast Encoders for Object Detection from\n         Point Clouds},\n         author={Lang, Alex H and Vora, Sourabh and Caesar,\n         Holger and Zhou, Lubing and Yang, Jiong and Beijbom,\n         Oscar},\n         journal={CVPR},\n         year={2019}\n         }\n```\n\n```   \n           @article{yan2018second,\n           title={Second: Sparsely embedded convolutional detection},\n           author={Yan, Yan and Mao, Yuxing and Li, Bo},\n           journal={Sensors},\n           volume={18},\n           number={10},\n           pages={3337},\n           year={2018},\n           publisher={Multidisciplinary Digital Publishing Institute}\n         }\n```\n\n## Acknowledgement\n\nOur work builds on the  excellent works, which include:\n\n- [pointpillars](\u003chttps://github.com/nutonomy/second.pytorch\u003e) \n- [second.pytorch](\u003chttps://github.com/traveller59/second.pytorch\u003e) \n\n​         \n\n​        \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhappinesslz%2FTANet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhappinesslz%2FTANet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhappinesslz%2FTANet/lists"}