{"id":13443395,"url":"https://github.com/mit-han-lab/bevfusion","last_synced_at":"2025-09-27T10:31:46.465Z","repository":{"id":37034298,"uuid":"497420077","full_name":"mit-han-lab/bevfusion","owner":"mit-han-lab","description":"[ICRA'23] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation","archived":true,"fork":false,"pushed_at":"2024-07-31T04:04:52.000Z","size":9016,"stargazers_count":2355,"open_issues_count":0,"forks_count":426,"subscribers_count":41,"default_branch":"main","last_synced_at":"2024-11-23T01:32:33.355Z","etag":null,"topics":["3d-perception","camera","lidar","object-detection","pytorch","semantic-segmentation","sensor-fusion"],"latest_commit_sha":null,"homepage":"https://bevfusion.mit.edu","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-han-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-28T20:20:53.000Z","updated_at":"2024-11-22T13:15:36.000Z","dependencies_parsed_at":"2024-07-11T22:07:48.172Z","dependency_job_id":"5d32ff2a-6df5-4b73-a79f-5403ddcebde7","html_url":"https://github.com/mit-han-lab/bevfusion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fbevfusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fbevfusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fbevfusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-han-lab%2Fbevfusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-han-lab","download_url":"https://codeload.github.com/mit-han-lab/bevfusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234426714,"owners_count":18830958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-perception","camera","lidar","object-detection","pytorch","semantic-segmentation","sensor-fusion"],"created_at":"2024-07-31T03:02:00.101Z","updated_at":"2025-09-27T10:31:40.535Z","avatar_url":"https://github.com/mit-han-lab.png","language":"Python","funding_links":[],"categories":["Python","Object Detection Applications","四、BEV Fusion","💻 Open-Source Projects"],"sub_categories":["1. List of BEV fusion methods","Papers"],"readme":"# BEVFusion\n\n### [website](http://bevfusion.mit.edu/) | [paper](https://arxiv.org/abs/2205.13542) | [video](https://www.youtube.com/watch?v=uCAka90si9E)\n\n![demo](assets/demo.gif)\n\n## News\n\n- **(2024/5)** BEVFusion is integrated into NVIDIA [DeepStream](https://developer.nvidia.com/blog/nvidia-deepstream-7-0-milestone-release-for-next-gen-vision-ai-development/) for sensor fusion.\n- **(2023/5)** NVIDIA provides a [TensorRT deployment solution](https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution/tree/master/CUDA-BEVFusion) of BEVFusion, achieving 25 FPS on Jetson Orin.\n- **(2023/4)** BEVFusion ranks first on [Argoverse](https://eval.ai/web/challenges/challenge-page/1710/overview) 3D object detection leaderboard among all solutions.\n- **(2023/1)** BEVFusion is integrated into [MMDetection3D](https://github.com/open-mmlab/mmdetection3d/tree/main/projects/BEVFusion).\n- **(2023/1)** BEVFusion is accepted to ICRA 2023!\n- **(2022/8)** BEVFusion ranks first on [Waymo](https://waymo.com/open/challenges/2020/3d-detection/) 3D object detection leaderboard among all solutions.\n- **(2022/6)** BEVFusion ranks first on [nuScenes](https://nuscenes.org/tracking?externalData=all\u0026mapData=all\u0026modalities=Any) 3D object detection leaderboard among all solutions.\n- **(2022/6)** BEVFusion ranks first on [nuScenes](https://nuscenes.org/object-detection?externalData=all\u0026mapData=all\u0026modalities=Any) 3D object detection leaderboard among all solutions.\n\n## Abstract\n\nMulti-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we break this deeply-rooted convention with BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than **40x**. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on the nuScenes benchmark, achieving **1.3%** higher mAP and NDS on 3D object detection and **13.6%** higher mIoU on BEV map segmentation, with **1.9x** lower computation cost.\n\n## Results\n\n### 3D Object Detection (on Waymo test)\n\n|   Model   | mAP-L1 | mAPH-L1  | mAP-L2  | mAPH-L2  |\n| :-------: | :------: | :--: | :--: | :--: |\n| [BEVFusion](https://waymo.com/open/challenges/entry/?challenge=DETECTION_3D\u0026challengeId=DETECTION_3D\u0026emailId=f58eed96-8bb3\u0026timestamp=1658347965704580) |    82.72   |  81.35  | 77.65  |  76.33 |\n| [BEVFusion-TTA](https://waymo.com/open/challenges/entry/?challenge=DETECTION_3D\u0026challengeId=DETECTION_3D\u0026emailId=94ddc185-d2ce\u0026timestamp=1663562767759105) | 86.04    |  84.76 | 81.22  |  79.97 |\n\nHere, BEVFusion only uses a single model without any test time augmentation. BEVFusion-TTA uses single model with test-time augmentation and no model ensembling is applied. \n\n### 3D Object Detection (on nuScenes test)\n\n|   Model   | Modality | mAP  | NDS  |\n| :-------: | :------: | :--: | :--: |\n| BEVFusion-e |   C+L    | 74.99 | 76.09 |\n| BEVFusion |   C+L    | 70.23 | 72.88 |\n| BEVFusion-base* |   C+L    | 71.72 | 73.83 |\n\n*: We scaled up MACs of the model to match the computation cost of concurrent work.\n\n### 3D Object Detection (on nuScenes validation)\n\n|        Model         | Modality | mAP  | NDS  | Checkpoint  |\n| :------------------: | :------: | :--: | :--: | :---------: |\n|    [BEVFusion](configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml)       |   C+L    | 68.52 | 71.38 | [Link](https://www.dropbox.com/scl/fi/ulaz9z4wdwtypjhx7xdi3/bevfusion-det.pth?rlkey=ovusfi2rchjub5oafogou255v\u0026dl=1) |\n| [Camera-Only Baseline](configs/nuscenes/det/centerhead/lssfpn/camera/256x704/swint/default.yaml) |    C     | 35.56 | 41.21 | [Link](https://www.dropbox.com/scl/fi/pxfaz1nc07qa2twlatzkz/camera-only-det.pth?rlkey=f5do81fawie0ssbg9uhrm6p30\u0026dl=1) |\n| [LiDAR-Only Baseline](configs/nuscenes/det/transfusion/secfpn/lidar/voxelnet_0p075.yaml)  |    L     | 64.68 | 69.28 | [Link](https://www.dropbox.com/scl/fi/b1zvgrg9ucmv0wtx6pari/lidar-only-det.pth?rlkey=fw73bmdh57jxtudw6osloywah\u0026dl=1) |\n\n*Note*: The camera-only object detection baseline is a variant of BEVDet-Tiny with a much heavier view transformer and other differences in hyperparameters. Thanks to our [efficient BEV pooling](mmdet3d/ops/bev_pool) operator, this model runs fast and has higher mAP than BEVDet-Tiny under the same input resolution. Please refer to [BEVDet repo](https://github.com/HuangJunjie2017/BEVDet) for the original BEVDet-Tiny implementation. The LiDAR-only baseline is TransFusion-L.\n\n### BEV Map Segmentation (on nuScenes validation)\n\n|        Model         | Modality | mIoU | Checkpoint  |\n| :------------------: | :------: | :--: | :---------: |\n| [BEVFusion](configs/nuscenes/seg/fusion-bev256d2-lss.yaml)       |   C+L    | 62.95 | [Link](https://www.dropbox.com/scl/fi/8lgd1hkod2a15mwry0fvd/bevfusion-seg.pth?rlkey=2tmgw7mcrlwy9qoqeui63tay9\u0026dl=1) |\n| [Camera-Only Baseline](configs/nuscenes/seg/camera-bev256d2.yaml) |    C     | 57.09 | [Link](https://www.dropbox.com/scl/fi/cwpcu80n0shmwraegi6z4/camera-only-seg.pth?rlkey=l60kdaz19fq3gwocsjk09e60z\u0026dl=1) |\n| [LiDAR-Only Baseline](configs/nuscenes/seg/lidar-centerpoint-bev128.yaml)  |    L     | 48.56 | [Link](https://www.dropbox.com/scl/fi/mi3w6uxvytdre9i42r9k7/lidar-only-seg.pth?rlkey=rve7hx80u3en1gfoi7tjucl72\u0026dl=1) |\n\n## Usage\n\n### Prerequisites\n\nThe code is built with following libraries:\n\n- Python \u003e= 3.8, \\\u003c3.9\n- OpenMPI = 4.0.4 and mpi4py = 3.0.3 (Needed for torchpack)\n- Pillow = 8.4.0 (see [here](https://github.com/mit-han-lab/bevfusion/issues/63))\n- [PyTorch](https://github.com/pytorch/pytorch) \u003e= 1.9, \\\u003c= 1.10.2\n- [tqdm](https://github.com/tqdm/tqdm)\n- [torchpack](https://github.com/mit-han-lab/torchpack)\n- [mmcv](https://github.com/open-mmlab/mmcv) = 1.4.0\n- [mmdetection](http://github.com/open-mmlab/mmdetection) = 2.20.0\n- [nuscenes-dev-kit](https://github.com/nutonomy/nuscenes-devkit)\n\nAfter installing these dependencies, please run this command to install the codebase:\n\n```bash\npython setup.py develop\n```\n\nWe also provide a [Dockerfile](docker/Dockerfile) to ease environment setup. To get started with docker, please make sure that `nvidia-docker` is installed on your machine. After that, please execute the following command to build the docker image:\n\n```bash\ncd docker \u0026\u0026 docker build . -t bevfusion\n```\n\nWe can then run the docker with the following command:\n\n```bash\nnvidia-docker run -it -v `pwd`/../data:/dataset --shm-size 16g bevfusion /bin/bash\n```\n\nWe recommend the users to run data preparation (instructions are available in the next section) outside the docker if possible. Note that the dataset directory should be an absolute path. Within the docker, please run the following command to clone our repo and install custom CUDA extensions:\n\n```bash\ncd home \u0026\u0026 git clone https://github.com/mit-han-lab/bevfusion \u0026\u0026 cd bevfusion\npython setup.py develop\n```\n\nYou can then create a symbolic link `data` to the `/dataset` directory in the docker.\n\n### Data Preparation\n\n#### nuScenes\n\nPlease follow the instructions from [here](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/datasets/nuscenes_det.md) to download and preprocess the nuScenes dataset. Please remember to download both detection dataset and the map extension (for BEV map segmentation). After data preparation, you will be able to see the following directory structure (as is indicated in mmdetection3d):\n\n```\nmmdetection3d\n├── mmdet3d\n├── tools\n├── configs\n├── data\n│   ├── nuscenes\n│   │   ├── maps\n│   │   ├── samples\n│   │   ├── sweeps\n│   │   ├── v1.0-test\n|   |   ├── v1.0-trainval\n│   │   ├── nuscenes_database\n│   │   ├── nuscenes_infos_train.pkl\n│   │   ├── nuscenes_infos_val.pkl\n│   │   ├── nuscenes_infos_test.pkl\n│   │   ├── nuscenes_dbinfos_train.pkl\n\n```\n\n### Evaluation\n\nWe also provide instructions for evaluating our pretrained models. Please download the checkpoints using the following script: \n\n```bash\n./tools/download_pretrained.sh\n```\n\nThen, you will be able to run:\n\n```bash\ntorchpack dist-run -np [number of gpus] python tools/test.py [config file path] pretrained/[checkpoint name].pth --eval [evaluation type]\n```\n\nFor example, if you want to evaluate the detection variant of BEVFusion, you can try:\n\n```bash\ntorchpack dist-run -np 8 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox\n```\n\nWhile for the segmentation variant of BEVFusion, this command will be helpful:\n\n```bash\ntorchpack dist-run -np 8 python tools/test.py configs/nuscenes/seg/fusion-bev256d2-lss.yaml pretrained/bevfusion-seg.pth --eval map\n```\n\n### Training\n\nWe provide instructions to reproduce our results on nuScenes.\n\nFor example, if you want to train the camera-only variant for object detection, please run:\n\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/centerhead/lssfpn/camera/256x704/swint/default.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth\n```\n\nFor camera-only BEV segmentation model, please run:\n\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/camera-bev256d2.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth\n```\n\nFor LiDAR-only detector, please run:\n\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/transfusion/secfpn/lidar/voxelnet_0p075.yaml\n```\n\nFor LiDAR-only BEV segmentation model, please run:\n\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/lidar-centerpoint-bev128.yaml\n```\n\nFor BEVFusion detection model, please run:\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth --load_from pretrained/lidar-only-det.pth \n```\n\nFor BEVFusion segmentation model, please run:\n```bash\ntorchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/fusion-bev256d2-lss.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth\n```\n\nNote: please run `tools/test.py` separately after training to get the final evaluation metrics.\n\n## Deployment on TensorRT\n[CUDA-BEVFusion](https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution/tree/master/CUDA-BEVFusion): Best practice for TensorRT, which provides INT8 acceleration solutions and achieves 25fps on ORIN.\n\n## FAQs\n\nQ: Can we directly use the info files prepared by mmdetection3d?\n\nA: We recommend re-generating the info files using this codebase since we forked mmdetection3d before their [coordinate system refactoring](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/changelog.md).\n\n## Acknowledgements\n\nBEVFusion is based on [mmdetection3d](https://github.com/open-mmlab/mmdetection3d). It is also greatly inspired by the following outstanding contributions to the open-source community: [LSS](https://github.com/nv-tlabs/lift-splat-shoot), [BEVDet](https://github.com/HuangJunjie2017/BEVDet), [TransFusion](https://github.com/XuyangBai/TransFusion), [CenterPoint](https://github.com/tianweiy/CenterPoint), [MVP](https://github.com/tianweiy/MVP), [FUTR3D](https://arxiv.org/abs/2203.10642), [CVT](https://github.com/bradyz/cross_view_transformers) and [DETR3D](https://github.com/WangYueFt/detr3d). \n\nPlease also check out related papers in the camera-only 3D perception community such as [BEVDet4D](https://arxiv.org/abs/2203.17054), [BEVerse](https://arxiv.org/abs/2205.09743), [BEVFormer](https://arxiv.org/abs/2203.17270), [M2BEV](https://arxiv.org/abs/2204.05088), [PETR](https://arxiv.org/abs/2203.05625) and [PETRv2](https://arxiv.org/abs/2206.01256), which might be interesting future extensions to BEVFusion.\n\n\n## Citation\n\nIf BEVFusion is useful or relevant to your research, please kindly recognize our contributions by citing our paper:\n\n```bibtex\n@inproceedings{liu2022bevfusion,\n  title={BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation},\n  author={Liu, Zhijian and Tang, Haotian and Amini, Alexander and Yang, Xingyu and Mao, Huizi and Rus, Daniela and Han, Song},\n  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fbevfusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-han-lab%2Fbevfusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-han-lab%2Fbevfusion/lists"}