Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/wangxiyang2022/DeepFusionMOT
Code for RA-L journal and IROS 2022 paper "DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association".
https://github.com/wangxiyang2022/DeepFusionMOT
Last synced: 15 days ago
JSON representation
Code for RA-L journal and IROS 2022 paper "DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association".
- Host: GitHub
- URL: https://github.com/wangxiyang2022/DeepFusionMOT
- Owner: wangxiyang2022
- License: mit
- Created: 2022-02-18T01:03:29.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2023-11-06T13:03:30.000Z (about 1 year ago)
- Last Synced: 2024-08-01T03:42:58.829Z (3 months ago)
- Language: Python
- Homepage:
- Size: 19.3 MB
- Stars: 190
- Watchers: 7
- Forks: 34
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# News
## This paper has been accepted as both a journal paper in Robotics and Automation Letters (RA-L), as well as being accepted as a conference paper at IROS 2022!!!
# DeepFusionMOT
This is the offical implementation of paper "[DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://arxiv.org/abs/2202.12100) "
## Update
* (2022/6/21)This paper has been accepted as a journal paper in Robotics and Automation Letters (RA-L).
* (2022/7/31)This paper has been accepted as a conference paper in IROS 2022.
* (2022/10/25)Online presentation on IROS2022.
* (2022/11/6)Another article"[StrongFusionMOT](https://ieeexplore.ieee.org/abstract/document/9976946/)" based on DeepFusionMOT improvement was accepted by IEEE Sensors Journal.
* (2023/4/28)The code has been refactored and different cost functions have been introduced, resulting in further improvement in performance.
## Abstract
In the recent literature, on the one hand, many 3D multi-object tracking (MOT) works have focused on tracking accuracy and neglected computation speed, commonly by designing rather complex cost functions and feature extractors. On the other hand, some methods have focused too much on computation speed at the expense of tracking accuracy. In view of these issues, this paper proposes a robust and fast camera-LiDAR fusion-based MOT method that achieves a good trade-off between accuracy and speed. Relying on the characteristics of camera and LiDAR sensors, an effective deep association mechanism is designed and embedded in the proposed MOT method. This association mechanism realizes tracking of an object in a 2D domain when the object is far away and only detected by the camera, and updating of the 2D trajectory with 3D information obtained when the object appears in the LiDAR field of view to achieve a smooth fusion of 2D and 3D trajectories. Extensive experiments based on the KITTI dataset indicate that our proposed method presents obvious advantages over the state-of-the-art MOT methods in terms of both tracking accuracy and processing speed.
## DeepFusionMOT
### Video examples on benchmarks dataset
## Dependencies
- Ubuntu 20.04
## Getting Started
#### *1. Clone the github repository.*
```
git clone https://github.com/wangxiyang2022/DeepFusionMOT
```
#### *2. Dataset preparation*
Please download the official KITTI [object tracking dataset](http://www.cvlibs.net/datasets/kitti/eval_tracking.php).
The final dataset organization should be like this:
```
DeepFusionMOT
├── data
│ ├── kitti
│ │ │── tracking
│ │ │ │──testing
| │ │ │ ├──calib
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....txt
| │ │ │ | └──0028.txt
| │ │ │ ├──image_02
| │ │ │ | ├──0000
| │ │ │ | ├──....
| │ │ │ | └──0028
| │ │ │ ├──oxts
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....
| │ │ │ | └──0028.txt
| │ │ │ ├──label_02
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....txt
| │ │ │ | └──0028.txt
| │ │ │ ├──velodyne
| │ │ │ | ├──0000
| │ │ │ | ├──....
| │ │ │ | └──0028
│ │ │ │──training # the structure is same as testing set
| │ │ │ ├──calib
| │ │ │ ├──image_02
| │ │ │ ├──pose
| │ │ │ ├──label_02
| │ │ │ └──velodyne
```
#### *3. Install dependency*
```
cd your_path/DeepFusionMOT
pip install -r requirements.txt
```
#### *4. 3D Object Detections & 2D Object Detections*
Thanks to these researchers for making their code public, in this repository, for convenience, we provide the 3D detections of PointRCNN and 2D detections of RRC on the KITTI MOT dataset for car. Of course you can also use the results of other detectors.
- **For 3D detections**
| Frame | Type | 2D BBOX (x1, y1, x2, y2) | Score | 3D BBOX (h, w, l, x, y, z, rot_y) | Alpha |
| :---: | :--: | :---------------------------------: | :----: | :-------------------------------------------------: | :-----: |
| 0 | 2 | 298.3125,165.1800,458.2292,293.4391 | 8.2981 | 1.9605,1.8137,4.7549,-4.5720,1.8435,13.5308,-2.1125 | -1.7867 |
- **For 2D detections**
| Frame | 2D BBOX (x1, y1, x2, y2) | Score |
| :---: | :-----------------------------------------: | :------: |
| 0 | 296.021000,160.173000,452.297000,288.372000 | 0.529230 |
The format definition can be found in the object development toolkit here: https://github.com/JonathonLuiten/TrackEval/blob/master/docs/KITTI-format.txt
#### *5. Run demo*
```
python main.py
```
#### *6. KITTI MOT Evaluation*
If you want to evaluate the tracking results using the evaluation tool on the KITTI website, you will need to go https://github.com/JonathonLuiten/TrackEval to download the evaluation code and follow the appropriate steps to set.
Using 3D detections of PointRCNN and 2D detections of RRC, the following results will be obtained.
| | HOAT( **↑)** | **DetA( **↑)**** | **AssA**(**↑)** | IDSW（↓） | MOTP(**↑)** | MOTA(**↑)** | FPS（↑） |
| :------------------: | :----------: | :--------------: | :-------------: | :-------: | :---------: | :---------: | :------: |
| **Training dataset** | 77.45% | 75.35% | 79.85% | 83 | 86.60% | 87.28% | 104 |
| **Testing dataset** | 75.46% | 71.54% | 80.05% | 84 | 85.02% | 84.63% | 110 |
### Acknowledgement
A portion code is borrowed from [AB3DMOT](https://github.com/xinshuoweng/AB3DMOT) and [Deepsort](https://github.com/nwojke/deep_sort), and the visualization code from [3D-Detection-Tracking-Viewer](https://github.com/hailanyi/3D-Detection-Tracking-Viewer). Many thanks to their wonderful work!
### Citation
If you find this work useful, please consider to cite our paper:
```
@ARTICLE{9810346,
author={Wang, Xiyang and Fu, Chunyun and Li, Zhankun and Lai, Ying and He, Jiawei},
journal={IEEE Robotics and Automation Letters},
title={DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association},
year={2022}, volume={7}, number={3}, pages={8260-8267}, doi={10.1109/LRA.2022.3187264}}
```