https://github.com/phiphiphi31/SuperSBT
https://github.com/phiphiphi31/SuperSBT
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/phiphiphi31/SuperSBT
- Owner: phiphiphi31
- License: mit
- Created: 2024-07-10T03:31:37.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-08-12T04:31:32.000Z (10 months ago)
- Last Synced: 2024-08-12T05:38:29.221Z (10 months ago)
- Language: Python
- Size: 1.44 MB
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- Awesome-Visual-Object-Tracking - [code
README
# [Correlation-Embedded Transformer Tracking: A Single-Branch Framework (SuperSBT)](https://arxiv.org/pdf/2401.12743)
```bibtex
Fei Xie, Wankou Yang, Chunyu Wang, Lei Chu, Yue Cao, Chao Ma, Wenjun Zeng
```
:star: SuperSBT is accepted by T-PAMI in Aug. 2024!
:star: The official implementation for the [SuperSBT](https://arxiv.org/pdf/2401.12743).
:star: SuperSBT is the improved version of the [SBT](https://github.com/phiphiphi31/SBT) which is published in **CVPR 2022** [Correlation-Aware Deep Tracking
](https://openaccess.thecvf.com/content/CVPR2022/papers/Xie_Correlation-Aware_Deep_Tracking_CVPR_2022_paper.pdf).
[[Pretrained Weight](https://drive.google.com/drive/folders/1mRawJdhJ9jGkdH5SyJnIOlUWKJwq0h5J?usp=sharing)] [[Models](https://drive.google.com/drive/folders/10fpCZqRUMJBfnD9ZiIXNiFu5POGp8v3s?usp=sharing)] [[Raw Results](https://drive.google.com/drive/folders/1ok7FnAHYM4hXOzshIXaxiO8exbOJjJWf?usp=sharing)]
## Highlights
### Improved One-stream Tracking Framework
SuperSBT adopts a hierarchical architecture with a local modeling layer to enhance shallow-level features. A unified relation modeling is proposed to remove complex handcrafted layer pattern
designs. SuperSBT is further improved by masked image modeling pre-training, integrating temporal modeling, and equipping with
dedicated prediction heads.
### Better Speed-performance Trade-off
| Tracker | GOT-10K (AO) | LaSOT (AUC) | TrackingNet (AUC) | TNL2K(AUC) |
|:--------------:|:------------:|:-----------:|:-----------------:|:----------:|
| SuperSBT-Base | 74.4 | 70.0 | 84.0 | 56.6 |
| SuperSBT-Small | 71.6 | 67.5 | 82.7 | 55.7 |
| SuperSBT-Light | 69.4 | 65.8 | 81.4 | 53.6 |
## Install the environment
**Option1**: Use the Anaconda (CUDA 10.2)
```
conda create -n supersbt python=3.8
conda activate supersbt
bash install.sh
```
**Option2**: Use the Anaconda (CUDA 11.3)
```
conda env create -f supersbt_cuda113_env.yaml
```
**Option3**: Use the docker file
We provide the full docker file here.
## Set project paths
Run the following command to set paths for this project
```
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output
```
After running this command, you can also modify paths by editing these two files
```
lib/train/admin/local.py # paths about training
lib/test/evaluation/local.py # paths about testing
```
## Data Preparation
Put the tracking datasets in ./data. It should look like:
```
${PROJECT_ROOT}
-- data
-- lasot
|-- airplane
|-- basketball
|-- bear
...
-- got10k
|-- test
|-- train
|-- val
-- coco
|-- annotations
|-- images
-- trackingnet
|-- TRAIN_0
|-- TRAIN_1
...
|-- TRAIN_11
|-- TEST
```
## Training
Download pre-trained MAE weight and put it under `$PROJECT_ROOT$/pretrained_models`
```
python tracking/train.py --script supersbt --config supersbt_base --save_dir ./output --mode multiple --nproc_per_node 4 --use_wandb 0
python -m torch.distributed.launch --nproc_per_node 8 lib/train/run_training.py --script supersbt --config supersbt_base --save_dir .
```
Replace `--config` with the desired model config under `experiments/supersbt`. We use [wandb](https://github.com/wandb/client) to record detailed training logs, in case you don't want to use wandb, set `--use_wandb 0`.
## Evaluation
Download the model weights mentioned above
Put the downloaded weights on `$PROJECT_ROOT$/output/checkpoints/train/supersbt`
Change the corresponding values of `lib/test/evaluation/local.py` to the actual benchmark saving paths
Some testing examples:
- LaSOT or other off-line evaluated benchmarks (modify `--dataset` correspondingly)
```
python tracking/test.py supersbt supersbt_base --dataset lasot --threads 16 --num_gpus 4
python tracking/analysis_results.py # need to modify tracker configs and names
```
- GOT10K-test
```
python tracking/test.py supersbt supersbt_base_got --dataset got10k_test --threads 16 --num_gpus 4
python lib/test/utils/transform_got10k.py --tracker_name supersbt --cfg_name supersbt_base_got
```
- TrackingNet
```
python tracking/test.py supersbt supersbt_base --dataset trackingnet --threads 16 --num_gpus 4
python lib/test/utils/transform_trackingnet.py --tracker_name supersbt --cfg_name supersbt_base
```
## Test FLOPs, and Speed
*Note:* The speeds reported in our paper were tested on a single RTX2080Ti GPU.
```
# Profiling supersbt_base
python tracking/profile_model.py --script supersbt --config supersbt_base
# Profiling vitb_384_mae_ce_32x4_ep300
python tracking/profile_model.py --script supersbt --config supersbt_base
```
## Acknowledgments
* Thanks for the [STARK](https://github.com/researchmm/Stark), [OStrack](https://github.com/botaoye/OSTrack), and [PyTracking](https://github.com/visionml/pytracking) library, which helps us to quickly implement our ideas.
* We use the implementation of the [HiViT](https://github.com/zhangxiaosong18/hivit).
## Citation
If our work is useful for your research, please consider citing:
```Bibtex
@article{xie_2024_SuperSBT,
title={Correlation-Embedded Transformer Tracking: A Single-Branch Framework},
author={Xie, Fei and Yang, Wankou and Wang, Chunyu and Chu, Lei and Cao, Yue and Ma, Chao and Zeng, Wenjun},
journal={arXiv preprint arXiv:2401.12743},
year={2024}
}
@InProceedings{xie_2022_SBT,
author = {Xie, Fei and Wang, Chunyu and Wang, Guangting and Cao, Yue and Yang, Wankou and Zeng, Wenjun},
title = {Correlation-Aware Deep Tracking},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022},
}
```