Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/vision4robotics/SAM-DA

This is the official code for the paper "SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation".
https://github.com/vision4robotics/SAM-DA

Last synced: about 1 month ago
JSON representation

This is the official code for the paper "SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation".

Host: GitHub
URL: https://github.com/vision4robotics/SAM-DA
Owner: vision4robotics
License: apache-2.0
Created: 2023-07-02T07:10:06.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2023-10-12T12:16:21.000Z (over 1 year ago)
Last Synced: 2024-04-29T02:35:16.594Z (9 months ago)
Language: Jupyter Notebook
Homepage:
Size: 25.4 MB
Stars: 46
Watchers: 3
Forks: 11
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE.txt

Awesome Lists containing this project

Awesome-Segment-Anything - [code

README

        # SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation

Changhong Fu*, Liangliang Yao, Haobo Zuo, Guangze Zheng, Jia Pan

* \* Corresponding author. 

**[Vision4robotics](https://vision4robotics.github.io/)**

## 📢 News

* SAM-DA is accepted by [IEEE ICARM](http://www.ieee-arm.org/).

* The paper “SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation” is awarded the Toshio Fukuda Best Paper Award in Mechatronics of ICARM 2024!

## 🏗️ Framework

![Framework](https://github.com/vision4robotics/SAM-DA/blob/main/assets/framework.png)

## 👀 Visualization of SAM-DA

![One-to-many generation](https://github.com/vision4robotics/SAM-DA/blob/main/assets/one-to-many_generation.png)

## 🛠️ Installation

This code has been tested on Ubuntu 18.04, Python 3.8.3, Pytorch 1.13.1, and CUDA 11.6. Please install related libraries before running this code:

Install Segment Anything:

```

bash install.sh

```

Install SAM-DA-Track:

```

pip install -r requirements.txt

```

## 😀 Getting started

### Test SAM-DA

* Download a **model checkpoint** below and put it in `./tracker/BAN/snapshot`.

  | Training data | Model | Source 1 | Source 2 | Source 3 |

  |  ----  | ----  |  ----  | ----  | ----  |

  |  SAM-NAT-B (base, default) | `sam-da-track-b` |  [Baidu](https://pan.baidu.com/s/1c_hlOxnyv-4bGyHzymlpRA?pwd=6prk)  | [Google](https://drive.google.com/file/d/1yiUTYQty52cAacmtGuqdgb53CnIe2l1W/view?usp=sharing)  | [Hugging face](https://huggingface.co/George-Zhuang/SAM-DA/resolve/main/sam-da-track-b.pth)  |

  |  SAM-NAT-S (small) | `sam-da-track-s` |  [Baidu](https://pan.baidu.com/s/1kUCZMXgRZs1HgD6gtx9hrQ?pwd=a48s)  | [Google](https://drive.google.com/file/d/1fxShaJ67XB1nMnE9ioQg7_LXYQBd6snI/view?usp=sharing)  | [Hugging face](https://huggingface.co/George-Zhuang/SAM-DA/resolve/main/sam-da-track-s.pth)   |

  |  SAM-NAT-T (tiny) | `sam-da-track-t` |  [Baidu](https://pan.baidu.com/s/11LrJwoz--AO3UzXavwa_GA?pwd=5qkj)  | [Google](https://drive.google.com/file/d/10Y9td4CJt4DqbcvCCLVUkCEx67MyilYC/view?usp=sharing)  | [Hugging face](https://huggingface.co/George-Zhuang/SAM-DA/resolve/main/sam-da-track-t.pth)   | 

  |  SAM-NAT-N (nano) | `sam-da-track-n` |  [Baidu](https://pan.baidu.com/s/1h1OROv17qINJmGU7zR4LTA?pwd=ujag)  | [Google](https://drive.google.com/file/d/1xR5i2XqHoDRoBEXH7O4ko5JZok0EPHTF/view?usp=sharing)  | [Hugging face](https://huggingface.co/George-Zhuang/SAM-DA/resolve/main/sam-da-track-n.pth)  | 

* Download [NUT-L](https://pan.baidu.com/s/1KWYp5UHflFuaPiWLFZvaKw?pwd=t4tr) dataset and put it in `./tracker/BAN/test_dataset`.

* Test and evalute on NUT-L with `default` settings. 

```bash

cd tracker/BAN

python tools/test.py 

python tools/eval.py

```

* (optional) Test with other checkpoints (e.g., `sam-da-track-s`):

```bash

cd tracker/BAN

python tools/test.py --snapshot sam-da-track-s

python tools/eval.py

```

### Train SAM-DA

* SAM-powered target domain training sample swelling on NAT2021-*train*.

  1. Download original nighttime dataset [NAT2021-*train*](https://vision4robotics.github.io/NAT2021/) and put it in `./tracker/BAN/train_dataset/sam_nat`. 

  2. Sam-powered target domain training sample swelling!

   ```

   bash swell.sh

   ```

    > ⚠️ warning: A huge passport is necessary for saving data.

    

    > Training jsons are here: [Baidu](https://pan.baidu.com/s/1pRwq_bf-Eu4CGID2u3Plog?pwd=cbob). 

* Prepare daytime dataset [VID] and [GOT-10K].

  1. Download [VID](https://image-net.org/challenges/LSVRC/2017/) and [GOT-10K](http://got-10k.aitestunion.com/downloads) and put them in `./tracker/BAN/train_dataset/vid` and `./tracker/BAN/train_dataset/got10k`, respectively.

  2. Crop data following the instruction for [VID](./tracker/BAN/train_dataset/vid/readme.md) and [GOT-10k](./tracker/BAN/train_dataset/got10k/readme.md).

* Train `sam-da-track-b` (default) and other models. 

  ```bash

  cd tracker/BAN

  python tools/train.py --model sam-da-track-b

  ```

##  🌈 Fewer data, better performance

**SAM-DA** aims to reach the few-better training for quick deployment of night-time tracking methods for UAVs.

* **SAM-DA** enriches the training samples and attributes (ambient intensity) of target domain.



* **SAM-DA** can achieve better performance on fewer raw images with quicker training.

    

  | Method | Training data | Images | Propotion | Training | AUC (NUT-L) |

  |  ----  | ----  |  :----:  | :----:  | :----:  |  :----:  | 

  |  Baseline  | NAT2021-*train*  | 276k | 100%  | 12h  | 0.377  | 

  |  **SAM-DA**  | SAM-NAT-N | 28k | 10%  | **2.4h**  | 0.411  | 

  |  **SAM-DA**  | SAM-NAT-T | 92k | 33%  | 4h  | 0.414  | 

  |  **SAM-DA**  | SAM-NAT-S | 138k | 50%  | 6h  | 0.419  | 

  |  **SAM-DA**  | SAM-NAT-B | 276k | 100%  | 12h  | **0.430**  | 

  For more details, please refer to the [paper](https://arxiv.org/abs/2307.01024).

  

> Training duration on a single A100 GPU.

# License

The model is licensed under the Apache License 2.0 license.

# Citations

Please consider citing the related paper(s) in your publications if it helps your research.

```

@Inproceedings{Yao2023SAMDA,

  title={{SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation}},

  author={Fu, Changhong and Yao, Liangliang and Zuo, Haobo and Zheng, Guangze and Pan, Jia},

  booktitle={Proceedings of the IEEE  International Conference on Advanced Robotics and Mechatronics (ICARM)},

  year={2024}

  pages={1-8}

}

@article{kirillov2023segment,

  title={{Segment Anything}},

  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C and Lo, Wan-Yen and others},

  journal={arXiv preprint arXiv:2304.02643},

  year={2023}

  pages={1-30}

}

@Inproceedings{Ye2022CVPR,

title={{Unsupervised Domain Adaptation for Nighttime Aerial Tracking}},

author={Ye, Junjie and Fu, Changhong and Zheng, Guangze and Paudel, Danda Pani and Chen, Guang},

booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},

year={2022},

pages={1-10}

}

```

# Acknowledgments

We sincerely thank the contribution of following repos: [SAM](https://github.com/facebookresearch/segment-anything), [SiamBAN](https://github.com/hqucv/siamban), and [UDAT](https://github.com/vision4robotics/UDAT).

# Contact

If you have any questions, please contact Liangliang Yao at [[email protected]](mailto:[email protected]) or Changhong Fu at [[email protected]](mailto:[email protected]).