https://github.com/xmu-xiaoma666/sdatr

Official Code for "Knowing what it is: Semantic-enhanced Dual Attention Transformer" (TMM2022)
https://github.com/xmu-xiaoma666/sdatr

pytorch

Last synced: about 1 month ago
JSON representation

Official Code for "Knowing what it is: Semantic-enhanced Dual Attention Transformer" (TMM2022)

• Host: GitHub
• URL: https://github.com/xmu-xiaoma666/sdatr
• Owner: xmu-xiaoma666
• License: mit
• Created: 2022-03-24T22:03:29.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2022-10-15T08:16:31.000Z (about 2 years ago)
• Last Synced: 2023-03-04T18:07:41.878Z (almost 2 years ago)
• Topics: pytorch
• Language: Python
• Homepage:
• Size: 1.06 MB
• Stars: 13
• Watchers: 1
• Forks: 2
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Knowing what it is: Semantic-enhanced Dual Attention Transformer

Official Code for ["Knowing what it is: Semantic-enhanced Dual Attention Transformer" (TMM2022)](https://ieeexplore.ieee.org/abstract/document/9749944)

![](images/SDATR.png)

## Environment setup

Please refer to [meshed-memory-transformer](https://github.com/aimagelab/meshed-memory-transformer)

## Data preparation

* **Annotation**. Download the annotation file [annotation.zip](https://drive.google.com/file/d/1i8mqKFKhqvBr8kEp3DbIh9-9UNAfKGmE/view?usp=sharing). Extarct and put it in the project root directory.

* **Feature**. You can download our ResNeXt-101 feature (hdf5 file) [here](https://pan.baidu.com/s/1xVZO7t8k4H_l3aEyuA-KXQ). Acess code: jcj6.

* **evaluation**. Download the evaluation tools [here](https://pan.baidu.com/s/1xVZO7t8k4H_l3aEyuA-KXQ). Acess code: jcj6. Extarct and put it in the project root directory.

## Training

```python

python train.py --exp_name SDATR --batch_size 50 --rl_batch_size 100 --workers 4 --head 8 --warmup 10000 --features_path /home/data/coco_grid_feats2.hdf5 --annotation /home/data/m2_annotations --logs_folder tensorboard_logs

```

## Evaluation

```python

python eval.py --batch_size 50 --exp_name SDATR --features_path /home/data/coco_grid_feats2.hdf5 --annotation /home/data/m2_annotations

```

Pretrained model is available [here](https://pan.baidu.com/s/1yg5GSI5SptuNl1O_WLPryw). Acess code: hdfw.

By evaluating the pretrained model, you will get

```bash

{'BLEU': [0.8130918073281019, 0.6613944708299153, 0.5173751961764315, 0.3973754802778509], 'METEOR': 0.29504133552506084, 'ROUGE': 0.5914517025819723, 'CIDEr': 1.3447844386129792, 'SPICE': 0.23130604798677026}

```

## Citation

```

@ARTICLE{

  ma2022knowing,  

  author={Ma, Yiwei and Ji, Jiayi and Sun, Xiaoshuai and Zhou, Yiyi and Wu, Yongjian and Huang, Feiyue and Ji, Rongrong},  

  journal={IEEE Transactions on Multimedia},   

  title={Knowing what it is: Semantic-enhanced Dual Attention Transformer},   

  year={2022},   

  doi={10.1109/TMM.2022.3164787}

}

```