https://github.com/peri044/stt

A multi-task model which does image captioning, sentence paraphrasing and cross-modal retrieval.
https://github.com/peri044/stt

common-vector-space cross-modal-retrieval deep-learning image-captioning sentence-paraphrasing sequence-to-sequence

Last synced: 22 days ago
JSON representation

A multi-task model which does image captioning, sentence paraphrasing and cross-modal retrieval.

• Host: GitHub
• URL: https://github.com/peri044/stt
• Owner: peri044
• Created: 2018-12-25T02:39:39.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2019-11-21T07:16:48.000Z (almost 5 years ago)
• Last Synced: 2024-10-04T13:35:09.457Z (about 1 month ago)
• Topics: common-vector-space, cross-modal-retrieval, deep-learning, image-captioning, sentence-paraphrasing, sequence-to-sequence
• Language: Python
• Size: 103 KB
• Stars: 18
• Watchers: 4
• Forks: 5
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Show, Translate and Tell

This repo contains code for training and evaluation of a multi-task model which performs image captioning, cross modal retrieval and sentence paraphrasing.

The paper and results can be found at  Show, Translate and Tell. This work has been accepted at  ICIP 2019.

The proposed architecture is as shown in the figure

![Alt text](figures/stt.PNG?raw=true)

## Generate Data

In the data folder, you can find scripts for generating TF-records for mscoco dataset.

Update (11/20/2019): `prepare_mscoco_pairs.py` is updated in the repo. It uses the `captions_train2014.json` annotations of MSCOCO to build paraphrases. This script should be used to generate `train_enc.txt` and `train_dec.txt` which are basically paraphrases.

Using 5 captions, it creates 20 permutations of paraphrases and writes them in the TF record (using coco_data_loader.py) along with the associated image. This script is not cleaned up and should only be used for reference (it might not be the final script that we used).

Checkout command line arguments in the scripts for setting paths

* To generate TF-records for MSCOCO

```

python -m data.coco_data_loader --num 10000

```

Args:

* `--num` : Number of images to be written in TF record. Do not specify this unless you want to generate a subset of entire dataset.

* Generate TF-records with Image, predicted caption and groundtruth caption

```

python -m data.coco_data_loader --precompute \

                                --record_path para_att_pred.tfrecord \

                                --feature_path coco_precomp/testall_ims.npy \

                                --captions_path 

```

* `feature_path`: This should be used if you have already extracted features for all the images. 

In this case, a sample in TF record would look like (Feature (1x2048 dim vector for an image), caption A, caption B) which are paraphrases.

## Training on COCO dataset

```

sh scripts/train_coco.sh

```

## Evaluation on COCO dataset

```

sh scripts/eval_coco.sh

```

If you find this research or codebase useful in your experiments, consider citing

```

@article{stt2019,

  title={Show, Translate and Tell},

  author={Peri, Dheeraj and Sah, Shagan and Ptucha, Raymond},

  journal={arXiv preprint arXiv:1903.06275},

  year={2019}

}

```