https://github.com/ck0123/improved-bertscore-for-image-captioning-evaluation

https://github.com/ck0123/improved-bertscore-for-image-captioning-evaluation

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/ck0123/improved-bertscore-for-image-captioning-evaluation
• Owner: ck0123
• Created: 2020-04-12T14:50:21.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2020-04-12T14:58:23.000Z (over 4 years ago)
• Last Synced: 2024-07-22T14:39:46.228Z (5 months ago)
• Language: Python
• Size: 1.98 MB
• Stars: 22
• Watchers: 1
• Forks: 3
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-diverse-captioning - [code

README

        
## Improved BERTScore for image captioning evaluation

Implementation of paper: Improving Image Captioning Evaluation by Considering Inter References Variance (ACL2020)

## Usage:

Recently, this repo provides two metrics ('with BERT' and 'simple')

* python3 run_metric.py    

    

* python3 run_metric_simple.py 

## example data:

example/example.json (you can modify this file for your own datasets)   

Fields explanation:  

* "refs": reference captions (each sample 5 references)    

* "cand": candidate caption (each sample 1 candidate)

* "refs_hid": contextual embeddings of reference captions

* "cand_hid": contextual embeddings of cand captions

* "mismatch": mismatches marks computed from all of reference captions

* "metric_result": scores on our metric  

  

NOTE:   

we also provide Flickr 8K Expert Annotation file with our format 'example/flickr.json'  

you can easily reproduce our result following run_metric.py lines 223-235.

## Dependencies:

pytorch-pretrained-bert==0.6.2 (old version of [huggingface/transformers](https://github.com/huggingface/transformers))     

torch==0.4.1  

bert_score==0.1.2 (already in this repo)