Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/anshler/clip-prefix_blind-guessing

CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing 👁️📜🖋️
https://github.com/anshler/clip-prefix_blind-guessing

clip computer-vision gpt-2 image-captioning nlp sentence-transformers text-classification

Last synced: about 1 month ago
JSON representation

CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing 👁️📜🖋️

Awesome Lists containing this project

README 

          
# CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing 👁️📜🖋️

[\[paper\]](https://doi.org/10.1007/978-3-031-67357-3_14) [\[model\]](https://huggingface.co/Anshler/selective-blind-guessing) [\[demo\]](https://colab.research.google.com/drive/19zLqmFVkE5kxbHJbyY62ZxJqjg6EcMGX?usp=sharing)



Image caption generation resides at the intersection of computer vision and natural language processing, with its primary goal being the creation of descriptive and coherent textual narratives that faithfully depict the content of an image. 



This paper presents two models that leverage CLIP as the image encoder and fine-tune GPT-2 for caption generation on the Flickr30k and Flickr8k datasets. The first model utilizes a straightforward mapping network and outperforms the original architecture with a BLEU-1 score of ```0.700```, BLEU-4 score of ```0.257```, and ROUGE score of ```0.569``` on the Flickr8k dataset. The second model constitutes a new architecture exploring the boundaries of minimal visual information required for captioning. It incorporates CLIP's text encoder to produce input for the generator, while the image embedding serves solely as a validation mechanism. Despite its relatively lower performance, with a BLEU-1 score of ```0.546```, BLEU-4 score of ```0.108```, and ROUGE score of ```0.444``` on the Flickr8k dataset, this model demonstrates the decoder's ability to create captions based on keyword descriptions alone, without direct access to the context vector.



## Dataset



We use the [Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k) and [Flickr30k](https://www.kaggle.com/datasets/eeshawn/flickr30k) dataset



## Evaluation



We use 6 metrics: Bleu-1 to 4, Meteor and Rouge



*Table 1: Result comparison of models trained on Flickr8k*



  

    

      Models

      BLEU-1

      BLEU-2

      BLEU-3

      BLEU-4

      METEOR

      ROUGE

    

  

  

    

      CLIP-prefix (Original)

      0.698

      0.508

      0.363

      0.259

      0.257

      0.565

    

    

      Ours: CLIP-prefix -- Gradient clipping

      0.700

      0.513

      0.372

      0.262

      0.257

      0.569

    

    

      Ours: CLIP-prefix -- Custom tokenizer

      0.682

      0.497

      0.351

      0.246

      0.249

      0.557

    

    

      Ours: SBG -- One caption

      0.499

      0.276

      0.153

      0.087

      0.167

      0.410

    

    

      Ours: SBG -- Top-2 caption

      0.520

      0.293

      0.161

      0.089

      0.179

      0.420

    

    

      Ours: SBG -- Top-5 caption

      0.546

      0.319

      0.186

      0.108

      0.192

      0.444

    

    

      Merge-RNN

      0.601

      0.411

      0.272

      0.179

      0.191

      0.439

    

  



*Table 2: Result comparison of models trained on Flickr30k*



  

    

      Models

      BLEU-1

      BLEU-2

      BLEU-3

      BLEU-4

      METEOR

      ROUGE

    

  

  

    

      CLIP-prefix (Original)

      0.715

      0.506

      0.351

      0.243

      0.232

      0.529

    

    

      Ours: CLIP-prefix -- Gradient clipping

      0.715

      0.503

      0.349

      0.235

      0.233

      0.528

    

    

      Ours: CLIP-prefix -- Custom tokenizer

      0.733

      0.525

      0.366

      0.254

      0.232

      0.536

    

    

      Ours: SBG -- One caption

      0.495

      0.261

      0.139

      0.076

      0.154

      0.378

    

    

      Ours: SBG -- Top-2 caption

      0.510

      0.279

      0.150

      0.082

      0.164

      0.391

    

    

      Ours: SBG -- Top-5 caption

      0.543

      0.304

      0.170

      0.095

      0.175

      0.411

    

    

      Merge-RNN

      0.596

      0.404

      0.270

      0.181

      0.175

      0.416

    

  



## Inference



Run our demo on Colab:

* [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1l7yHUMrhL_6JF_2_VQcvjHltLCEKy5ZH?usp=sharing) CLIP-prefix

* [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19zLqmFVkE5kxbHJbyY62ZxJqjg6EcMGX?usp=sharing) SBG



We also create a _[Stable diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) extension_ to interact with our models (_Clip-prefix gradient8k & SBG 8k_) locally. Load from this [repo](https://github.com/Anshler/ICG_sd_extension)

## Models

Our model weights are published on Huggingface:

*   [CLIP-prefix](https://huggingface.co/Anshler/clip-prefix)

*   [SBG](https://huggingface.co/Anshler/selective-blind-guessing) _(flickr8k)_



CLIP model used is ViT-L-14



## Contact

Team members: [Triet Minh Huynh](https://www.facebook.com/anshler), [Duy Linh Nguyen](https://www.facebook.com/ngnd.linh), [Thanh Tri Nguyen](https://www.facebook.com/thantri222)

## Citation



```bibtex

@InProceedings{10.1007/978-3-031-67357-3_14,

author="Huynh, Triet Minh

and Nguyen, Duy Linh

and Nguyen, Thanh Tri

and Vu, Thuy-Duong Thi

and Dang-Ngoc, Hanh

and Dang, Duc Ngoc Minh",

editor="Vo, Nguyen-Son

and Ha, Dac-Binh

and Jung, Haejoon",

title="CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing",

booktitle="Industrial Networks and Intelligent Systems",

year="2024",

publisher="Springer Nature Switzerland",

address="Cham",

pages="189--203",

abstract="Image caption generation resides at the intersection of computer vision and natural language processing, with its primary goal being the creation of descriptive and coherent textual narratives that faithfully depict the content of an image. This paper presents two models that leverage CLIP as the image encoder and fine-tune GPT-2 for caption generation on the Flickr30k and Flickr8k datasets. The first model utilizes a straightforward mapping network and outperforms the original architecture with a BLEU-1 score of 0.700, BLEU-4 score of 0.257, and ROUGE score of 0.569 on the Flickr8k dataset. The second model constitutes a new architecture exploring the boundaries of minimal visual information required for captioning. It incorporates CLIP's text encoder to produce input for the generator, while the image embedding serves solely as a validation mechanism. Despite its relatively lower performance, with a BLEU-1 score of 0.546, BLEU-4 score of 0.108, and ROUGE score of 0.444 on the Flickr8k dataset, this model demonstrates the decoder's ability to create captions based on keyword descriptions alone, without direct access to the context vector.",

isbn="978-3-031-67357-3"

}

```



## Acknowledgments



_This project was inspired by_ [CLIP_prefix_caption](https://github.com/rmokady/CLIP_prefix_caption)