https://github.com/skp-1997/image-captioning-with-mlflow

This repository trains image captioning model using CNN and Transformers.
https://github.com/skp-1997/image-captioning-with-mlflow

cnn-keras dagshub generative-ai image-caption-generator jupyter-notebook mlflow-tracking nlp-machine-learning transformers

Last synced: about 2 months ago
JSON representation

This repository trains image captioning model using CNN and Transformers.

• Host: GitHub
• URL: https://github.com/skp-1997/image-captioning-with-mlflow
• Owner: skp-1997
• Created: 2024-08-20T01:00:13.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-08-23T06:45:09.000Z (6 months ago)
• Last Synced: 2024-11-09T00:57:40.730Z (3 months ago)
• Topics: cnn-keras, dagshub, generative-ai, image-caption-generator, jupyter-notebook, mlflow-tracking, nlp-machine-learning, transformers
• Language: Jupyter Notebook
• Homepage:
• Size: 1.54 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Image Captioning using Transformers

This project implements an image captioning model utilizing CNN and Transformer architectures. The model architecture is as follows:

## Architecture

![ImageCaptioningArchitecture drawio](https://github.com/user-attachments/assets/737588fe-1cb8-43b5-866a-065a5cff3069)

1. CNN as Feature Extractor:

    - We use EfficientNetB0 as the Convolutional Neural Network (CNN) to extract features from the images.

2. Transformer Encoder:

    - The extracted features from the CNN are passed to the encoder, which consists of multiple Transformer layers.

    - These layers process the features and prepare them for decoding.

3. Transformer Decoder:

    - The decoder, which also comprises Transformer layers, generates captions for the images based on the features provided by the encoder.

4. Data

    - Each input image is associated with five captions.

    - Data preprocessing and augmentation techniques are applied to improve the model's robustness and performance.

5. Model Training

    - Once the model is trained, the results, metrics, and the trained model itself are logged using MLFlow and DagsHub for efficient tracking and management of model development.

## Output Examples



  

  



## MLFlow Experiment Tracking