{"id":15783633,"url":"https://github.com/deepmancer/vit-gpt2-image-captioning","last_synced_at":"2025-04-01T16:31:06.987Z","repository":{"id":163661661,"uuid":"639121117","full_name":"deepmancer/vit-gpt2-image-captioning","owner":"deepmancer","description":"Fine-tuning an encoder-decoder transformer (ViT-Base-Patch16-224-In21k and DistilGPT2) for image captioning on the COCO dataset","archived":false,"fork":false,"pushed_at":"2024-08-16T12:29:44.000Z","size":8819,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-05T20:01:18.908Z","etag":null,"topics":["bert","coco-dataset","distilbert","encoder-decoder","gpt-2","image-captioning","imagenet","pre-trained-language-models","pytorch","torch","transformer","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepmancer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-10T20:02:32.000Z","updated_at":"2024-08-24T23:13:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"16e29888-3959-4c2d-b4ed-f37f27d4f09b","html_url":"https://github.com/deepmancer/vit-gpt2-image-captioning","commit_stats":null,"previous_names":["alirezaheidari-cs/image-captioning","deepmancer/image-captioning","deepmancer/vit-gpt2-image-captioning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fvit-gpt2-image-captioning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fvit-gpt2-image-captioning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fvit-gpt2-image-captioning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepmancer%2Fvit-gpt2-image-captioning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepmancer","download_url":"https://codeload.github.com/deepmancer/vit-gpt2-image-captioning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246620683,"owners_count":20806819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","coco-dataset","distilbert","encoder-decoder","gpt-2","image-captioning","imagenet","pre-trained-language-models","pytorch","torch","transformer","vision-transformer"],"created_at":"2024-10-04T20:00:23.201Z","updated_at":"2025-04-01T16:31:05.515Z","avatar_url":"https://github.com/deepmancer.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🖼️ Image Captioning with Fine-Tuned ViT and GPT-2\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?style=for-the-badge\u0026logo=PyTorch\u0026logoColor=white\" alt=\"PyTorch\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Hugging%20Face-FFD21E?style=for-the-badge\u0026logo=huggingface\u0026logoColor=000\" alt=\"Hugging Face Transformers\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/scikit--learn-%23F7931E.svg?style=for-the-badge\u0026logo=scikit-learn\u0026logoColor=white\" alt=\"scikit-learn\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/COCO%20Dataset-009688.svg?style=for-the-badge\u0026logo=OpenCV\u0026logoColor=white\" alt=\"COCO Dataset\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3670A0?style=for-the-badge\u0026logo=python\u0026logoColor=ffdd54\" alt=\"Python\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Jupyter-F37626.svg?\u0026style=for-the-badge\u0026logo=Jupyter\u0026logoColor=white\" alt=\"Jupyter Notebook\"\u003e\n\u003c/p\u003e\n\nWelcome to the **Image Captioning** project! This repository implements an advanced image captioning module that leverages state-of-the-art models, including the **ViT-Base-Patch16-224-In21k** (Vision Transformer) as the encoder and **DistilGPT-2** as the decoder. This project aims to generate descriptive captions for images from the COCO dataset, utilizing the powerful capabilities of the Transformers library.\n\n---\n\n## 📝 Project Description\n\nThis project focuses on creating an image captioning system by integrating the following key components:\n\n- **Encoder**: The project uses the Google **ViT-Base-Patch16-224-In21k** pretrained model to encode image features. ViT (Vision Transformer) is known for its superior performance in image classification and feature extraction tasks.\n- **Decoder**: The **DistilGPT-2** model, a distilled version of GPT-2, is employed to decode the image features into natural language captions. GPT-2 excels at generating coherent and contextually relevant text.\n\n### 🎯 Objective\n\nThe primary goal is to fine-tune these models on the COCO dataset for the image captioning task. The resulting captions are evaluated using popular NLP metrics like **ROUGE**, **BLEU**, and **BERTScore** to measure their quality and relevance.\n\n---\n\n## 📚 Dataset\n\nThe project utilizes the **COCO dataset** (Common Objects in Context), which is a rich dataset consisting of:\n\n- **118,000** training images\n- **5,000** validation images\n- Each image is paired with **5 corresponding captions**, providing diverse descriptions of the visual content.\n\nThis dataset is well-suited for training and evaluating image captioning models due to its variety and scale.\n\n---\n\n## ⚙️ Implementation Details\n\n### Frameworks \u0026 Libraries\n\n- **PyTorch**: The deep learning framework used for model implementation and training.\n- **Transformers**: Hugging Face's library is employed to access and fine-tune the ViT and GPT-2 models.\n\n### Model Architecture\n\n- **Vision Transformer (ViT)**: Acts as the encoder, transforming images into feature-rich embeddings.\n- **DistilGPT-2**: Serves as the decoder, generating textual descriptions based on the encoded image features.\n\n### Training Process\n\n- **Fine-Tuning**: Both models are fine-tuned on the COCO dataset over **2 epochs**. This process adapts the pretrained models to the specific task of image captioning, optimizing their performance on this task.\n\n---\n\n## 🧪 Evaluation Metrics\n\nThe generated captions are evaluated using the following metrics:\n\n- **ROUGE**: Measures the overlap between the predicted and reference captions.\n- **BLEU**: Evaluates the precision of n-grams in the generated captions compared to reference captions.\n- **BERTScore**: Uses BERT embeddings to assess the semantic similarity between generated\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepmancer%2Fvit-gpt2-image-captioning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepmancer%2Fvit-gpt2-image-captioning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepmancer%2Fvit-gpt2-image-captioning/lists"}