{"id":23317548,"url":"https://github.com/git-khandelwal/cnn-to-gpt2","last_synced_at":"2025-04-07T04:30:00.798Z","repository":{"id":258780296,"uuid":"863493360","full_name":"git-khandelwal/CNN-to-GPT2","owner":"git-khandelwal","description":"Image Captioning using CNNs and Transformers","archived":false,"fork":false,"pushed_at":"2024-11-14T11:49:32.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-13T08:52:48.618Z","etag":null,"topics":["captioning","cnn","image","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/git-khandelwal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-26T11:48:08.000Z","updated_at":"2024-11-14T11:49:35.000Z","dependencies_parsed_at":"2024-12-20T16:33:12.234Z","dependency_job_id":"b62609f0-22cf-45cd-a100-0361336c3212","html_url":"https://github.com/git-khandelwal/CNN-to-GPT2","commit_stats":null,"previous_names":["git-khandelwal/cnn-to-gpt2"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git-khandelwal%2FCNN-to-GPT2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git-khandelwal%2FCNN-to-GPT2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git-khandelwal%2FCNN-to-GPT2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/git-khandelwal%2FCNN-to-GPT2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/git-khandelwal","download_url":"https://codeload.github.com/git-khandelwal/CNN-to-GPT2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247593631,"owners_count":20963751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["captioning","cnn","image","transformer"],"created_at":"2024-12-20T16:33:08.366Z","updated_at":"2025-04-07T04:30:00.779Z","avatar_url":"https://github.com/git-khandelwal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Image Captioning Using CNNs and Transformers\n\nThis project implements an image captioning model that generates descriptive captions for input images. It uses a ResNet-50 model for image feature extraction and GPT-2 for generating captions based on these features. The model is trained on the Flickr dataset and deployed to process images by uploading them via a web interface.\n\nProject Overview\nThe image captioning system consists of two main components:\n\nImage Encoder: ResNet-50 (pretrained on ImageNet) is used to extract feature vectors from images.\nCaption Generator: GPT-2 (pretrained) generates natural language captions based on the image features extracted by ResNet-50.\n\nKey Features\nPretrained Models: Both ResNet-50 and GPT-2 are pretrained on large datasets to leverage transfer learning.\nImage-to-Text Translation: The model takes an image as input and outputs a descriptive caption.\nWeb Interface: Users can upload an image through a simple frontend, and the backend generates and displays captions. \n\nTraining:\nTrain the model on the Flickr dataset for desired number of epochs(Took around 4 hrs of training on NVIDIA 4070 Ti 12G for 5 epochs with lr=1e-4)\n\nRunning Web Application:\nOnce the training is completed, check whether the caption is being generated or not by running the main.py file. Run the app.py file and go to localhost server to get the web interface for image captioning. \n\nTry it yourself:\n1. git clone https://github.com/git-khandelwal/CNN-to-GPT2\n2. pip install -r requirements.txt\n3. Download the pt weights: https://drive.google.com/file/d/1uANPY6WZusGcFPumj-jZgl3UP0IhLW9-/view?usp=sharing\n4. flask run --host=0.0.0.0 --port=5000\n5. Go to the IP address and upload images(.jpg/.png formats supported) to generate captions\n\nWeb Interface:\n![image](https://github.com/user-attachments/assets/aea7b0fb-2f93-46ca-8ec8-d78c357d9eac)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit-khandelwal%2Fcnn-to-gpt2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgit-khandelwal%2Fcnn-to-gpt2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgit-khandelwal%2Fcnn-to-gpt2/lists"}