{"id":29005446,"url":"https://github.com/absarraashid3/captioncrafter","last_synced_at":"2025-06-25T11:33:14.211Z","repository":{"id":301031545,"uuid":"1007953523","full_name":"AbsarRaashid3/CaptionCrafter","owner":"AbsarRaashid3","description":"CaptionCrafter is a vision-language fine-tuning project that combines the power of BLIP-2 and the efficiency of LoRA (Low-Rank Adaptation) to generate intelligent, human-like captions for images. Using the Flickr8k dataset, this project showcases how to fine-tune large multimodal models for image captioning tasks in a memory-efficient way.","archived":false,"fork":false,"pushed_at":"2025-06-24T20:02:07.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-24T20:43:21.828Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsarRaashid3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-24T19:43:36.000Z","updated_at":"2025-06-24T20:02:10.000Z","dependencies_parsed_at":"2025-06-24T20:43:24.743Z","dependency_job_id":"65614852-c951-4e7b-acff-7c97d265c37a","html_url":"https://github.com/AbsarRaashid3/CaptionCrafter","commit_stats":null,"previous_names":["absarraashid3/captioncrafter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AbsarRaashid3/CaptionCrafter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsarRaashid3%2FCaptionCrafter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsarRaashid3%2FCaptionCrafter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsarRaashid3%2FCaptionCrafter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsarRaashid3%2FCaptionCrafter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsarRaashid3","download_url":"https://codeload.github.com/AbsarRaashid3/CaptionCrafter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsarRaashid3%2FCaptionCrafter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261863483,"owners_count":23221561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-25T11:31:18.542Z","updated_at":"2025-06-25T11:33:14.193Z","avatar_url":"https://github.com/AbsarRaashid3.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CaptionCrafter: Fine-Tuning BLIP-2 with LoRA on Flickr8k\nCaptionCrafter is a vision-language fine-tuning project that combines the power of BLIP-2 and the efficiency of LoRA (Low-Rank Adaptation) to generate intelligent, human-like captions for images. Using the Flickr8k dataset, this project showcases how to fine-tune large multimodal models for image captioning tasks in a memory-efficient way—making cutting-edge AI more accessible.\n\n# Project Overview\nThe goal of CaptionCrafter is to:\n\nFine-tune BLIP-2, a state-of-the-art vision-language model, for image captioning on the Flickr8k dataset.\n\nOptimize training using LoRA, which updates only small trainable adapters instead of the full model.\n\nEnable practical and efficient fine-tuning on limited hardware while achieving strong captioning performance.\n\n# Project Description\nThis project demonstrates:\n\nFine-tuning Salesforce/blip2-flan-t5-xl to generate captions for images from Flickr8k.\n\nLeveraging LoRA to reduce GPU memory consumption and training time.\n\nGenerating fluent, relevant captions for unseen images using learned visual-linguistic representations.\n\nIdeal for:\n\n Researchers working on custom vision-language tasks.\n\n Developers seeking to reduce compute costs using LoRA.\n\n Enthusiasts exploring image understanding via language generation.\n\n# Key Features\n# BLIP-2 for Image Captioning\nFine-tunes the pre-trained BLIP-2 model on the Flickr8k dataset.\n\nLearns to generate accurate, descriptive image captions from raw visual input.\n\n# LoRA Integration\nIncorporates LoRA for efficient training by freezing most of the model and updating only low-rank adapters.\n\nSignificantly reduces memory and computational cost without sacrificing performance.\n\n# Flickr8k Dataset\nConsists of 8,000 images, each with 5 human-written captions.\n\nSuitable for training and evaluating image captioning models due to its caption diversity and manageable size.\n\n# Evaluation Metrics\nThe performance of the fine-tuned model is evaluated using standard captioning metrics:\n\nBLEU: Measures n-gram precision.\n\nMETEOR: Balances precision, recall, and synonym matching.\n\nCIDEr: Consensus-based image description evaluation.\n\nROUGE: Measures overlap with reference captions.\n\n# Tech Stack\nComponent\tTool/Model\nVision-Language Model\tSalesforce/blip2-flan-t5-xl (HuggingFace)\nLoRA Adapter\tpeft (HuggingFace PEFT library)\nDataset\tFlickr8k (8,000 images, 5 captions/image)\nEvaluation\tBLEU, METEOR, CIDEr, ROUGE\nFrameworks\tPyTorch, HuggingFace Transformers, Datasets\n\n\n# Developed by\nAbsar Raashid\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsarraashid3%2Fcaptioncrafter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsarraashid3%2Fcaptioncrafter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsarraashid3%2Fcaptioncrafter/lists"}