{"id":18710485,"url":"https://github.com/vision-cair/visualgpt","last_synced_at":"2025-08-05T23:11:14.628Z","repository":{"id":43922269,"uuid":"339015279","full_name":"Vision-CAIR/VisualGPT","owner":"Vision-CAIR","description":"VisualGPT, CVPR 2022 Proceeding, GPT as a decoder for vision-language models","archived":false,"fork":false,"pushed_at":"2023-05-16T06:13:12.000Z","size":6476,"stargazers_count":336,"open_issues_count":5,"forks_count":54,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-07-29T06:55:00.820Z","etag":null,"topics":["data-efficient-image-caption","image-caption","visualgpt"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Vision-CAIR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-15T08:45:53.000Z","updated_at":"2025-07-20T21:00:48.000Z","dependencies_parsed_at":"2024-11-07T12:50:40.697Z","dependency_job_id":null,"html_url":"https://github.com/Vision-CAIR/VisualGPT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Vision-CAIR/VisualGPT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vision-CAIR%2FVisualGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vision-CAIR%2FVisualGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vision-CAIR%2FVisualGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vision-CAIR%2FVisualGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Vision-CAIR","download_url":"https://codeload.github.com/Vision-CAIR/VisualGPT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Vision-CAIR%2FVisualGPT/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268987471,"owners_count":24340667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-05T02:00:12.334Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-efficient-image-caption","image-caption","visualgpt"],"created_at":"2024-11-07T12:34:24.956Z","updated_at":"2025-08-05T23:11:14.570Z","avatar_url":"https://github.com/Vision-CAIR.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# VisualGPT\n\nOur Paper [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://arxiv.org/abs/2102.10407)\n\n## Main Architecture of Our VisualGPT\n![image](images/final_architecture.jpg)\n\n\n## Download the GPT-2 pretrained weights\n```\ncurl --output gpt2-pytorch_model.bin https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin\n```\n\n## Enviroment setup\nClone the repository and create the `visualgpt` conda environmnet\n\n\n```\nconda env create -f environment.yml\nconda activate visualgpt\n```\n\nThen download spacy data\n\n```\npython -m spacy download en\n```\n\n## Data preparation\nWe provide the COCO dataset for downloading. Please download the annotations file [annotations.zip](https://drive.google.com/file/d/1i8mqKFKhqvBr8kEp3DbIh9-9UNAfKGmE/view?usp=sharing) and extract it.\nand [coco_detections.hdf5](https://drive.google.com/open?id=1MV6dSnqViQfyvgyHrmAT_lLpFbkzp3mx), in which the data is stored in a `\u003ckey, value\u003e` where key is the image id and value is a tensor (N, 2048). N it the number of detections\n\n## code structure\n\n\n\n\ncreate the log folder ``mkdir logs`` and start the training\n\n## Train the model\n```\npython train_visualGPT.py --batch_size 50 --head 12 --tau 0.2 --features_path coco_detections.hdf5 --annotation_folder annotations --lr 1e-4 --gpt_model_type gpt --random_seed 42 --log_file logs/log --exp_name experiment_log --lr 1e-4 --decoder_layer 12 --optimizer_type adamw  --gradient_accumulation_steps 2 --train_percentage 0.001 --split_train_data\n```\n\n\n\n## Acknowledgement\nThis code used resources from [Meshed Memory Transformer](https://github.com/aimagelab/meshed-memory-transformer) and [Transformers](https://github.com/huggingface/transformers)\n\n\n\nPlease cite our paper from the following bibtex\n\n```\n@@InProceedings{Chen_2022_CVPR,\n    author    = {Chen, Jun and Guo, Han and Yi, Kai and Li, Boyang and Elhoseiny, Mohamed},\n    title     = {VisualGPT: Data-Efficient Adaptation of Pretrained Language Models for Image Captioning},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2022},\n    pages     = {18030-18040}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvision-cair%2Fvisualgpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvision-cair%2Fvisualgpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvision-cair%2Fvisualgpt/lists"}