{"id":34858423,"url":"https://github.com/anshler/clip-prefix_blind-guessing","last_synced_at":"2026-05-25T10:02:04.190Z","repository":{"id":213784458,"uuid":"734920520","full_name":"Anshler/CLIP-prefix_blind-guessing","owner":"Anshler","description":"CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing  👁️📜🖋️","archived":false,"fork":false,"pushed_at":"2024-08-02T15:42:08.000Z","size":470,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-27T05:39:58.720Z","etag":null,"topics":["clip","computer-vision","gpt-2","image-captioning","nlp","sentence-transformers","text-classification"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Anshler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-23T03:02:46.000Z","updated_at":"2024-08-02T15:42:11.000Z","dependencies_parsed_at":"2024-01-29T20:38:12.318Z","dependency_job_id":null,"html_url":"https://github.com/Anshler/CLIP-prefix_blind-guessing","commit_stats":{"total_commits":53,"total_committers":2,"mean_commits":26.5,"dds":"0.37735849056603776","last_synced_commit":"1f7b2db81f2f71f86bc93cb62f21f7aba29df3d8"},"previous_names":["anshler/clip-prefix_blind-guessing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Anshler/CLIP-prefix_blind-guessing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anshler%2FCLIP-prefix_blind-guessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anshler%2FCLIP-prefix_blind-guessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anshler%2FCLIP-prefix_blind-guessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anshler%2FCLIP-prefix_blind-guessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Anshler","download_url":"https://codeload.github.com/Anshler/CLIP-prefix_blind-guessing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anshler%2FCLIP-prefix_blind-guessing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33469418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-25T06:32:55.349Z","status":"ssl_error","status_checked_at":"2026-05-25T06:32:35.322Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","computer-vision","gpt-2","image-captioning","nlp","sentence-transformers","text-classification"],"created_at":"2025-12-25T20:35:24.499Z","updated_at":"2026-05-25T10:02:04.183Z","avatar_url":"https://github.com/Anshler.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing 👁️📜🖋️\n[\\[paper\\]](https://doi.org/10.1007/978-3-031-67357-3_14) [\\[model\\]](https://huggingface.co/Anshler/selective-blind-guessing) [\\[demo\\]](https://colab.research.google.com/drive/19zLqmFVkE5kxbHJbyY62ZxJqjg6EcMGX?usp=sharing)\n\nImage caption generation resides at the intersection of computer vision and natural language processing, with its primary goal being the creation of descriptive and coherent textual narratives that faithfully depict the content of an image. \n\nThis paper presents two models that leverage CLIP as the image encoder and fine-tune GPT-2 for caption generation on the Flickr30k and Flickr8k datasets. The first model utilizes a straightforward mapping network and outperforms the original architecture with a BLEU-1 score of ```0.700```, BLEU-4 score of ```0.257```, and ROUGE score of ```0.569``` on the Flickr8k dataset. The second model constitutes a new architecture exploring the boundaries of minimal visual information required for captioning. It incorporates CLIP's text encoder to produce input for the generator, while the image embedding serves solely as a validation mechanism. Despite its relatively lower performance, with a BLEU-1 score of ```0.546```, BLEU-4 score of ```0.108```, and ROUGE score of ```0.444``` on the Flickr8k dataset, this model demonstrates the decoder's ability to create captions based on keyword descriptions alone, without direct access to the context vector.\n\n## Dataset\n\nWe use the [Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k) and [Flickr30k](https://www.kaggle.com/datasets/eeshawn/flickr30k) dataset\n\n## Evaluation\n\nWe use 6 metrics: Bleu-1 to 4, Meteor and Rouge\n\n\u003c!-- GitHub Markdown with LaTeX table --\u003e\n*Table 1: Result comparison of models trained on Flickr8k*\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth style=\"text-align:center;\"\u003eModels\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-1\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-2\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-3\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-4\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eMETEOR\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eROUGE\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003e\u003ca href=\"https://github.com/rmokady/CLIP_prefix_caption\"\u003eCLIP-prefix\u003c/a\u003e (Original)\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.698\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.508\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.363\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.259\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.257\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.565\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: CLIP-prefix -- Gradient clipping\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.700\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.513\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.372\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.262\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.257\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.569\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: CLIP-prefix -- Custom tokenizer\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.682\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.497\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.351\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.246\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.249\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.557\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- One caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.499\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.276\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.153\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.087\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.167\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.410\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- Top-2 caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.520\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.293\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.161\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.089\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.179\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.420\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- Top-5 caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.546\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.319\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.186\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.108\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.192\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.444\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003e\u003ca href=\"https://github.com/mtanti/rnn-role\"\u003eMerge-RNN\u003c/a\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.601\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.411\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.272\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.179\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.191\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.439\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- GitHub Markdown with converted LaTeX table --\u003e\n*Table 2: Result comparison of models trained on Flickr30k*\n\u003ctable\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth style=\"text-align:center;\"\u003eModels\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-1\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-2\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-3\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eBLEU-4\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eMETEOR\u003c/th\u003e\n      \u003cth style=\"text-align:center;\"\u003eROUGE\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eCLIP-prefix (Original)\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.715\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.506\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.351\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.243\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.232\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.529\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: CLIP-prefix -- Gradient clipping\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.715\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.503\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.349\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.235\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.233\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.528\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: CLIP-prefix -- Custom tokenizer\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.733\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.525\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.366\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.254\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.232\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.536\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- One caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.495\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.261\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.139\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.076\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.154\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.378\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- Top-2 caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.510\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.279\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.150\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.082\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.164\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.391\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eOurs: SBG -- Top-5 caption\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.543\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.304\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.170\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.095\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.175\u003c/strong\u003e\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e\u003cstrong\u003e0.411\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd style=\"text-align:left;\"\u003eMerge-RNN\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.596\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.404\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.270\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.181\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.175\u003c/td\u003e\n      \u003ctd style=\"text-align:center;\"\u003e0.416\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n## Inference\n\nRun our demo on Colab:\n* [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1l7yHUMrhL_6JF_2_VQcvjHltLCEKy5ZH?usp=sharing) CLIP-prefix\n* [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19zLqmFVkE5kxbHJbyY62ZxJqjg6EcMGX?usp=sharing) SBG\n\nWe also create a _[Stable diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) extension_ to interact with our models (_Clip-prefix gradient8k \u0026 SBG 8k_) locally. Load from this [repo](https://github.com/Anshler/ICG_sd_extension)\n## Models\nOur model weights are published on Huggingface:\n* \u003ca\u003e \u003cimg src=\"https://workable-application-form.s3.amazonaws.com/advanced/production/61557f91d9510741dc62e7f8/c3635b59-a3d2-444a-b636-a9d0061dcdde\" style=\"height: 1em;\"\u003e\u003c/a\u003e [CLIP-prefix](https://huggingface.co/Anshler/clip-prefix)\n* \u003ca\u003e \u003cimg src=\"https://workable-application-form.s3.amazonaws.com/advanced/production/61557f91d9510741dc62e7f8/c3635b59-a3d2-444a-b636-a9d0061dcdde\" style=\"height: 1em;\"\u003e\u003c/a\u003e [SBG](https://huggingface.co/Anshler/selective-blind-guessing) _(flickr8k)_\n\nCLIP model used is ViT-L-14\n\n## Contact\nTeam members: [Triet Minh Huynh](https://www.facebook.com/anshler), [Duy Linh Nguyen](https://www.facebook.com/ngnd.linh), [Thanh Tri Nguyen](https://www.facebook.com/thantri222)\n## Citation\n\n```bibtex\n@InProceedings{10.1007/978-3-031-67357-3_14,\nauthor=\"Huynh, Triet Minh\nand Nguyen, Duy Linh\nand Nguyen, Thanh Tri\nand Vu, Thuy-Duong Thi\nand Dang-Ngoc, Hanh\nand Dang, Duc Ngoc Minh\",\neditor=\"Vo, Nguyen-Son\nand Ha, Dac-Binh\nand Jung, Haejoon\",\ntitle=\"CLIP-Prefix for Image Captioning and an Experiment on Blind Image Guessing\",\nbooktitle=\"Industrial Networks and Intelligent Systems\",\nyear=\"2024\",\npublisher=\"Springer Nature Switzerland\",\naddress=\"Cham\",\npages=\"189--203\",\nabstract=\"Image caption generation resides at the intersection of computer vision and natural language processing, with its primary goal being the creation of descriptive and coherent textual narratives that faithfully depict the content of an image. This paper presents two models that leverage CLIP as the image encoder and fine-tune GPT-2 for caption generation on the Flickr30k and Flickr8k datasets. The first model utilizes a straightforward mapping network and outperforms the original architecture with a BLEU-1 score of 0.700, BLEU-4 score of 0.257, and ROUGE score of 0.569 on the Flickr8k dataset. The second model constitutes a new architecture exploring the boundaries of minimal visual information required for captioning. It incorporates CLIP's text encoder to produce input for the generator, while the image embedding serves solely as a validation mechanism. Despite its relatively lower performance, with a BLEU-1 score of 0.546, BLEU-4 score of 0.108, and ROUGE score of 0.444 on the Flickr8k dataset, this model demonstrates the decoder's ability to create captions based on keyword descriptions alone, without direct access to the context vector.\",\nisbn=\"978-3-031-67357-3\"\n}\n```\n\n## Acknowledgments\n\n_This project was inspired by_ [CLIP_prefix_caption](https://github.com/rmokady/CLIP_prefix_caption)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshler%2Fclip-prefix_blind-guessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanshler%2Fclip-prefix_blind-guessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanshler%2Fclip-prefix_blind-guessing/lists"}