{"id":13564168,"url":"https://github.com/jmisilo/clip-gpt-captioning","last_synced_at":"2025-04-09T18:16:09.292Z","repository":{"id":175620503,"uuid":"541259634","full_name":"jmisilo/clip-gpt-captioning","owner":"jmisilo","description":"CLIPxGPT Captioner is Image Captioning Model based on OpenAI's CLIP and GPT-2.","archived":false,"fork":false,"pushed_at":"2025-02-17T10:56:02.000Z","size":894,"stargazers_count":117,"open_issues_count":5,"forks_count":32,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T18:16:02.586Z","etag":null,"topics":["computer-vision","cv","deep-learning","image-caption","image-caption-generator","image-captioning","machine-learning","nlp","python","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmisilo.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-25T17:51:31.000Z","updated_at":"2025-04-07T01:08:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ef15185-5bd3-4b23-ab69-72b7490e62ad","html_url":"https://github.com/jmisilo/clip-gpt-captioning","commit_stats":null,"previous_names":["jmisilo/clip-gpt-captioning"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmisilo%2Fclip-gpt-captioning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmisilo%2Fclip-gpt-captioning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmisilo%2Fclip-gpt-captioning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmisilo%2Fclip-gpt-captioning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmisilo","download_url":"https://codeload.github.com/jmisilo/clip-gpt-captioning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085322,"owners_count":21045139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","cv","deep-learning","image-caption","image-caption-generator","image-captioning","machine-learning","nlp","python","pytorch"],"created_at":"2024-08-01T13:01:27.467Z","updated_at":"2025-04-09T18:16:09.264Z","avatar_url":"https://github.com/jmisilo.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# CLIPxGPT Captioner\n\n## Description\n\n**`CLIPxGPT Captioner`** is Image Captioning Model based on [OpenAI's](https://openai.com/) [CLIP](https://openai.com/blog/clip/) and [GPT-2](https://openai.com/blog/better-language-models/). The Model uses a Mapping module to \"translate\" CLIP embeddings ​​to GPT-2. The model is trained on the [Flickr30k](https://shannon.cs.illinois.edu/DenotationGraph/) dataset, downloaded from [Kaggle](https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset)\n\n**The goal** of the project was to find out about the possibility of CLIP + GPT-2 connection and to check whether, with a relatively short training time and a small dataset, the model will be able to recognize situations in the pictures. The model achieved satisfactory results.\n\nThe Model uses prefixes as in the [ClipCap](https://arxiv.org/abs/2111.09734) paper. In my original idea, the length of the prefix was 1, but after reading publication, the length of the prefix was changed to 4, thanks to which the performance increased.\n\nThe Model was trained with a frozen CLIP, a fully trained Mapping Module (5-6x Transformer Encoder Layers) and with partially frozen GPT-2 (the first and last 14 layers were trained).\n\nThe training process was carried out using the [Kaggle](https://www.kaggle.com/) P100 GPU.\n\n### Model Versions\n\n\u003e **Small** - [Download](https://drive.google.com/file/d/1pSQruQyg8KJq6VmzhMLFbT_VaHJMdlWF/view?usp=sharing)\n\u003e\n\u003e - Text Model - GPT-2 Small - 124M parameters\n\u003e - Mapping Module - 6x Transformer Encoder Layers\n\u003e - CLIP Base - Patch 32 model\n\u003e - 256M Parameters\n\n\u003e **Large** - [Download](https://drive.google.com/file/d/1Gh32arzhW06C1ZJyzcJSSfdJDi3RgWoG/view?usp=sharing)\n\u003e\n\u003e - Text Model - GPT-2 Medium - 355M parameters\n\u003e - Mapping Module - 5x Transformer Encoder Layers\n\u003e - CLIP Large - Patch 14 model\n\u003e - 736M Parameters\n\n## Example results\n\n![Example1](./examples/23012796-RS.jpg)\n![Example2](./examples/36979-RS.jpg)\n![Example3](./examples/89407459-RL.jpg)\n\n## Usage\n\nClone repository using:\n\n```bash\ngit clone https://github.com/jmisilo/clip-gpt-captioning\n\ncd clip-gpt-captioning\n```\n\nCreate environment and install requirements:\n\n```bash\npython -m venv venv\n# for windows\n.\\venv\\Scripts\\activate\n# for linux/mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\nAnd run prediction:\n\n```bash\npython .\\src\\predict.py -I \u003cimage_path\u003e -S \u003cmodel_size [S/L]\u003e -C \u003ccheckpoint_name\u003e\n```\n\n### References:\n\n- [OpenAI](https://openai.com/)\n- [CLIP](https://openai.com/blog/clip/)\n- [GPT-2](https://openai.com/blog/better-language-Models/)\n- [ClipCap; Mokady, Ron and Hertz, Amir and Bermano, Amit H](https://arxiv.org/abs/2111.09734)\n- [HuggingFace](https://huggingface.co/)\n\nIn cooperation with [knm](https://knmstudio.com/?utm_source=github\u0026utm_medium=clip-gpt-captioner)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmisilo%2Fclip-gpt-captioning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmisilo%2Fclip-gpt-captioning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmisilo%2Fclip-gpt-captioning/lists"}