{"id":21039014,"url":"https://github.com/j-min/clip-caption-reward","last_synced_at":"2025-04-10T05:11:29.901Z","repository":{"id":38354045,"uuid":"485519339","full_name":"j-min/CLIP-Caption-Reward","owner":"j-min","description":"PyTorch code for \"Fine-grained Image Captioning with CLIP Reward\" (Findings of NAACL 2022)","archived":false,"fork":false,"pushed_at":"2022-08-06T20:52:17.000Z","size":2763,"stargazers_count":242,"open_issues_count":7,"forks_count":26,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-03T01:13:31.848Z","etag":null,"topics":["clip","image-captioning","reinforcement-learning","vision-and-language"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2205.13115","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j-min.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-25T20:09:05.000Z","updated_at":"2025-03-31T07:01:12.000Z","dependencies_parsed_at":"2022-07-12T02:17:19.297Z","dependency_job_id":null,"html_url":"https://github.com/j-min/CLIP-Caption-Reward","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-min%2FCLIP-Caption-Reward","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-min%2FCLIP-Caption-Reward/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-min%2FCLIP-Caption-Reward/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-min%2FCLIP-Caption-Reward/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j-min","download_url":"https://codeload.github.com/j-min/CLIP-Caption-Reward/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161276,"owners_count":21057555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","image-captioning","reinforcement-learning","vision-and-language"],"created_at":"2024-11-19T13:37:14.803Z","updated_at":"2025-04-10T05:11:29.879Z","avatar_url":"https://github.com/j-min.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fine-grained Image Captioning with CLIP Reward\n\n* Authors: [Jaemin Cho](https://j-min.io), [David Seunghyun Yoon](https://david-yoon.github.io/), [Ajinkya Kale](https://www.linkedin.com/in/kaleajinkya/), [Franck Dernoncourt](https://research.adobe.com/person/franck-dernoncourt), [Trung Bui](https://sites.google.com/site/trungbuistanford/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/)\n* [Findings of NAACL 2022 Paper](https://arxiv.org/abs/2205.13115)\n* [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/j-min/CLIP-Caption-Reward/blob/main/Inference_example.ipynb) (Inference using pretrained model on custom image)\n* Integrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/NAACL2022/CLIP-Caption-Reward)\n* Try Replicate web demo and docker image here [![Replicate](https://replicate.com/j-min/clip-caption-reward/badge)](https://replicate.com/j-min/clip-caption-reward)\n\u003cimg src=\"./assets/teaser.png\" alt=\"teaser image\" width=\"800\"/\u003e\n\n\n\n# Code structure\n```bash\n# Configurations\n./configs/\n    # MLE\n    phase1/\n    # RL\n    phase2/\n\n# COCO caption evaluation\n./cider\n./coco-caption\n\n# Preprocessing\n./clip # CLIP feature extractor\n./scripts # COCO preprocessing\n./scripts_FineCapEval # FineCapEval preprocessing\n./data # Storing preprocessed features\n\n# Core model / Rewards / Data loading\n./captioning\n\n# Training / Evaluation\n./tools\n\n# Fine-tuning CLIP Text encoder\n./retrieval\n\n# Pretrained checkpoints\n./save\n\n# Storing original dataset files\n./datasets\n```\n\n# Setup\n\n## Install Dependencies\n\n\n```bash\n# Create python environment (optional)\nconda create -n clip4caption python=3.7\nsource activate clip4caption\n\n# python dependenceies\npip install -r requirements.txt\n\n## Install this repo as package\npip install -e .\n\n# Install Detectron2 (optional for training utilities)\npip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html\n\n# Setup coco-caption (optional for text metric evaluation)\ngit clone https://github.com/clip-vil/cider\ngit clone https://github.com/clip-vil/coco-caption\n\ncd coco-caption\nbash get_stanford_models.sh\nbash get_google_word2vec_model.sh\n\n# Install java (optional for METEOR evaluation as part of text metrics)\nsudo apt install default-jre\n```\n\n\n\n## Download Pretrained models\nWe host model checkpoints via [google drive](https://drive.google.com/drive/folders/1-1BQbBlwwDzcqX1iMPn7UpjTeGYtAUfT).\nDownload checkpoints as below.\nThe `.ckpt` file size for captioning and CLIP models are 669.65M and 1.12G, respectively.\n```bash\n# Captioning model\n./save/\n    clipRN50_cider/\n        clipRN50_cider-last.ckpt\n    clipRN50_cider_clips/\n        clipRN50_cider_clips-last.ckpt\n    clipRN50_clips/\n        clipRN50_clips-last.ckpt\n    clipRN50_clips_grammar/\n        clipRN50_clips_grammar-last.ckpt\n    clipRN50_mle/\n        clipRN50_mle-last.ckpt\n\n# Finetuned CLIP Text encoder\n./retrieval/\n    save/\n        clip_negative_text-last.ckpt\n```\n\n\n\n# Dataset preparation\n\n\n```\n# Original dataset files - to be downloaded\n./datasets/\n    # Download from http://mscoco.org/dataset/#download\n    COCO/\n        images/\n            train2014/\n            val2014/\n        annotations/\n            captions_train2014.json\n            captions_val2014.json\n\n    # Download from https://drive.google.com/drive/folders/1jlwInAsVo-PdBdJlmHKPp34dLnxIIMLx\n    FineCapEval/\n        images/\n            XXX.jpg\n```\n\n## MS COCO\n\n* Download files\n```bash\n./datasets/\n    # Download from http://mscoco.org/dataset/#download\n     COCO/\n        images/\n            train2014/\n            val2014/\n        annotations/\n            captions_train2014.json\n            captions_val2014.json\n\n\n./data/\n    # Download from http://cs.stanford.edu/people/karpathy/deepimagesent/caption_datasets.zip\n    dataset_coco.json\n\n    # Download from from https://drive.google.com/drive/folders/1eCdz62FAVCGogOuNhy87Nmlo5_I0sH2J\n    coco-train-words.p\n```\n\n* Text processing\n```bash\npython scripts/prepro_labels.py --input_json data/dataset_coco.json --output_json data/cocotalk.json --output_h5 data/cocotalk\n```\n\n* Visual feature extraction\n```bash\npython scripts/clip_prepro_feats.py --input_json data/dataset_coco.json --output_dir data/cocotalk --images_root datasets/COCO/images --model_type RN50\n\n# optional (n_jobs)\n--n_jobs 4 --job_id 0\n```\n\n\n* Visual fetaure extraction for CLIP-S Reward\n```bash\npython scripts/clipscore_prepro_feats.py --input_json data/dataset_coco.json --output_dir data/cocotalk --images_root datasets/COCO/images\n\n# optional (n_jobs)\n--n_jobs 4 --job_id 0\n```\n\n## FineCapEval\n* Download files from https://drive.google.com/drive/folders/1jlwInAsVo-PdBdJlmHKPp34dLnxIIMLx?usp=sharing\n```bash\n./datasets/\n    FineCapEval/\n        images/\n            XXX.jpg\n\n./data/\n    FineCapEval.json\n    FineCapEval.csv\n```\n\n* Visual feature extraction\n```bash\npython scripts_FineCapEval/clip_prepro_feats.py --input_json data/FineCapEval.json --output_dir data/FineCapEval --images_root datasets/FineCapEval/images --model_type RN50\n\n# optional (n_jobs)\n--n_jobs 4 --job_id 0\n```\n\n# Training and Evaluation\n\n## 1) MLE training\n```bash\nexport MLE_ID='clipRN50_mle'\n\n# Training\npython tools/train_pl.py --cfg configs/phase1/$MLE_ID.yml --id $MLE_ID\n\n# Evaluation\nEVALUATE=1 python tools/train_pl.py --cfg configs/phase1/$MLE_ID.yml --id $MLE_ID\n\n# Text-to-Iage Retrieval with CLIP VIT-B/32\npython tools/eval_clip_retrieval.py --gen_caption_path \"./eval_results/$MLE_ID.json\"\n\n# Evaluation on FineCapEval\npython tools/finecapeval_inference.py --reward mle\npython tools/eval_finecapeval.py --generated_id2caption ./FineCapEval_results/clipRN50_mle.json\n```\n\n## 2) RL finetuning\n\n### Reward: CIDEr\n\n```bash\nexport REWARD='cider'\nexport MLE_ID='clipRN50_mle'\nexport RL_ID='clipRN50_'$REWARD\n\n# Copy MLE checkpoint as starting point of RL finetuning\nmkdir save/$RL_ID\ncp save/$MLE_ID/$MLE_ID-last.ckpt save/$RL_ID/$RL_ID-last.ckpt\n\n# Training\npython tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Evaluation\nEVALUATE=1 python tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Text-to-Iage Retrieval with CLIP VIT-B/32\npython tools/eval_clip_retrieval.py --gen_caption_path \"./eval_results/$RL_ID.json\"\n\n# Evaluation on FineCapEval\npython tools/finecapeval_inference.py --reward $REWARD\npython tools/eval_finecapeval.py --generated_id2caption ./FineCapEval_results/$RL_ID.json\n```\n\n### Reward: CLIP-S\n```bash\nexport REWARD='clips'\nexport MLE_ID='clipRN50_mle'\nexport RL_ID='clipRN50_'$REWARD\n\n# Copy MLE checkpoint as starting point of RL finetuning\nmkdir save/$RL_ID\ncp save/$MLE_ID/$MLE_ID-last.ckpt save/$RL_ID/$RL_ID-last.ckpt\n\n# Training\npython tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Evaluation\nEVALUATE=1 python tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Text-to-Iage Retrieval with CLIP VIT-B/32\npython tools/eval_clip_retrieval.py --gen_caption_path \"./eval_results/$RL_ID.json\"\n\n# Evaluation on FineCapEval\npython tools/finecapeval_inference.py --reward $REWARD\npython tools/eval_finecapeval.py --generated_id2caption ./FineCapEval_results/$RL_ID.json\n```\n\n### Reward: CLIP-S + CIDEr\n```bash\nexport REWARD='clips_cider'\nexport MLE_ID='clipRN50_mle'\nexport RL_ID='clipRN50_'$REWARD\n\n# Copy MLE checkpoint as starting point of RL finetuning\nmkdir save/$RL_ID\ncp save/$MLE_ID/$MLE_ID-last.ckpt save/$RL_ID/$RL_ID-last.ckpt\n\n# Training\npython tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Evaluation\nEVALUATE=1 python tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Text-to-Iage Retrieval with CLIP VIT-B/32\npython tools/eval_clip_retrieval.py --gen_caption_path \"./eval_results/$RL_ID.json\"\n\n# Evaluation on FineCapEval\npython tools/finecapeval_inference.py --reward $REWARD\npython tools/eval_finecapeval.py --generated_id2caption ./FineCapEval_results/$RL_ID.json\n```\n\n\n### Reward: CLIP-S + Grammar\n1) Run CLIP Finetuning (for grammar) following [./retrieval/README.md](./retrieval/README.md)\n\n2) Run RL training using the updated CLIP\n```bash\nexport REWARD='clips_grammar'\nexport MLE_ID='clipRN50_mle'\nexport RL_ID='clipRN50_'$REWARD\n\n# Copy MLE checkpoint as starting point of RL finetuning\nmkdir save/$RL_ID\ncp save/$MLE_ID/$MLE_ID-last.ckpt save/$RL_ID/$RL_ID-last.ckpt\n\n# Training\npython tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Evaluation\nEVALUATE=1 python tools/train_pl.py --cfg configs/phase2/$RL_ID.yml --id $RL_ID\n\n# Text-to-Iage Retrieval with CLIP VIT-B/32\npython tools/eval_clip_retrieval.py --gen_caption_path \"./eval_results/$RL_ID.json\"\n\n# Evaluation on FineCapEval\npython tools/finecapeval_inference.py --reward $REWARD\npython tools/eval_finecapeval.py --generated_id2caption ./FineCapEval_results/$RL_ID.json\n```\n\n\n# Acknowledgments\nWe thank the developers of [CLIP-ViL](https://github.com/clip-vil/CLIP-ViL/tree/master/CLIP-ViL-Direct/caption), [ImageCaptioning.pytorch](https://github.com/ruotianluo/ImageCaptioning.pytorch), [CLIP](https://github.com/openai/CLIP), [coco-caption](https://github.com/tylin/coco-caption), [cider](https://github.com/vrama91/cider) for their public code release.\n\n\n# Reference\nPlease cite our paper if you use our models in your works:\n\n\n```bibtex\n@inproceedings{Cho2022CLIPReward,\n  title     = {Fine-grained Image Captioning with CLIP Reward},\n  author    = {Jaemin Cho and Seunghyun Yoon and Ajinkya Kale and Franck Dernoncourt and Trung Bui and Mohit Bansal},\n  booktitle = {Findings of NAACL},\n  year      = {2022}\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-min%2Fclip-caption-reward","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj-min%2Fclip-caption-reward","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-min%2Fclip-caption-reward/lists"}