{"id":13737985,"url":"https://github.com/ruotianluo/self-critical.pytorch","last_synced_at":"2025-05-16T09:03:59.849Z","repository":{"id":45270001,"uuid":"90339318","full_name":"ruotianluo/self-critical.pytorch","owner":"ruotianluo","description":"Unofficial pytorch implementation for Self-critical Sequence Training for Image Captioning. and others.","archived":false,"fork":false,"pushed_at":"2023-10-05T04:01:53.000Z","size":614,"stargazers_count":977,"open_issues_count":86,"forks_count":286,"subscribers_count":21,"default_branch":"master","last_synced_at":"2024-02-14T13:32:24.441Z","etag":null,"topics":["image-captioning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruotianluo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-05-05T05:15:04.000Z","updated_at":"2024-02-13T17:44:43.000Z","dependencies_parsed_at":"2024-01-13T09:36:45.407Z","dependency_job_id":"9c5dea13-9413-41a1-ad38-f1f4a3ca2b74","html_url":"https://github.com/ruotianluo/self-critical.pytorch","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruotianluo%2Fself-critical.pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruotianluo%2Fself-critical.pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruotianluo%2Fself-critical.pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruotianluo%2Fself-critical.pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruotianluo","download_url":"https://codeload.github.com/ruotianluo/self-critical.pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254501556,"owners_count":22081528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-captioning"],"created_at":"2024-08-03T03:02:08.206Z","updated_at":"2025-05-16T09:03:59.828Z","avatar_url":"https://github.com/ruotianluo.png","language":"Python","funding_links":[],"categories":["Python","Paper implementations｜论文实现","Model Deployment library","Paper implementations","Popular Implementations"],"sub_categories":["Other libraries｜其他库:","PyTorch \u003ca name=\"pytorch\"/\u003e","Other libraries:","PyTorch"],"readme":"# An Image Captioning codebase\n\nThis is a codebase for image captioning research.\n\nIt supports:\n- Self critical training from [Self-critical Sequence Training for Image Captioning](https://arxiv.org/abs/1612.00563)\n- Bottom up feature from [ref](https://arxiv.org/abs/1707.07998).\n- Test time ensemble\n- Multi-GPU training. (DistributedDataParallel is now supported with the help of pytorch-lightning, see [ADVANCED.md](ADVANCED.md) for details)\n- Transformer captioning model.\n\nA simple demo colab notebook is available [here](https://colab.research.google.com/github/ruotianluo/ImageCaptioning.pytorch/blob/colab/notebooks/captioning_demo.ipynb)\n\n## Requirements\n- Python 3\n- PyTorch 1.3+ (along with torchvision) (Test with 1.13)\n- cider (already been added as a submodule)\n- coco-caption (already been added as a submodule) (**Remember to follow initialization steps in coco-caption/README.md**)\n- yacs\n- lmdbdict\n- Optional: pytorch-lightning (Tested with 2.0)\n\n## Install\n\nIf you have difficulty running the training scripts in `tools`. You can try installing this repo as a python package:\n```\npython -m pip install -e .\n```\n\n## Pretrained models\n\nCheckout [MODEL_ZOO.md](MODEL_ZOO.md).\n\nIf you want to do evaluation only, you can then follow [this section](#generate-image-captions) after downloading the pretrained models (and also the pretrained resnet101 or precomputed bottomup features, see [data/README.md](data/README.md)).\n\n## Train your own network on COCO/Flickr30k\n\n### Prepare data.\n\nWe now support both flickr30k and COCO. See details in [data/README.md](data/README.md). (Note: the later sections assume COCO dataset; it should be trivial to use flickr30k.)\n\n### Start training\n\n```bash\n$ python tools/train.py --id fc --caption_model newfc --input_json data/cocotalk.json --input_fc_dir data/cocotalk_fc --input_att_dir data/cocotalk_att --input_label_h5 data/cocotalk_label.h5 --batch_size 10 --learning_rate 5e-4 --learning_rate_decay_start 0 --scheduled_sampling_start 0 --checkpoint_path log_fc --save_checkpoint_every 6000 --val_images_use 5000 --max_epochs 30\n```\n\nor \n\n```bash\n$ python tools/train.py --cfg configs/fc.yml --id fc\n```\n\nThe train script will dump checkpoints into the folder specified by `--checkpoint_path` (default = `log_$id/`). By default only save the best-performing checkpoint on validation and the latest checkpoint to save disk space. You can also set `--save_history_ckpt` to 1 to save every checkpoint.\n\nTo resume training, you can specify `--start_from` option to be the path saving `infos.pkl` and `model.pth` (usually you could just set `--start_from` and `--checkpoint_path` to be the same).\n\nTo checkout the training curve or validation curve, you can use tensorboard. The loss histories are automatically dumped into `--checkpoint_path`.\n\nThe current command use scheduled sampling, you can also set `--scheduled_sampling_start` to -1 to turn off scheduled sampling.\n\nIf you'd like to evaluate BLEU/METEOR/CIDEr scores during training in addition to validation cross entropy loss, use `--language_eval 1` option, but don't forget to pull the submodule `coco-caption`.\n\nFor all the arguments, you can specify them in a yaml file and use `--cfg` to use the configurations in that yaml file. The configurations in command line will overwrite cfg file if there are conflicts.  \n\nFor more options, see `opts.py`. \n\n\u003c!-- **A few notes on training.** To give you an idea, with the default settings one epoch of MS COCO images is about 11000 iterations. After 1 epoch of training results in validation loss ~2.5 and CIDEr score of ~0.68. By iteration 60,000 CIDEr climbs up to about ~0.84 (validation loss at about 2.4 (under scheduled sampling)). --\u003e\n\n### Train using self critical\n\nFirst you should preprocess the dataset and get the cache for calculating cider score:\n```\n$ python scripts/prepro_ngrams.py --input_json data/dataset_coco.json --dict_json data/cocotalk.json --output_pkl data/coco-train --split train\n```\n\nThen, copy the model from the pretrained model using cross entropy. (It's not mandatory to copy the model, just for back-up)\n```\n$ bash scripts/copy_model.sh fc fc_rl\n```\n\nThen\n```bash\n$ python tools/train.py --id fc_rl --caption_model newfc --input_json data/cocotalk.json --input_fc_dir data/cocotalk_fc --input_att_dir data/cocotalk_att --input_label_h5 data/cocotalk_label.h5 --batch_size 10 --learning_rate 5e-5 --start_from log_fc_rl --checkpoint_path log_fc_rl --save_checkpoint_every 6000 --language_eval 1 --val_images_use 5000 --self_critical_after 30 --cached_tokens coco-train-idxs --max_epoch 50 --train_sample_n 5\n```\n\nor \n```bash\n$ python tools/train.py --cfg configs/fc_rl.yml --id fc_rl\n```\n\n\nYou will see a huge boost on Cider score, : ).\n\n**A few notes on training.** Starting self-critical training after 30 epochs, the CIDEr score goes up to 1.05 after 600k iterations (including the 30 epochs pertraining).\n\n## Generate image captions\n\n### Evaluate on raw images\n\n**Note**: this doesn't work for models trained with bottomup feature.\nNow place all your images of interest into a folder, e.g. `blah`, and run\nthe eval script:\n\n```bash\n$ python tools/eval.py --model model.pth --infos_path infos.pkl --image_folder blah --num_images 10\n```\n\nThis tells the `eval` script to run up to 10 images from the given folder. If you have a big GPU you can speed up the evaluation by increasing `batch_size`. Use `--num_images -1` to process all images. The eval script will create an `vis.json` file inside the `vis` folder, which can then be visualized with the provided HTML interface:\n\n```bash\n$ cd vis\n$ python -m SimpleHTTPServer\n```\n\nNow visit `localhost:8000` in your browser and you should see your predicted captions.\n\n### Evaluate on Karpathy's test split\n\n```bash\n$ python tools/eval.py --dump_images 0 --num_images 5000 --model model.pth --infos_path infos.pkl --language_eval 1 \n```\n\nThe defualt split to evaluate is test. The default inference method is greedy decoding (`--sample_method greedy`), to sample from the posterior, set `--sample_method sample`.\n\n**Beam Search**. Beam search can increase the performance of the search for greedy decoding sequence by ~5%. However, this is a little more expensive. To turn on the beam search, use `--beam_size N`, N should be greater than 1.\n\n### Evaluate on COCO test set\n\n```bash\n$ python tools/eval.py --input_json cocotest.json --input_fc_dir data/cocotest_bu_fc --input_att_dir data/cocotest_bu_att --input_label_h5 none --num_images -1 --model model.pth --infos_path infos.pkl --language_eval 0\n```\n\nYou can download the preprocessed file `cocotest.json`, `cocotest_bu_att` and `cocotest_bu_fc` from [link](https://drive.google.com/open?id=1eCdz62FAVCGogOuNhy87Nmlo5_I0sH2J).\n\n## Miscellanea\n**Using cpu**. The code is currently defaultly using gpu; there is even no option for switching. If someone highly needs a cpu model, please open an issue; I can potentially create a cpu checkpoint and modify the eval.py to run the model on cpu. However, there's no point using cpus to train the model.\n\n**Train on other dataset**. It should be trivial to port if you can create a file like `dataset_coco.json` for your own dataset.\n\n**Live demo**. Not supported now. Welcome pull request.\n\n## For more advanced features:\n\nCheckout [ADVANCED.md](ADVANCED.md).\n\n## Reference\n\nIf you find this repo useful, please consider citing (no obligation at all):\n\n```\n@article{luo2018discriminability,\n  title={Discriminability objective for training descriptive captions},\n  author={Luo, Ruotian and Price, Brian and Cohen, Scott and Shakhnarovich, Gregory},\n  journal={arXiv preprint arXiv:1803.04376},\n  year={2018}\n}\n```\n\nOf course, please cite the original paper of models you are using (You can find references in the model files).\n\n## Acknowledgements\n\nThanks the original [neuraltalk2](https://github.com/karpathy/neuraltalk2) and awesome PyTorch team.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruotianluo%2Fself-critical.pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruotianluo%2Fself-critical.pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruotianluo%2Fself-critical.pytorch/lists"}