{"id":17128328,"url":"https://github.com/jiasenlu/adaptiveattention","last_synced_at":"2025-04-09T18:18:09.056Z","repository":{"id":78161063,"uuid":"82971430","full_name":"jiasenlu/AdaptiveAttention","owner":"jiasenlu","description":"Implementation of \"Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning\"","archived":false,"fork":false,"pushed_at":"2017-12-26T18:51:21.000Z","size":3927,"stargazers_count":335,"open_issues_count":19,"forks_count":74,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-09T18:18:02.000Z","etag":null,"topics":["attention-mechanism","image-captioning","torch"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1612.01887","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jiasenlu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-02-23T21:06:19.000Z","updated_at":"2025-02-23T09:04:06.000Z","dependencies_parsed_at":"2023-02-27T21:30:40.493Z","dependency_job_id":null,"html_url":"https://github.com/jiasenlu/AdaptiveAttention","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiasenlu%2FAdaptiveAttention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiasenlu%2FAdaptiveAttention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiasenlu%2FAdaptiveAttention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiasenlu%2FAdaptiveAttention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jiasenlu","download_url":"https://codeload.github.com/jiasenlu/AdaptiveAttention/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085326,"owners_count":21045139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","image-captioning","torch"],"created_at":"2024-10-14T19:06:46.485Z","updated_at":"2025-04-09T18:18:09.039Z","avatar_url":"https://github.com/jiasenlu.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AdaptiveAttention\nImplementation of \"[Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://arxiv.org/pdf/1612.01887.pdf)\"\n\n![teaser results](https://raw.github.com/jiasenlu/AdaptiveAttention/master/demo/fig1.png)\n\n### Requirements\n\nTo train the model require GPU with 12GB Memory, if you do not have GPU, you can directly use the pretrained model for inference. \n\nThis code is written in Lua and requires [Torch](http://torch.ch/). The preprocssinng code is in Python, and you need to install [NLTK](http://www.nltk.org/) if you want to use NLTK to tokenize the caption.\n\nYou also need to install the following package in order to sucessfully run the code.\n\n- [cudnn.torch](https://github.com/soumith/cudnn.torch)\n- [torch-hdf5](https://github.com/deepmind/torch-hdf5)\n- [lua-cjson](http://www.kyne.com.au/~mark/software/lua-cjson.php)\n- [iTorch](https://github.com/facebook/iTorch)\n\n##### Pretrained Model\nThe pre-trained model for COCO can be download [here](https://filebox.ece.vt.edu/~jiasenlu/codeRelease/AdaptiveAttention/model/COCO/).\nThe pre-trained model for Flickr30K can be download [here](https://filebox.ece.vt.edu/~jiasenlu/codeRelease/AdaptiveAttention/model/Flickr30k/). \n\n##### Vocabulary File\nDownload the corresponding Vocabulary file for [COCO](https://filebox.ece.vt.edu/~jiasenlu/codeRelease/AdaptiveAttention/data/COCO/) and [Flickr30k](https://filebox.ece.vt.edu/~jiasenlu/codeRelease/AdaptiveAttention/data/Flickr30k/) \n\n##### Download Dataset\nThe first thing you need to do is to download the data and do some preprocessing. Head over to the `data/` folder and run the correspodning ipython script. It will download, preprocess and generate coco_raw.json. \n\nDownload [COCO](http://mscoco.org/) and Flickr30k image dataset, extract the image and put under somewhere. \n \n\n### training a new model on MS COCO\nFirst train the Language model without finetune the image. \n```\nth train.lua -batch_size 20 \n```\nWhen finetune the CNN, load the saved model and train for another 15~20 epoch. \n```\nth train.lua -batch_size 16 -startEpoch 21 -start_from 'model_id1_20.t7'\n```\n\n\n### More Result about spatial attention and visual sentinel\n\n![teaser results](https://raw.github.com/jiasenlu/AdaptiveAttention/master/demo/fig2.png)\n\n![teaser results](https://raw.github.com/jiasenlu/AdaptiveAttention/master/demo/fig3.png)\n\nFor more visualization result, you can visit [here](https://filebox.ece.vt.edu/~jiasenlu/demo/caption_atten/demo.html)\n(it will load more than 1000 image and their result...)\n\n### Reference\nIf you use this code as part of any published research, please acknowledge the following paper\n```\n@misc{Lu2017Adaptive,\nauthor = {Lu, Jiasen and Xiong, Caiming and Parikh, Devi and Socher, Richard},\ntitle = {Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning},\njournal = {CVPR},\nyear = {2017}\n}\n```\n\n### Acknowledgement \n\nThis code is developed based on [NeuralTalk2](https://github.com/karpathy/neuraltalk2). \n\nThanks [Torch](http://torch.ch/) team and Facebook [ResNet](https://github.com/facebook/fb.resnet.torch) implementation. \n\n### License\n\nBSD 3-Clause License\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiasenlu%2Fadaptiveattention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjiasenlu%2Fadaptiveattention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiasenlu%2Fadaptiveattention/lists"}