{"id":18344305,"url":"https://github.com/lupantech/dual-mfa-vqa","last_synced_at":"2026-02-19T18:02:09.105Z","repository":{"id":153669199,"uuid":"111102906","full_name":"lupantech/dual-mfa-vqa","owner":"lupantech","description":"Co-attending Regions and Detections for VQA.","archived":false,"fork":false,"pushed_at":"2018-06-02T12:32:43.000Z","size":1512,"stargazers_count":40,"open_issues_count":1,"forks_count":14,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-21T22:39:43.694Z","etag":null,"topics":["aaai","attention-mechanism","caffe","faster-rcnn","multi-gpu","multi-modal","object-detection","torch","visual-question-answering","vqa"],"latest_commit_sha":null,"homepage":"","language":"Matlab","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lupantech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-17T12:49:02.000Z","updated_at":"2025-01-23T05:46:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa217d21-b708-471e-8998-03741c230842","html_url":"https://github.com/lupantech/dual-mfa-vqa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lupantech/dual-mfa-vqa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2Fdual-mfa-vqa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2Fdual-mfa-vqa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2Fdual-mfa-vqa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2Fdual-mfa-vqa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lupantech","download_url":"https://codeload.github.com/lupantech/dual-mfa-vqa/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2Fdual-mfa-vqa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29626612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T13:04:20.082Z","status":"ssl_error","status_checked_at":"2026-02-19T13:03:33.775Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aaai","attention-mechanism","caffe","faster-rcnn","multi-gpu","multi-modal","object-detection","torch","visual-question-answering","vqa"],"created_at":"2024-11-05T21:05:33.935Z","updated_at":"2026-02-19T18:02:09.086Z","avatar_url":"https://github.com/lupantech.png","language":"Matlab","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Dual-MFA-VQA\n\n\n\nCo-attending Regions and Detections with Multi-modal Multiplicative Embedding for VQA. \n\nThe network has two attention branches with the proposed multiplicative feature embedding scheme: one branch attends free-form image regions, another branch attends detection boxes for encoding question-related visual features.\n\nThis current code can get **66.09** on Open-Ended and 69.97 on Multiple-Choice on test-standard split for the VQA 1.0 dataset.\n\n**Spotlights**\n- **Paper on arXiv**: https://arxiv.org/abs/1711.06794\n- **Bounding box-based attention** with Faster R-CNN pre-trained model\n- **Co-attention** with whole image and object detection\n- **Multi-modal multiplicative embedding** feature method\n- **Torch implementation with multi-GPU acceleration**\n\n\n![ | center](https://github.com/lupantech/dual-mfa-vqa/blob/master/model.png?raw=true)\n\n### Dependencies\n\nThis main part of code is written in Lua and requires [Torch](http://torch.ch/docs/getting-started.html#). After installing torch, you can install these dependencies by running the following:\n\n##### Installing related torch libraries\n``` \ncd ~/torch\nluarocks install loadcaffe\nluarocks install hdf5\npip install h5py\nluarocks install optim\nluarocks install nn\nluarocks install math\nluarocks install image\nluarocks install dp\n```\n\n##### Installing rnn dependencies\n```\ncd ~/torch\ngit clone git@github.com:Element-Research/rnn.git\ncd rnn\nluarocks make rocks/rnn-scm-1.rockspec\n```\n\n\n##### Installing Cjson\n```\ncd /usr/local/\nsudo wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz\nsudo tar -xzvf  lua-cjson-2.1.0.tar.gz\ncd lua-cjson-2.1.0\nsudo luarocks make\nsudo rm ../lua-cjson-2.1.0.tar.gz\n```\n\n\n##### Installing NLTK\n\n``` \ncd /usr/share/\nsudo mkdir nltk_data\nsudo pip install -U nltk\npython -m nltk.downloader all\n```\n\n##### Installing GPU acceleration\n- If have an NVIDIA GPU and want to accelerate the model with CUDA, you'll also need to install [torch/cutorch](https://github.com/torch/cutorch) and [torch/cunn](https://github.com/torch/cunn); you can install these by running:\n\n``` \nluarocks install cutorch\nluarocks install cunn\nluarocks install cudnn\n```\n##### Installing cuDNN (Optional)\n- If you want to use NVIDIA's cuDNN library, you'll need to register for the CUDA Developer Program (it's free) and download the library from [NVIDIA's website](https://developer.nvidia.com/cudnn). An running script **just for an example**:\n``` \ncd ~/torch\n# download the right cudnn file to cuda version\ntar -xzvf  cudnn-7.5-linux-x64-v5.1.tgz\nsudo cp cuda/lib64/libcudnn* /usr/local/cuda-7.5/lib64/\nsudo cp cuda/include/cudnn.h /usr/local/cuda-7.5/include/\n```\n- You'll also need to install the [cuDNN bindings for Torch](https://github.com/soumith/cudnn.torch) by running:\n``` \nluarocks install cudnn\n```\n\n##### Installing nccl for Multi-GPU Optimization (Optional)\n- Optimized primitives for collective multi-GPU communication. Note that [NVIDIA/nccl](https://github.com/NVIDIA/nccl) is an optimization tool, and you can run the codes on multi-GPUs without nccl.\n``` \ncd torch/\ngit clone https://github.com/NVIDIA/nccl.git\n# build the library\ncd nccl/\nmake CUDA_HOME=/usr/local/cuda-7.5 test\n# update LIBRARY_PATH\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/plu/torch/nccl/build/lib\nsource ~/.bashrc\n# test demo\n$ ./build/test/single/all_reduce_test\n$ ./build/test/single/all_reduce_test 10000000\nluarocks install nccl\n```\n\n\n#### Caffe and faster-rcnn\nExtracting and visualizing bounding boxes are supported by [caffe](http://caffe.berkeleyvision.org/installation.html) and [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn). You can install Caffe and faster-rcnn following the [instructions](faster-rcnn-vqa/README.md).\n\nThen copy the faster-rcnn for vqa files to the target folder:\n``` \ncp ~/dual-mfa-vqa/faster-rcnn-vqa/tools/*.py  ~/py-faster-rcnn/tools/\n```\n\n\n### VQA Dataset\n\n#### Download mscoco images\n``` \nmkdir -p ~/VQA/Images/mscoco\ncd ~/VQA/Images/mscoco\nwget http://msvocds.blob.core.windows.net/coco2014/train2014.zip\nunzip train2014.zip\ncd ~/VQA/Images/mscoco\nwget http://msvocds.blob.core.windows.net/coco2014/val2014.zip\nunzip val2014.zip\ncd ~/VQA/Images/mscoco\nwget http://msvocds.blob.core.windows.net/coco2015/test2015.zip\nunzip test2015.zip\nln -s test2015 test-dev2015\n```\n\n#### Download VQA question and answers\n``` \nmkdir -p ~/VQA/Annotations\ncd ~/dual-mfa-vqa/data_train-val_test-dev_2k\npython vqa_preprocess.py --download 1\npython prepro_vqa.py\ncd ~/dual-mfa-vqa/data_train_test-dev_2k\npython vqa_preprocess.py\npython prepro_vqa.py\ncd ~/dual-mfa-vqa\nth prepro/prepro_seconds.lua\n```\n\n\n#### Download resnet image model\n``` \nmkdir -p ~/VQA/Images/Image_model\ncd ~/VQA/Image_model\nwget https://d2j0dndfm35trm.cloudfront.net/resnet-152.t7\nwget https://raw.githubusercontent.com/facebook/fb.resnet.torch/master/datasets/transforms.lua\n```\n\n#### Download faster-rcnn model\n``` \ncd ~/py-faster-rcnn/data/\nmkdir faster_rcnn_models\ncd faster_rcnn_models\nwget https://dl.dropboxusercontent.com/s/cotx0y81zvbbhnt/coco_vgg16_faster_rcnn_final.caffemodel?dl=0\nmv coco_vgg16_faster_rcnn_final.caffemodel?dl=0  coco_vgg16_faster_rcnn_final.caffemodel\n```\n\n#### Download skipthoughts models\nYou can download the pretrained Skipthoughts models to folder `skipthoughts_model/` for learning ([See more details](https://github.com/jnhwkim/MulLowBiVQA/tree/master/skipthoughts_model)):\n- [gru.t7 (332M)](https://drive.google.com/uc?export=download\u0026confirm=fFRb\u0026id=0B-75nmZV6j-JZng3VEtmeDdxVXM)\n- [lookup_2k.t7 (142M)](https://drive.google.com/uc?export=download\u0026confirm=liRP\u0026id=0B-75nmZV6j-JRmhIMnctVXpGMEU)\n\n#### Download pretrained VQA models\nThis current code can get **66.01** on Open-Ended and 70.04 on Multiple-Choice on test-tev split for the VQA 1.0 dataset. Download the pre-trained model `vqa_dual-mfa_model_6601.t7` (315M) from [here](https://drive.google.com/open?id=1OdMhh6lmqWgUuZ1hBxR1ymFbtCL12ihC) into folder `dual-mfa-vqa/model/save/`.\n\n\n### Extracting Image Features\n#### Free-form region based features\n- Image features for train dataset\n``` \ncd prepro\nth prepro_res_train.lua -batch_size 8\n```\n- Image features for test-dev dataset\n``` \nth prepro_res_test.lua -batch_size 8\n```\n\n#### Detection region based features\n- Image features for train dataset\n``` \npython extract_box_feat_train.py\n```\n- Image features for test dataset\n``` \npython extract_box_feat_train.py\n```\n- Bounding box coordinates for test dataset. You can directly download the result file `faster-rcnn_box4_19_test.h5` from [here](https://drive.google.com/open?id=1nhxbRcUyH2xoEQRajNbKLm0bxCuOzOCx).\n``` \npython extract_box_test.py\n```\n\n\n### Training\nNow, everything is ready, let's train the vqa network. Here are some common training ways for different needs.\n- Training the network on train dataset and validating on val dataset\n```\nth train.lua -phase 1 -val_nqs -1 -nGPU 4\n```\n- Training the network on train-val dataset with multi-GPUs (4 GPUs)\n```\nth train.lua -phase 2 -nGPU 4 -batchsize 300\n```\n- Training the network loading image features into memory (much faster, 200-300G memory is needed)\n```\nth train.lua -phase 1 -val_nqs 10000 -nGPU 4 -memory_ms -memory_frms\n```\n- Training the network from the previous checkpoint\n```\nth train.lua -phase 2 -nGPU 4 -memory_ms -load_checkpoint_path model/save/vqa_model_dual-mfa_6601.t7 -previous_iters 350000\n```\n\n- Main options useful training are listed as follows:\n\t - `phase`：training phase, `1`: train on Train, `2`: train on Train+Val\n\t - `vqa_type`: vqa dataset type, `vqa` or `coco-qa`\n\t - `memory_ms`: load image resnet feature to memory\n\t - `memory_frms`: load image fast-rcnn feature to memory\n\t - `val`: running validation\n\t - `val_nqs`: number of validation questions, `-1` for all questions\n\t - `batch_size`: batch_size for each iterations, change it to smaller value if out of the memory\n\t - `run_id`: running model id\n\t - `model_label`: model label name\n\t - `save_checkpoint_every`:  how often to save a model checkpoint\n\t - `skip_save_model`: skip saving t7 model\n\t - `cg_every`: How often do we collectgarbage in the training process, change it to smaller value if out of the memory\n\t - `quick_check`: quick check for code\n\t - `quickquick_check`: very quick check for code\n\t - `nGPU`: how many GPUs to use, 1 = use 1 GPU, change it to larger value if out of the memory\n\n\n### Evaluation \nEvaluate the pre-trained model on VQA dataset:\n``` \ncd ~/dual-mfa-vqa\nth eval.lua -model_path model/vqa_model_dual-mfa_6601.t7 -output_model_name vqa_model_dual-mfa_6601 -batch_size 10\n```\nThen you can submit the result jsons and obtain the evaluation scores:\n- Open-Ended for real images: [Submission](https://competitions.codalab.org/competitions/6961#participate-submit_results)\n- Multiple-Choice for real images: [Submission](https://competitions.codalab.org/competitions/6971#participate-submit_results)\n\n### COCO-QA Dataset\n\n- Download the dataset\n```\ncd data_coco\npython cocoqa_preprocess.py --download 1\n```\n- Preprocess the dataset\n```\npython prepro_cocoqa.py\n```\n\n- Extract the free-form image features\n``` \ncd prepro\nth prepro_res_coco.lua -batch_size 8\n```\n\n- Training the network\n``` \nth train.lua -vqa_type coco-qa -learning_rate 4e-4 -nGPU 4 -batch_size 300 \\\n-model_id 1 -model_label dual-mfa\n```\n- Evaluation based on WUPS\n``` bash\ncd ~/dual-mfa-vqa/metric\npython gen_wups_input.py\npython calculate_wups.py gt_ans_save.txt pd_ans_save.txt 0.9 \npython calculate_wups.py gt_ans_save.txt pd_ans_save.txt 0.0 \n```\n\n### Visualization\n- Generate the attention maps. You can directly download the result file of attention map from [here](https://drive.google.com/open?id=1PvXId4VOPrvw_y98Q-iPHjlxxg8IKWdf).\n``` \ncd ~/dual-mfa-vqa\nth eval_vis_att.lua -model_path model/vqa_model_dual-mfa_6601.t7 -output_model_name vqa_model_dual-mfa_6601 -batch_size 8\n```\n- Preprocess the question data before visualization:\n``` \ncd vis_att \npython vis_prepro.py\n```\n- Run the matlab file `vis_attention_demo.m` to show the results of attention maps.\n- Run the matlab file `vis_attention.m` to save the results of attention maps.\n\n![ | center ](https://github.com/lupantech/dual-mfa-vqa/blob/master/attention_map.png?raw=true)\n\n### Reference\n- **Paper on arXiv**: https://arxiv.org/abs/1711.06794\n- **Paper on AAAI**: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16249/16315\n\n\nIf you use this code as part of any published research, please acknowledge the following paper.\n```\n@inproceedings{lu2018co-attending,\n\ttitle={Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering.},\n\tauthor={Lu, Pan and Li, Hongsheng and Zhang, Wei and Wang, Jianyong and Wang, Xiaogang},\n\tbooktitle={AAAI 2018},\n\tpages={7218-7225},\n\tyear={2018}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupantech%2Fdual-mfa-vqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flupantech%2Fdual-mfa-vqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupantech%2Fdual-mfa-vqa/lists"}