{"id":18077563,"url":"https://github.com/rentainhe/trar-vqa","last_synced_at":"2025-08-28T12:11:38.395Z","repository":{"id":119477613,"uuid":"388655896","full_name":"rentainhe/TRAR-VQA","owner":"rentainhe","description":"[ICCV 2021] Official implementation of the paper \"TRAR: Routing the Attention Spans in Transformers for Visual Question Answering\"","archived":false,"fork":false,"pushed_at":"2021-10-11T06:02:29.000Z","size":949,"stargazers_count":65,"open_issues_count":5,"forks_count":18,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-31T11:45:24.788Z","etag":null,"topics":["attention","clevr","dynamic-network","iccv2021","local-and-global","multi-modal","multi-modal-learning","multi-modality","multi-scale-features","official","pytorch","transformer","vision-and-language","visual-question-answering","visualization","vqav2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rentainhe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-07-23T02:34:46.000Z","updated_at":"2024-09-28T12:38:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"501a537f-4340-4d12-8d28-1887b2c2ef9a","html_url":"https://github.com/rentainhe/TRAR-VQA","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rentainhe%2FTRAR-VQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rentainhe%2FTRAR-VQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rentainhe%2FTRAR-VQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rentainhe%2FTRAR-VQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rentainhe","download_url":"https://codeload.github.com/rentainhe/TRAR-VQA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223510333,"owners_count":17157306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","clevr","dynamic-network","iccv2021","local-and-global","multi-modal","multi-modal-learning","multi-modality","multi-scale-features","official","pytorch","transformer","vision-and-language","visual-question-answering","visualization","vqav2"],"created_at":"2024-10-31T11:45:36.774Z","updated_at":"2024-11-07T12:03:37.770Z","avatar_url":"https://github.com/rentainhe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TRAnsformer Routing Networks (TRAR)\r\nThis is an official implementation for ICCV 2021 paper [\"TRAR: Routing the Attention Spans in Transformers for Visual Question Answering\"](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhou_TRAR_Routing_the_Attention_Spans_in_Transformer_for_Visual_Question_ICCV_2021_paper.pdf). It currently includes the code for training TRAR on **VQA2.0** and **CLEVR** dataset. Our TRAR model for REC task is coming soon.\r\n\r\n## Updates\r\n- (2021/10/10) Release our TRAR-VQA project.\r\n- (2021/08/31) Release our pretrained `CLEVR` TRAR model on `train` split: [TRAR CLEVR Pretrained Models](MODEL.md#CLEVR).\r\n- (2021/08/18) Release our pretrained TRAR model on `train+val` split and `train+val+vg` split: [VQA-v2 TRAR Pretrained Models](MODEL.md#VQA-v2) \r\n- (2021/08/16) Release our `train2014`, `val2014` and `test2015` data. Please check our dataset setup page [DATA.md](DATA.md) for more details.\r\n- (2021/08/15) Release our pretrained weight on `train` split. Please check our model page [MODEL.md](MODEL.md) for more details.\r\n- (2021/08/13) The project page for TRAR is avaliable.\r\n\r\n## Introduction\r\n**TRAR vs Standard Transformer**\r\n\u003cp align=\"center\"\u003e\r\n\t\u003cimg src=\"misc/trar_block.png\" width=\"550\"\u003e\r\n\u003c/p\u003e\r\n\r\n**TRAR Overall**\r\n\u003cp align=\"center\"\u003e\r\n\t\u003cimg src=\"misc/trar_overall.png\" width=\"550\"\u003e\r\n\u003c/p\u003e\r\n\r\n## Table of Contents\r\n0. [Installation](#Installation)\r\n1. [Dataset setup](#Dataset-setup)\r\n2. [Config Introduction](#Config-Introduction)\r\n3. [Training](#Training)\r\n4. [Validation and Testing](#Validation-and-Testing)\r\n5. [Models](#Models)\r\n\r\n### Installation\r\n- Clone this repo\r\n```bash\r\ngit clone https://github.com/rentainhe/TRAR-VQA.git\r\ncd TRAR-VQA\r\n```\r\n\r\n- Create a conda virtual environment and activate it\r\n```bash\r\nconda create -n trar python=3.7 -y\r\nconda activate trar\r\n```\r\n\r\n- Install `CUDA==10.1` with `cudnn7` following the [official installation instructions](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)\r\n- Install `Pytorch==1.7.1` and `torchvision==0.8.2` with `CUDA==10.1`:\r\n```bash\r\nconda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch\r\n```\r\n- Install [Spacy](https://spacy.io/) and initialize the [GloVe](https://github-releases.githubusercontent.com/84940268/9f4d5680-4fed-11e9-9dd2-988cce16be55?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210815%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20210815T072922Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=1bd1bd4fc52057d8ac9eec7720e3dd333e63c234abead471c2df720fb8f04597\u0026X-Amz-SignedHeaders=host\u0026actor_id=48727989\u0026key_id=0\u0026repo_id=84940268\u0026response-content-disposition=attachment%3B%20filename%3Den_vectors_web_lg-2.1.0.tar.gz\u0026response-content-type=application%2Foctet-stream) as follows:\r\n```bash\r\npip install -r requirements.txt\r\nwget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz\r\npip install en_vectors_web_lg-2.1.0.tar.gz\r\n```\r\n\r\n### Dataset setup\r\nsee [DATA.md](DATA.md)\r\n\r\n### Config Introduction\r\nIn [trar.yml](configs/vqa/trar.yml) config we have these specific settings for `TRAR` model\r\n```\r\nORDERS: [0, 1, 2, 3]\r\nIMG_SCALE: 8 \r\nROUTING: 'hard' # {'soft', 'hard'}\r\nPOOLING: 'attention' # {'attention', 'avg', 'fc'}\r\nTAU_POLICY: 1 # {0: 'SLOW', 1: 'FAST', 2: 'FINETUNE'}\r\nTAU_MAX: 10\r\nTAU_MIN: 0.1\r\nBINARIZE: False\r\n```\r\n- `ORDERS=list`, to set the local attention window size for routing.`0` for global attention.\r\n- `IMG_SCALE=int`, which should be equal to the `image feature size` used for training. You should set `IMG_SCALE: 16` for `16 × 16` training features.\r\n- `ROUTING={'hard', 'soft'}`, to set the `Routing Block Type` in TRAR model.\r\n- `POOLING={'attention', 'avg', 'fc}`, to set the `Downsample Strategy` used in `Routing Block`.\r\n- `TAU_POLICY={0, 1, 2}`, to set the `temperature schedule` in training TRAR when using `ROUTING: 'hard'`.\r\n- `TAU_MAX=float`, to set the maximum temperature in training.\r\n- `TAU_MIN=float`, to set the minimum temperature in training.\r\n- `BINARIZE=bool`, binarize the predicted alphas (alphas: the prob of choosing one path), which means **during test time**, we only keep the maximum alpha and set others to zero. If `BINARIZE=False`, it will keep all of the alphas and get a weight sum of different routing predict result by alphas. **It won't influence the training time, just a small difference during test time**.\r\n\r\n**Note that please set `BINARIZE=False` when `ROUTING='soft'`, it's no need to binarize the path prob in soft routing block.**\r\n\r\n**`TAU_POLICY` visualization**\r\n\r\nFor `MAX_EPOCH=13` with `WARMUP_EPOCH=3` we have the following policy strategy:\r\n\u003cp align=\"center\"\u003e\r\n\t\u003cimg src=\"misc/policy_visualization.png\" width=\"550\"\u003e\r\n\u003c/p\u003e\r\n\r\n### Training\r\n**Train model on VQA-v2 with default hyperparameters:**\r\n```bash\r\npython3 run.py --RUN='train' --DATASET='vqa' --MODEL='trar'\r\n```\r\nand the training log will be seved to:\r\n```\r\nresults/log/log_run_\u003cVERSION\u003e.txt\r\n```\r\nArgs:\r\n- `--DATASET={'vqa', 'clevr'}` to choose the task for training\r\n- `--GPU=str`, e.g. `--GPU='2'` to train model on specific GPU device.\r\n- `--SPLIT={'train', 'train+val', train+val+vg'}`, which combines different training datasets. The default training split is `train`.\r\n- `--MAX_EPOCH=int` to set the total training epoch number.\r\n\r\n\r\n**Resume Training**\r\n\r\nResume training from specific saved model weights\r\n```bash\r\npython3 run.py --RUN='train' --DATASET='vqa' --MODEL='trar' --RESUME=True --CKPT_V=str --CKPT_E=int\r\n```\r\n- `--CKPT_V=str`: the specific checkpoint version\r\n- `--CKPT_E=int`: the resumed epoch number\r\n\r\n**Multi-GPU Training and Gradient Accumulation**\r\n1. Multi-GPU Training:\r\nAdd `--GPU='0, 1, 2, 3...'` after the training scripts.\r\n```bash\r\npython3 run.py --RUN='train' --DATASET='vqa' --MODEL='trar' --GPU='0,1,2,3'\r\n```\r\nThe batch size on each GPU will be divided into `BATCH_SIZE/GPUs` automatically.\r\n\r\n2. Gradient Accumulation:\r\nAdd `--ACCU=n` after the training scripts\r\n```bash\r\npython3 run.py --RUN='train' --DATASET='vqa' --MODEL='trar' --ACCU=2\r\n```\r\nThis makes the optimizer accumulate gradients for `n` mini-batches and update the model weights once. `BATCH_SIZE` should be divided by `n`.\r\n\r\n### Validation and Testing\r\n**Warning**: The args `--MODEL` and `--DATASET` should be set to the same values as those in the training stage.\r\n\r\n**Validate on Local Machine**\r\nOffline evaluation only support the evaluations on the `coco_2014_val` dataset now.\r\n1. Use saved checkpoint\r\n```bash\r\npython3 run.py --RUN='val' --MODEL='trar' --DATASET='{vqa, clevr}' --CKPT_V=str --CKPT_E=int\r\n```\r\n\r\n2. Use the absolute path\r\n```bash\r\npython3 run.py --RUN='val' --MODEL='trar' --DATASET='{vqa, clevr}' --CKPT_PATH=str\r\n```\r\n\r\n**Online Testing**\r\nAll the evaluations on the `test` dataset of VQA-v2 and CLEVR benchmarks can be achieved as follows:\r\n```bash\r\npython3 run.py --RUN='test' --MODEL='trar' --DATASET='{vqa, clevr}' --CKPT_V=str --CKPT_E=int\r\n```\r\n\r\nResult file are saved at:\r\n\r\n`results/result_test/result_run_\u003cCKPT_V\u003e_\u003cCKPT_E\u003e.json`\r\n\r\nYou can upload the obtained result json file to [Eval AI](https://evalai.cloudcv.org/web/challenges/challenge-page/163/overview) to evaluate the scores.\r\n\r\n### Models\r\nHere we provide our pretrained model and log, please see [MODEL.md](MODEL.md)\r\n\r\n## Acknowledgements\r\n- [openvqa](https://github.com/MILVLG/openvqa)\r\n- [grid-feats-vqa](https://github.com/facebookresearch/grid-feats-vqa)\r\n\r\n## Citation\r\nif TRAR is helpful for your research or you wish to refer the baseline results published here, we'd really appreciate it if you could cite this paper:\r\n```\r\n@InProceedings{Zhou_2021_ICCV,\r\n    author    = {Zhou, Yiyi and Ren, Tianhe and Zhu, Chaoyang and Sun, Xiaoshuai and Liu, Jianzhuang and Ding, Xinghao and Xu, Mingliang and Ji, Rongrong},\r\n    title     = {TRAR: Routing the Attention Spans in Transformer for Visual Question Answering},\r\n    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\r\n    month     = {October},\r\n    year      = {2021},\r\n    pages     = {2074-2084}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frentainhe%2Ftrar-vqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frentainhe%2Ftrar-vqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frentainhe%2Ftrar-vqa/lists"}