{"id":18690299,"url":"https://github.com/cdancette/rubi.bootstrap.pytorch","last_synced_at":"2025-07-03T14:34:10.770Z","repository":{"id":73707752,"uuid":"188815682","full_name":"cdancette/rubi.bootstrap.pytorch","owner":"cdancette","description":"NeurIPS 2019 Paper: RUBi : Reducing Unimodal Biases for Visual Question Answering","archived":false,"fork":false,"pushed_at":"2021-03-29T17:15:05.000Z","size":187,"stargazers_count":60,"open_issues_count":3,"forks_count":14,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T05:54:19.445Z","etag":null,"topics":["bias","bias-reduction","deep-learning","pytorch","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cdancette.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-27T09:42:38.000Z","updated_at":"2025-04-09T07:42:50.000Z","dependencies_parsed_at":"2023-02-27T06:45:47.151Z","dependency_job_id":null,"html_url":"https://github.com/cdancette/rubi.bootstrap.pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Frubi.bootstrap.pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Frubi.bootstrap.pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Frubi.bootstrap.pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdancette%2Frubi.bootstrap.pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cdancette","download_url":"https://codeload.github.com/cdancette/rubi.bootstrap.pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525156,"owners_count":21118616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bias","bias-reduction","deep-learning","pytorch","vqa"],"created_at":"2024-11-07T10:47:13.120Z","updated_at":"2025-04-12T05:54:29.312Z","avatar_url":"https://github.com/cdancette.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RUBi : Reducing Unimodal Biases for Visual Question Answering\n\nThis is the code for the NeurIPS 2019 article available here: https://arxiv.org/abs/1906.10169.\n\nThis paper was written by [Rémi Cadene](http://www.remicadene.com/), [Corentin Dancette](https://cdancette.fr), Hedi Ben Younes, [Matthieu Cord](http://webia.lip6.fr/~cord/) and [Devi Parikh](https://www.cc.gatech.edu/~parikh/).\n\n\n**RUBi** is a learning strategy to reduce biases in VQA models. \nIt relies on a question-only branch plugged at the end of a VQA model. \n\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/cdancette/rubi.bootstrap.pytorch/blob/master/assets/model_classic.png?raw=true\" width=\"300px\" style=\"margin-right:50px;\"/\u003e    \n    \u003cimg src=\"https://github.com/cdancette/rubi.bootstrap.pytorch/blob/master/assets/model_rubi.png?raw=true\" width=\"300px\" style=\"margin-left: 50px\";/\u003e\n\u003c/p\u003e\n\n#### Summary\n\n* [Installation](#installation)\n    * [As a standalone project](#1-as-standalone-project)\n    * [As a python library](#1-as-a-python-library)\n    * [Download datasets](#3-download-datasets)\n* [Quick start](#quick-start)\n    * [Train a model](#train-a-model)\n    * [Evaluate a model](#evaluate-a-model)\n* [Reproduce results](#reproduce-results)\n    * [VQACP2](#vqa-CP-v2-dataset)\n    * [VQA2](#vqa-v2-dataset)\n* [Useful commands](#useful-commands)\n* [Authors](#authors)\n* [Acknowledgment](#acknowledgment)\n\n\n## Installation\n\nWe don't provide support for python 2. We advise you to install python 3 with [Anaconda](https://www.continuum.io/downloads). Then, you can create an environment.\n\n### 1. As standalone project\n\n```\nconda create --name rubi python=3.7\nsource activate rubi\ngit clone --recursive https://github.com/cdancette/rubi.bootstrap.pytorch.git\ncd rubi.bootstrap.pytorch\npip install -r requirements.txt\n```\n\n### (1. As a python library)\n\nTo install the library \n```\ngit clone https://github.com/cdancette/rubi.bootstrap.pytorch.git\npython setup.py install\n```\n\nThen by importing the `rubi` python module, you can access datasets and models in a simple way.\n\n```python\nfrom rubi.models.networks.rubi import RUBiNet\n```\n\n\n**Note:** This repo is built on top of [block.bootstrap.pytorch](https://github.com/Cadene/block.bootstrap.pytorch). We import VQA2, TDIUC, VGenome from this library.\n\n### 2. Download datasets\n\nDownload annotations, images and features for VQA experiments:\n```\nbash rubi/datasets/scripts/download_vqa2.sh\nbash rubi/datasets/scripts/download_vqacp2.sh\n```\n\n\n## Quick start\n\n### The RUBi model\n\nThe main model is RUBi. \n\n```python\nfrom rubi.models.networks.rubi import RUBiNet\n```\n\nRUBi takes as input another VQA model, adds a question branch around it. The question predictions are merged with the original predictions.\nRUBi returns the new predictions that are used to train the VQA model.\n\nFor an example base model, you can check the [baseline model](https://github.com/cdancette/rubi.pytorch/blob/master/rubi/models/networks/baseline_net.py). The model must return the raw predictions (before softmax) in a dictionnary, with the key `logits`.\n\n\n### Train a model\n\nThe [boostrap/run.py](https://github.com/Cadene/bootstrap.pytorch/blob/master/bootstrap/run.py) file load the options contained in a yaml file, create the corresponding experiment directory and start the training procedure. For instance, you can train our best model on VQA2 by running:\n```\npython -m bootstrap.run -o rubi/options/vqacp2/rubi.yaml\n```\nThen, several files are going to be created in `logs/vqa2/rubi`:\n- [options.yaml](https://github.com/Cadene/block.bootstrap.pytorch/blob/master/assets/logs/vrd/block/options.yaml) (copy of options)\n- [logs.txt](https://github.com/Cadene/block.bootstrap.pytorch/blob/master/assets/logs/vrd/block/logs.txt) (history of print)\n- [logs.json](https://github.com/Cadene/block.bootstrap.pytorch/blob/master/assets/logs/vrd/block/logs.json) (batchs and epochs statistics)\n- [view.html](http://htmlpreview.github.io/?https://raw.githubusercontent.com/Cadene/block.bootstrap.pytorch/master/assets/logs/vrd/block/view.html?token=AEdvLlDSYaSn3Hsr7gO5sDBxeyuKNQhEks5cTF6-wA%3D%3D) (learning curves)\n- ckpt_last_engine.pth.tar (checkpoints of last epoch)\n- ckpt_last_model.pth.tar\n- ckpt_last_optimizer.pth.tar\n- ckpt_best_eval_epoch.accuracy_top1_engine.pth.tar (checkpoints of best epoch)\n- ckpt_best_eval_epoch.accuracy_top1_model.pth.tar\n- ckpt_best_eval_epoch.accuracy_top1_optimizer.pth.tar\n\nMany options are available in the [options directory](https://github.com/cdancette/rubi.bootstrap.pytorch/blob/master/rubi/options).\n\n### Evaluate a model\n\nThere is no testing set on VQA-CP v2, our main dataset. The evaluation is done on the validation set.\n\nFor a model trained on VQA v2, you can evaluate your model on the testing set. In this example, [boostrap/run.py](https://github.com/Cadene/bootstrap.pytorch/blob/master/bootstrap/run.py) load the options from your experiment directory, resume the best checkpoint on the validation set and start an evaluation on the testing set instead of the validation set while skipping the training set (train_split is empty). Thanks to `--misc.logs_name`, the logs will be written in the new `logs_predicate.txt` and `logs_predicate.json` files, instead of being appended to the `logs.txt` and `logs.json` files.\n```\npython -m bootstrap.run \\\n-o logs/vqa2/rubi/baseline.yaml \\\n--exp.resume best_accuracy_top1 \\\n--dataset.train_split \\\n--dataset.eval_split test \\\n--misc.logs_name test\n```\n\n\n## Reproduce results\n\n### VQA-CP v2 dataset\n\nUse this simple setup to reproduce our results on the valset of VQA-CP v2.\n\nBaseline: \n\n```bash\npython -m bootstrap.run \\\n-o rubi/options/vqacp2/baseline.yaml \\\n--exp.dir logs/vqacp2/baseline\n```\n\nRUBi : \n\n```bash\npython -m bootstrap.run \\\n-o rubi/options/vqacp2/rubi.yaml \\\n--exp.dir logs/vqacp2/rubi\n```\n\n#### Compare experiments on valset\n\nYou can compare experiments by displaying their best metrics on the valset.\n\n```\npython -m rubi.compare_vqacp2_rubi -d logs/vqacp2/rubi logs/vqacp2/baseline\n```\n\n### VQA v2 dataset\n\nBaseline: \n\n```bash\npython -m bootstrap.run \\\n-o rubi/options/vqa2/baseline.yaml \\\n--exp.dir logs/vqa2/baseline\n```\n\nRUBi : \n\n```bash\npython -m bootstrap.run \\\n-o rubi/options/vqa2/rubi.yaml \\\n--exp.dir logs/vqa2/rubi\n```\n\n\nYou can compare experiments by displaying their best metrics on the valset.\n\n```\npython -m rubi.compare_vqa2_rubi_val -d logs/vqa2/rubi logs/vqa2/baseline\n```\n\n#### Evaluation on test set\n\n```bash\npython -m bootstrap.run \\\n-o logs/vqa2/rubi/options.yaml \\\n--exp.resume best_eval_epoch.accuracy_top1 \\\n--dataset.train_split '' \\\n--dataset.eval_split test \\\n--misc.logs_name test\n```\n\n## Weights of best model\n\n\nThe weights for the model trained on VQA-CP v2 can be downloaded here : http://webia.lip6.fr/~cadene/rubi/ckpt_last_model.pth.tar\n\nTo use it : \n* Run this command once to create the experiment folder. Cancel it when the training starts\n\n```bash\npython -m bootstrap.run \\\n-o rubi/options/vqacp2/rubi.yaml \\\n--exp.dir logs/vqacp2/rubi\n```\n\n* Move the downloaded file to the experiment folder, and use the flag `--exp.resume last` to use this checkpoint : \n\n```bash\npython -m bootstrap.run \\\n-o logs/vqacp2/rubi/options.yaml \\\n--exp.resume last\n```\n\n\n## Useful commands\n\n### Use tensorboard instead of plotly\n\nInstead of creating a `view.html` file, a tensorboard file will be created:\n```\npython -m bootstrap.run -o rubi/options/vqacp2/rubi.yaml \\\n--view.name tensorboard\n```\n\n```\ntensorboard --logdir=logs/vqa2\n```\n\nYou can use plotly and tensorboard at the same time by updating the yaml file like [this one](https://github.com/Cadene/bootstrap.pytorch/blob/master/bootstrap/options/mnist_plotly_tensorboard.yaml#L38).\n\n\n### Use a specific GPU\n\nFor a specific experiment:\n```\nCUDA_VISIBLE_DEVICES=0 python -m boostrap.run -o rubi/options/vqacp2/rubi.yaml\n```\n\nFor the current terminal session:\n```\nexport CUDA_VISIBLE_DEVICES=0\n```\n\n### Overwrite an option\n\nThe boostrap.pytorch framework makes it easy to overwrite a hyperparameter. In this example, we run an experiment with a non-default learning rate. Thus, I also overwrite the experiment directory path:\n```\npython -m bootstrap.run -o rubi/options/vqacp2/rubi.yaml \\\n--optimizer.lr 0.0003 \\\n--exp.dir logs/vqacp2/rubi_lr,0.0003\n```\n\n### Resume training\n\nIf a problem occurs, it is easy to resume the last epoch by specifying the options file from the experiment directory while overwritting the `exp.resume` option (default is None):\n```\npython -m bootstrap.run -o logs/vqacp2/rubi/options.yaml \\\n--exp.resume last\n```\n\n## Cite\n\n```\n@article{cadene2019rubi,\n  title={RUBi: Reducing Unimodal Biases for Visual Question Answering},\n  author={Cadene, Remi and Dancette, Corentin and Cord, Matthieu and Parikh, Devi and others},\n  journal={Advances in Neural Information Processing Systems},\n  volume={32},\n  pages={841--852},\n  year={2019}\n}\n```\n\n## Authors\n\nThis code was made available by [Corentin Dancette](https://cdancette.fr) and [Rémi Cadene](http://www.remicadene.com/)\n\n## Acknowledgment\n\nSpecial thanks to the authors of [VQA2](TODO), [TDIUC](TODO), [VisualGenome](TODO) and [VQACP2](TODO), the datasets used in this research project.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdancette%2Frubi.bootstrap.pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcdancette%2Frubi.bootstrap.pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdancette%2Frubi.bootstrap.pytorch/lists"}