{"id":18344310,"url":"https://github.com/lupantech/iconqa","last_synced_at":"2026-03-09T12:04:13.233Z","repository":{"id":60109580,"uuid":"398151408","full_name":"lupantech/IconQA","owner":"lupantech","description":"Data and code for NeurIPS 2021 Paper \"IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning\".","archived":false,"fork":false,"pushed_at":"2024-01-28T08:14:02.000Z","size":3741,"stargazers_count":52,"open_issues_count":1,"forks_count":15,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-21T20:46:34.109Z","etag":null,"topics":["commensense","dataset","mathai","pytorch","reasoning","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lupantech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-20T04:10:00.000Z","updated_at":"2025-03-20T16:26:17.000Z","dependencies_parsed_at":"2023-01-19T04:15:13.735Z","dependency_job_id":null,"html_url":"https://github.com/lupantech/IconQA","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2FIconQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2FIconQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2FIconQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupantech%2FIconQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lupantech","download_url":"https://codeload.github.com/lupantech/IconQA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247450743,"owners_count":20940938,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commensense","dataset","mathai","pytorch","reasoning","vqa"],"created_at":"2024-11-05T21:05:35.509Z","updated_at":"2026-03-09T12:04:08.198Z","avatar_url":"https://github.com/lupantech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Introduction\n\n![PyTorch](https://img.shields.io/badge/PyTorch-v1.9.0-green) ![Huggingface](https://img.shields.io/badge/Hugging%20Face-v0.0.12-green) ![Torchvision](https://img.shields.io/badge/Torchvision-v0.10.0-green) \n\n![VQA](https://img.shields.io/badge/Task-VQA-orange) ![MathAI](https://img.shields.io/badge/Task-MathAI-orange) ![Diagram](https://img.shields.io/badge/Task-Diagram-orange) ![IconQA](https://img.shields.io/badge/Dataset-IconQA%20-blue) ![Icon645](https://img.shields.io/badge/Dataset-Icon645-blue) ![Transformer](https://img.shields.io/badge/Model-Transformer-red) ![Pre-trained](https://img.shields.io/badge/Model-Pre--trained-red) \n\nData and code for NeurIPS 2021 Paper \"[IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://openreview.net/pdf?id=uXa9oBDZ9V1)\".\n\nWe propose a new challenging benchmark, icon question answering (IconQA), which aims to highlight the importance of **abstract diagram understanding** and **comprehensive cognitive reasoning** in real-world diagram word problems. For this benchmark, we build up a large-scale IconQA dataset that consists of three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. Compared to existing VQA benchmarks, IconQA requires not only **perception skills** like object recognition and text understanding, but also diverse **cognitive reasoning** skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning.\n\n![iconqa examples](data/iconqa_examples.png)\n\nThere are three different sub-tasks in **IconQA**:\n\n- 57,672 multi-image-choice questions\n- 31,578 multi-text-choice questions\n- 18,189 filling-in-the-blank questions\n\n| Sub-Tasks              | Train  | Validation | Test   | Total  |\n| ---------------------- | ------ | ---------- | ------ | ------ |\n| *Multi-image-choice*   | 34,603 | 11,535     | 11,535 | 57,672 |\n| *Multi-text-choice*    | 18,946 | 6,316      | 6,316  | 31,578 |\n| *Filling-in-the-blank* | 10,913 | 3,638      | 3,638  | 18,189 |\n\nWe further develop a strong model, **Patch-TRM**, which parses the diagram in a pyramid layout and applies cross-modal Transformers to learn the joint diagram-question feature. Patch-TRM takes patches parsed from a hierarchical pyramid layout and embeds them through ResNet pre-trained on our Icon645 dataset. The joint diagram-question feature is learned via cross-modal Transformers followed by the attention module.\n\n![model](model.png)\n\nFor more details, you can find our website [here](https://iconqa.github.io/) and our paper [here](https://openreview.net/pdf?id=uXa9oBDZ9V1).\n\n\n\n## Download the IconQA Dataset\n\nYou can download **IconQA** [here](https://iconqa2021.s3.us-west-1.amazonaws.com/iconqa_data.zip) or from [Google Drive](https://drive.google.com/file/d/1Xqdt1zMcMZU5N_u1SAIjk-UAclriynGx), then unzip the dataset into `root_dir/data`. \n\nNext, download pre-trained models [here](https://iconqa2021.s3.us-west-1.amazonaws.com/saved_models.zip) or from [Google Drive](https://drive.google.com/file/d/1cGHqvOK-aMqby21qeCLs4vv6wnWK3n4E), then unzip them into `root_dir`. \n\nOr run the command by:\n\n```shell\n. tools/download_data_and_models.sh\n```\n\n\n\n## Run the Patch-TRM model for IconQA\n\n### Requirements\n\n```shell\npython=3.6.9\nh5py=3.1.0\nhuggingface-hub=0.0.12\nnumpy=1.19.5\nPillow=8.3.1\ntorch=1.9.0+cu111\ntorchvision=0.10.0+cu111\ntqdm=4.61.2\n```\n\nInstall all required python dependencies:\n\n```shell\npip install -r requirements.txt\n```\n\n### Process IconQA Data\n\nGenerate the question dictionary:\n\n```shell\ncd tools\npython create_dictionary.py\n```\n\nGenerate answer labels:\n\n```shell\npython create_ans_label.py\n```\n\n### Generate image features\n\nGenerate the image patch features from the icon classifier model that is pre-trained on our proposed Icon645 dataset:\n\n```shell\npython generate_img_patch_feature.py --icon_pretrained True --patch_split 79\n```\n\n- `--icon_pretrained True`: the backbone network is pre-trained on icon data\n- `--patch_split 79`: the image is hierarchically parsed into 79 patches before feature extraction\n\nGenerate the image choice features for the `multi-image-choice` sub-task from the icon classifier model that is pre-trained on our proposed Icon645 dataset:\n\n```shell\npython generate_img_choice_feature.py --icon_pretrained True\n```\n\n- `--icon_pretrained True`: the backbone network is pre-trained on icon data\n\nOptionally, you can set `--icon_pretrained False` to generate image features from the ResNet101 model pre-trained on natural image dataset ImageNet.\n\nThe above steps are time-consuming and can take several hours. Instead, you can alternatively download the extracted features [here](https://iconqa2021.s3.us-west-1.amazonaws.com/embeddings.zip) or from [Google Drive](https://drive.google.com/file/d/1VuEpfqUCnv1gVa3roo9HpxtjsQ5o4Zqd), then unzip them into `root_dir/data`. Or run the command by:\n\n```shell\n. tools/download_img_feats.sh\n```\n\nBefore moving on, please check the following directories:\n\n```\ndata/\n├── dictionary.pkl\n├── iconqa_data\n│   └── iconqa\n│       ├── test\n│       ├── train\n│       └── val\n├── img_choice_embeddings\n│   └── resnet101_pool5_icon\n│       ├── iconqa_test_choose_img_resnet101_pool5_icon.pth\n│       ├── iconqa_train_choose_img_resnet101_pool5_icon.pth\n│       └── iconqa_val_choose_img_resnet101_pool5_icon.pth\n├── patch_embeddings\n│   └── resnet101_pool5_79_icon\n│       ├── iconqa_test_choose_img_resnet101_pool5_79_icon.pth\n│       ├── iconqa_test_choose_txt_resnet101_pool5_79_icon.pth\n│       ├── iconqa_test_fill_in_blank_resnet101_pool5_79_icon.pth\n│       ├── iconqa_train_choose_img_resnet101_pool5_79_icon.pth\n│       ├── iconqa_train_choose_txt_resnet101_pool5_79_icon.pth\n│       ├── iconqa_train_fill_in_blank_resnet101_pool5_79_icon.pth\n│       ├── iconqa_val_choose_img_resnet101_pool5_79_icon.pth\n│       ├── iconqa_val_choose_txt_resnet101_pool5_79_icon.pth\n│       └── iconqa_val_fill_in_blank_resnet101_pool5_79_icon.pth\n├── pid_splits.json\n├── problems.json\n├── trainval_choose_img_ans2label.pkl\n├── trainval_choose_img_label2ans.pkl\n├── trainval_choose_txt_ans2label.pkl\n├── trainval_choose_txt_label2ans.pkl\n├── trainval_fill_in_blank_ans2label.pkl\n└── trainval_fill_in_blank_label2ans.pkl\n\nsaved_models/\n├── choose_img\n│   └── exp_paper\n│       └── best_model.pth\n├── choose_txt\n│   └── exp_paper\n│       └── best_paper.pth\n├── fill_in_blank\n│   └── exp_paper\n│       └── best_paper.pth\n└── icon_classification_ckpt\n    └── icon_resnet101_LDAM_DRW_lr0.01_0\n        └── ckpt.epoch66_best.pth.tar\n```\n\n### Run the the *filling-in-the-blank* sub-task\n\nTrain the Patch_TRM model for the *filling-in-the-blank* sub-task:\n\n```shell\ncd run_fill_in_blank\npython train.py --model patch_transformer_ques_bert --label exp0 \n```\n\nEvaluate the Patch_TRM model for the *filling-in-the-blank* sub-task:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp0\n```\n\nOr, you can evaluate the Patch_TRM model for the *filling-in-the-blank* sub-task with our trained model:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp_paper\n```\n\n### Run the *multi-text-choice* sub-task\n\nTrain the Patch_TRM model for the *multi-text-choice* sub-task:\n\n```shell\ncd run_choose_txt\npython train.py --model patch_transformer_ques_bert --label exp0 \n```\n\nEvaluate the Patch_TRM model for the *multi-text-choice* sub-task:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp0\n```\n\nOr, you can evaluate the Patch_TRM model for the *multi-text-choice* sub-task with our trained model:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp_paper\n```\n\n### Run the *multi-image-choice* sub-task\n\nTrain the Patch_TRM model for the *multi-image-choice* sub-task:\n\n```shell\ncd run_choose_img\npython train.py --model patch_transformer_ques_bert --label exp0 \n```\n\nEvaluate the Patch_TRM model for the *multi-image-choice* sub-task:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp0\n```\n\nOr, you can evaluate the Patch_TRM model for the *multi-image-choice* sub-task with our trained model:\n\n```shell\npython eval.py --model patch_transformer_ques_bert --label exp_paper\n```\n\n### Evaluate the IconQA results\n\nCalculate the accuracies over different skills based on result json files reported in the paper:\n\n```shell\ncd tools\npython sub_acc.py \\\n--fill_in_blank_result exp_patch_transformer_ques_bert.json \\\n--choose_txt_result exp_patch_transformer_ques_bert.json \\\n--choose_img_result exp_patch_transformer_ques_bert.json\n```\n\nCalculate the accuracies over different skills based on user-specified result json files:\n\n```shell\npython sub_acc.py \\\n--fill_in_blank_result exp0_patch_transformer_ques_bert.json \\\n--choose_txt_result exp0_patch_transformer_ques_bert.json \\\n--choose_img_result exp0_patch_transformer_ques_bert.json\n```\n\n\n\n## Icon645 Dataset\n\nIn addition to **IconQA**, we also present **Icon645**, a large-scale dataset of icons that cover a wide range of objects:\n\n- **645,687** colored icons\n- **377** different icon classes (class mapping is stored in [icon645_classes.json](https://github.com/lupantech/IconQA/blob/main/data/icon645_classes.json))\n\nThese collected icon classes are frequently mentioned in the IconQA questions. In this work, we use the icon data to pre-train backbone networks on the icon classification task in order to extract semantic representations from abstract diagrams in IconQA. On top of pre-training encoders, the large-scale icon data could also contribute to open research on abstract aesthetics and symbolic visual understanding. \n\n![icon_examples](data/icon645_examples.png)\n\nYou can download **Icon645** [here](https://iconqa2021.s3.us-west-1.amazonaws.com/icon645.zip) or from [Google Drive](https://drive.google.com/file/d/1AsqzjBjgJedgnVAOpYA9WRfMN5k6w9an). Or run the command by:\n\n```shell\ncd data\nwget https://iconqa2021.s3.us-west-1.amazonaws.com/icon645.zip\nunzip icon645.zip\n```\n\nFile structures for the **Icon645** dataset:\n\n```\nicon645\n|   LICENCE.md\n|   metadata.json\n└───colored_icons_final\n    |\n    └───acorn\n    |   |   image_id1.png\n    |   |   image_id2.png\n    |   |   ...\n    |   \n    └───airplane\n    |   |   image_id3.png\n    |   |   ...\n    |      \n    |   ...\n```\n\n\n\n## Citation\n\nIf the paper or the dataset inspires you, please cite us:\n\n```\n@inproceedings{lu2021iconqa,\n  title = {IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning},\n  author = {Lu, Pan and Qiu, Liang and Chen, Jiaqi and Xia, Tony and Zhao, Yizhou and Zhang, Wei and Yu, Zhou and Liang, Xiaodan and Zhu, Song-Chun},\n  booktitle = {The 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks},\n  year = {2021}\n}\n```\n\n\n\n## License\n\n[![License: CC BY-SA 4.0](https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-sa/4.0/)\n\nOur dataset is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-nc-sa/4.0/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupantech%2Ficonqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flupantech%2Ficonqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupantech%2Ficonqa/lists"}