{"id":19520154,"url":"https://github.com/osu-nlp-group/mqa","last_synced_at":"2025-04-26T07:31:35.038Z","repository":{"id":198067305,"uuid":"700010507","full_name":"OSU-NLP-Group/MQA","owner":"OSU-NLP-Group","description":"Multimodal Question Answering for Unified Information Extraction","archived":false,"fork":false,"pushed_at":"2024-10-22T15:00:17.000Z","size":2341,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-04T10:03:32.219Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-03T19:14:47.000Z","updated_at":"2025-02-21T09:33:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"9b3a5420-e801-4da3-b886-cfcc9c7d4814","html_url":"https://github.com/OSU-NLP-Group/MQA","commit_stats":null,"previous_names":["osu-nlp-group/mqa-mie"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FMQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/MQA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250953586,"owners_count":21513395,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:06.129Z","updated_at":"2025-04-26T07:31:30.327Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003ccentering\u003e MQA\n\nThis is the official repo for [Multimodal Question Answering for Unified Information Extraction](https://arxiv.org/abs/2310.03017)\n\n# Abstract\n\nMultimodal information extraction (MIE) aims to extract structured information from unstructured multimedia content. Due to the diversity of tasks and settings, most current MIE models are task-specific and data-intensive, which limits their generalization to real-world scenarios with diverse task requirements and limited labeled data. To address these issues, we propose a novel multimodal question answering (MQA) framework to unify three MIE tasks by reformulating them into a unified span extraction and multi-choice QA pipeline. Extensive experiments on six datasets show that: 1) Our MQA framework consistently and significantly improves the performances of various off-the-shelf large multimodal models (LMM) on MIE tasks, compared to vanilla prompting. 2) In the zero-shot setting, MQA outperforms previous state-of-the-art baselines by a large margin. In addition, the effectiveness of our framework can successfully transfer to the few-shot setting, enhancing LMMs on a scale of 10B parameters to be competitive or outperform much larger language models such as ChatGPT and GPT-4. Our MQA framework can serve as a general principle of utilizing LMMs to better solve MIE and potentially other downstream multimodal tasks.\n\n![image-20240310231907687](./imgs/image-20240310231907687.png)\n\n\n\n# Method\n\n![image-20240310232122683](./imgs/image-20240310232122684)\n\nWith the vanilla prompting strategy, LMMs directly identify entities. This complicated and error-prone process may lead to inferior results (e.g., one same span can be classified as two different entity types). In contrast, our MQA framework decomposes a MIE task into two cascaded phases: **span extraction and multichoice QA**. Spans are extracted as candidates for later multi-choice QA. Each candidate span can be classified into pre-defined categories and one additional none-of-the-above option (E) to discard false positives from the span extraction stage.\n\n\n\n\n\n# Usage\n\n\n\n## Installation\n\n```\nconda create -n mqa python=3.8\nconda activate mqa\npip install salesforce-lavis\n```\n\n\n\n## Data Preparation\n\n**Twitter2015 \u0026 Twitter2017**\n\nThe text data follows the conll format. You can download the Twitter2015 data via this [link](https://drive.google.com/file/d/1qAWrV9IaiBadICFb7mAreXy3llao_teZ/view?usp=sharing) and download the Twitter2017 data via this [link](https://drive.google.com/file/d/1ogfbn-XEYtk9GpUECq1-IwzINnhKGJqy/view?usp=sharing). Please place them in `data/`.\n\n**MNRE-V1 \u0026 MNRE-V2**\n\nFor MRE-V1, please download the data from https://github.com/thecharm/MNRE/tree/main/Version-1\n\nAnd for MRE-V2, please refer to the https://github.com/thecharm/MNRE\n\n**MEE**\n\nThe images and text articles are in [m2e2_rawdata](https://drive.google.com/file/d/1xtFMjt_eYgeBts5rBomOWbPo7wV_mnhy/view?usp=sharing), and annotations are in [m2e2_annotation](http://blender.cs.illinois.edu/software/m2e2/m2e2_v0.1/m2e2_annotations.zip).\n\n\n\nThe final data structural should be ordered as below:\n\n```\nMQA\n |-- data\n |    |-- mner\n |    |    |-- twitter2015  # text data\n |    |    |    |-- train.txt\n |    |    |    |-- valid.txt\n |    |    |    |-- test.txt\n |    |    |-- twitter2015_images       # raw image data\n |    |    |-- twitter2017\n |    |    |-- twitter2017_images\n |    |-- mre\n |    |    |-- mre_v1         \n |    |    |-- mre_v2   \n |    |-- mee\n |    |    |-- annotations       \n |    |    |-- raw_data     \n```\n\n\n\n### Run experiments\n\n\u003e Twitter\n\nTwitter2017, take BLIP2-Flan-T5 XL as an example\n\n```\nbash scripts/blip2_flant5xl/ner_span_17.sh\nbash scripts/blip2_flant5xl/ner_et_17.sh\n```\n\nTwitter2015, take BLIP2-Flan-T5 XL as an example\n\n```\nbash scripts/blip2_flant5xl/ner_span_15.sh\nbash scripts/blip2_flant5xl/ner_et_15.sh\n```\n\n\n\n\u003e #### MRE\n\nMNRE-V1, take BLIP2-Flan-T5 XL as an example\n\n```\nbash scripts/blip2_flant5xl/re.sh\n```\n\n\n\n\u003e #### MIED\n\n```\nbash scripts/blip2_flant5xl/iee.sh\n```\n\n\n\n\u003e #### MTED\n\n```\nbash scripts/blip2_flant5xl/ee_span.sh\nbash scripts/blip2_flant5xl/ee_et.sh\n```\n\n\n\n# Citation\n\nIf you find MQA useful for your work, please cite using the following BibTeX:\n\n```\n@article{sun2023multimodal,\n  title={Multimodal Question Answering for Unified Information Extraction},\n  author={Sun, Yuxuan and Zhang, Kai and Su, Yu},\n  journal={arXiv preprint arXiv:2310.03017},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fmqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fmqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fmqa/lists"}