{"id":21441747,"url":"https://github.com/dmis-lab/liquid","last_synced_at":"2025-07-08T10:11:43.651Z","repository":{"id":97388631,"uuid":"597084627","full_name":"dmis-lab/LIQUID","owner":"dmis-lab","description":"LIQUID: A Framework for List Question Anwering Dataset Generation (AAAI 2023)","archived":false,"fork":false,"pushed_at":"2023-06-07T13:49:53.000Z","size":19,"stargazers_count":26,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-08T12:11:57.788Z","etag":null,"topics":["bioasq","nlp","question-answering","question-generator"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2302.01691","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-03T15:41:55.000Z","updated_at":"2025-01-13T18:06:24.000Z","dependencies_parsed_at":"2024-11-23T01:41:34.459Z","dependency_job_id":null,"html_url":"https://github.com/dmis-lab/LIQUID","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmis-lab/LIQUID","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FLIQUID","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FLIQUID/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FLIQUID/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FLIQUID/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/LIQUID/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FLIQUID/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264248024,"owners_count":23579060,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioasq","nlp","question-answering","question-generator"],"created_at":"2024-11-23T01:41:30.727Z","updated_at":"2025-07-08T10:11:43.642Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LIQUID\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/liquid-a-framework-for-list-question/question-answering-on-multispanqa)](https://paperswithcode.com/sota/question-answering-on-multispanqa?p=liquid-a-framework-for-list-question) \u003cbr\u003e\nThis is the official repository for the paper \"[**LIQUID: A Framework for List Question Answering Dataset Generation**](https://arxiv.org/abs/2302.01691)\" (presented at [***AAAI 2023***](https://aaai.org/Conferences/AAAI-23/)). This repository provides the implementation of the LIQUID model, guidelines on how to run the model to synthesize list QA data. Also, you can download generated datasets without having to create them from scratch (see **[here](#data-downloads)**).\n\n\n## Quick Links\n\n* [Overview](#overview)\n* [Data Downloads](#data-downloads)\n* [Requirements](#requirements)\n* [Dataset Generation](#dataset-generation)\n* [List Question Answering](#list-question-answering)\n* [Reference](#reference)\n* [Contact](#contact)\n\n## Overview\n\nLIQUID is an automated framework for generating list QA datasets from unlabeled corpora. Generated datasets by LIQUID can be used to improve list QA performance by supplementing insufficient human-labeled data. When training a list QA model using the generated data and then fine-tuning it on the target training data, we achieved a new **state-of-the-art** performance on **[MultiSpanQA](https://multi-span.github.io/)** and outperformed baselines on several benchmakrs including **[Quoref](https://arxiv.org/abs/1908.05803)** and **[BioASQ](http://bioasq.org/)**.\n\n![Model-1](https://user-images.githubusercontent.com/72010172/185115620-3dbd69cd-5e37-4da0-acd6-e9dfb9ab0021.png)\n\nLIQUID comprises the following four stages (please refer to **[our paper](https://arxiv.org/abs/2302.01691)** for details).\n\n* (1) Answer extraction: the named entities belonging to the same entity type (e.g., organization type) in a summary are extracted by an NER model and used as candidate answers. \n* (2) Question generation: the candidate answers and the original passage are fed into a QG model to generate list questions. \n* (3) Iterative filtering: incorrect answers (e.g., Hanszen) are iteratively filtered based on the confidence score assigned by a QA model. \n* (4) Answer expansion: correct but omitted answers (e.g., Yale) are identified by the QA model.\n\n## Data Downloads\n\nUse the links below to download the synthetic datasets without having to create a dataset from scratch. ✶ indicates they are the same data used in our experiments. Our data format follows that of SQuAD-v1.1.\n\n|              Name              | Corpus | Size | Link |\n|:----------------------------------|:--------|:--------|:--------|\n| liquid-wiki-140k (✶) | Wikipedia | 140k | http://nlp.dmis.korea.edu/projects/liquid-lee-et-al-2023/liquid-wiki-140k.json |\n| liquid-pubmed-140k (✶) | PubMed | 140k | http://nlp.dmis.korea.edu/projects/liquid-lee-et-al-2023/liquid-pubmed-140k.json |\n\n## Requirements\n\nDownload this repository and set up an environment as follows.\n\n```bash\n# Clone the repository\ngit clone https://github.com/sylee0520/LIQUID.git\ncd LIQUID\n\n# Create a conda virtual environment\nconda create -n liquid python=3.8\nconda activate liquid\n\n# Install all requirements\npip install -r requirements.txt\n```\n\n### Unlabeled Corpus\n\nDownload an unlabeled source corpus to be annotated and extract/unpack it to the correct directory. Choose either Wikipedia or PubMed depending on your target domain. ✶ indicates they are the same data used in our experiments.\n\n|              Description              | Directory | Link |\n|:----------------------------------|:--------|:--------|\n| 2018-12-20 version of **Wikipedia** (✶) | `./data/unlabeled/wiki/` | http://nlp.dmis.korea.edu/projects/liquid-lee-et-al-2023/wiki181220.zip |\n| 2019-01-02 version of **PubMed** (✶) | `./data/unlabeled/pubmed/` | http://nlp.dmis.korea.edu/projects/liquid-lee-et-al-2023/pubmed190102.zip |\n\nNote that passages in each file have not been shuffled. You will have to randomly sample passages from the entire corpus files (e.g., \"0000.json\" to \"5620.json\" for Wikipedia) if you want to use sampled passages.\n\n### NER Models\n\nIn LIQUID, two types of NER models are used to extract candidate answers for the *general* and *biomedical* domains, respectively. Please refer to the instructions below to install the NER models.\n\n* For the *general* domain, run `python -m spacy download en_core_web_sm` to install **spaCy** NER system.\n* For the *biomedican* domain, install **BERN2** from the official GitHub repository (**[link](https://github.com/dmis-lab/BERN2)**). After installation is complete, refer to the instructions below and run the model in the background. Note that you need to create a new conda environment for BERN2, instead of reusing the environment for LIQUID.\n\n```bash\n# Run BERN2 model\nexport CUDA_VISIBLE_DEVICES=0\nconda activate BERN2\ncd BERN2/scripts\n\n# For Linux and MacOS\nbash run_bern2.sh\n\n# For Windows\nbash run_bern2_windows.sh\n```\n\n## Dataset Generation\n\nOnce you have installed all the requirements, you are ready to create your list QA datasets. Please see the example script below. \n\n```bash\nexport CUDA_VISIBLE_DEVICES=0\nexport DATA_FILE=./data/unlabeled/wiki/0000.json\nexport OUTPUT_FILE=./data/synthetic/wiki/0000.json\npython generate.py \\\n    \t--data_file ${DATA_FILE} \\\n    \t--output_file ${OUTPUT_FILE} \\\n    \t--batch_size 8 \\\n    \t--summary_min_length 64 \\\n    \t--summary_max_length 128 \\\n    \t--summary_model_name_or_path facebook/bart-large-cnn \\\n    \t--qg_min_length 64 \\\n    \t--qg_max_length 128 \\\n    \t--qg_model_name_or_path mrm8488/t5-base-finetuned-question-generation-ap \\\n    \t--qa_model_name_or_path thatdramebaazguy/roberta-base-squad \\\n    \t--do_summary \\\n    \t--device 0\n```\n\n### Argument Description\n- `batch_size`: Number of passages to process simultaneously in one batch.\n- `summary_min_length`, `summary_max_length`, `qg_min_length`, and `qg_max_length`: Minimum and maximum lengths of the output summary and question, respectively.\n- `summary_model_name_or_path`, `qg_model_name_or_path`, and `qa_model_name_or_path`: Model path for loading the summarization model, question-generation model, and question-answering model, respectively. For the *biomedical* domain, you can use `dmis-lab/biobert-base-cased-v1.1-squad` as the QA model.\n- `is_biomedical`: Use this option when the target domain is biomedicine.\n- `do_summary`: (**Recommended**) Use this option if you want to summarize input passages and extract candidate answers from the summaries.\n- `device`: Set to `0` if you want to use our framework on GPU; otherwise `-1`.\n\n## List Question Answering\n\nTo be updated soon.\n\n\n## Reference\nPlease cite our paper if it is helpful or relevant to your work.\n\n```bash\n@article{lee2023liquid,\n  title={LIQUID: A Framework for List Question Answering Dataset Generation},\n  author={Lee, Seongyun and Kim, Hyunjae and Kang, Jaewoo},\n  journal={arXiv preprint arXiv:2302.01691},\n  year={2023}\n}\n```\n\n## Contact\nFeel free to email us (`sy-lee@korea.ac.kr` and `hyunjae-kim@korea.ac.kr`) if you have any!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fliquid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fliquid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fliquid/lists"}