{"id":21441737,"url":"https://github.com/dmis-lab/seqtagqa","last_synced_at":"2025-07-14T17:32:00.960Z","repository":{"id":43049949,"uuid":"473028772","full_name":"dmis-lab/SeqTagQA","owner":"dmis-lab","description":"Sequence Tagging for Biomedical Extractive Question Answering (Bioinformatics'2020)","archived":false,"fork":false,"pushed_at":"2023-07-03T17:02:11.000Z","size":73,"stargazers_count":11,"open_issues_count":2,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-05-14T00:23:20.522Z","etag":null,"topics":["bioasq","question-answering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-23T03:53:26.000Z","updated_at":"2023-12-06T03:10:14.000Z","dependencies_parsed_at":"2022-07-09T06:30:19.292Z","dependency_job_id":null,"html_url":"https://github.com/dmis-lab/SeqTagQA","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSeqTagQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSeqTagQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSeqTagQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSeqTagQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/SeqTagQA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225990493,"owners_count":17556152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioasq","question-answering"],"created_at":"2024-11-23T01:41:21.941Z","updated_at":"2024-11-23T01:41:22.347Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sequence Tagging for Biomedical Extractive Question Answering\n\nSource codes and resources for [Sequence Tagging for Biomedical Extractive Question Answering.](https://arxiv.org/abs/2104.07535)\n\nPlease cite:\n ```bib\n @article{yoon-etal-2022-sequence,\n    author = {Yoon, Wonjin and Jackson, Richard and Lagerberg, Aron and Kang, Jaewoo},\n    title = \"{Sequence tagging for biomedical extractive question answering}\",\n    journal = {Bioinformatics},\n    year = {2022},\n    month = {06},\n    issn = {1367-4803},\n    doi = {10.1093/bioinformatics/btac397},\n}\n ```\n```\nWonjin Yoon, Richard Jackson, Aron Lagerberg, Jaewoo Kang, Sequence tagging for biomedical extractive question answering, Bioinformatics, 2022;, btac397, https://doi.org/10.1093/bioinformatics/btac397\n```\n## _Naturally posed_ biomedical questions\nIn the paper, we investigated the characteristics of naturally posed biomedical questions as a preliminary study. \nAs a wish to explore the nature of biomedical question answering with the community, we made the collection of biomedical questions available for download in this repository.\n\n[**See the online preview of the questions here**](https://1drv.ms/x/s!AjwviG8mocn7hbp16IRH8ICJQKcTXw?e=uYwXiV)\n\nThe questions are from _Clinical Questions Collection (CQC)_ (D’Alessandro et al., 2004; Ely et al., 1999, 1997) and _PubMed queries_ (Herskovic et al., 2007), with **our _query screening algorithm_ (Please check [Section 3 of our paper](https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac397/6609766#364116020) and [appendix](https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac397/6609766#supplementary-data)).** \n\nFrom our analysis in the sampled question subset, (where an equal number of questions are randomly sampled from two sources of questions), about half of the questions are \"Others\", which means they are not answerable with current settings. \nThe resons can be various including: \n* One or more conditions are missing: (as suggested in the [ConditionalQA (Sun et al., 2021)](https://arxiv.org/abs/2110.06884))\n* The question is too abstractive or requires a long answer that cannot be extracted from a paragraph: (ex. \"what is the history of calcium?\" or \"what can i do?\") \n* Incompleteness in our query screening algorithm: (ex. \"who diagnostic criteria\" or \"Will and Joondeph\")\n\n#### I am interested in exploring the characteristics of biomedical questions or building QA datasets of naturally posed biomedical questions. (As a future work)\nThe work on the characteristics of naturally posed biomedical questions in this paper is a preliminary study and I think there is huge room for improvement. \nAny researchers who want to collaborate with me on this topic are welcome! To discuss, please contact me: `wonjin.info (_at_) gmail.com`. If you are planning to attend the 10th BioASQ (2022), we can discuss it there!\n\n-----\n\n## How to train/evaluate\n\n#### Note: We recently uploaded a pre-release version of pre-processing code under `preprocessing` folder. \n\nFor datasets (including biomedical questions), please download them from: [here](https://drive.google.com/file/d/1m0GnVlKqvUHfDdpZ9KDDor5EiIqhPwp3/view?usp=sharing)\n\u003cbr\u003eTo use the BioASQ dataset, you need to register in [the BioASQ website](http://participants-area.bioasq.org/general_information/general_information_registration/) which authorizes the use of the dataset. \n\nFor example training and evaluation bash script: please see [`train_list_example.sh`](./train_list_example.sh)\n\n### Detailed instructions\n#### Requirements and Installation\nWe tested our codes using python3.8 on a GPU-enabled Linux (Ubuntu 18.04) machine.\n\u003cbr\u003ePlease install pytorch(\u003e=1.7.1, \u003c2.0) from https://pytorch.org/get-started/previous-versions/ before following this section.\n\n```bash\npip install --use-feature=2020-resolver -r requirements.txt\n```\nNote that pip updated dependency resolver on Oct 2020: [Announcement](https://discuss.python.org/t/announcement-pip-20-2-release/4863). Please include `--use-feature=2020-resolver` to make the installation work. \n\n#### Training\nThe following instructions are example scripts for training LM on the BioASQ8b dataset.\n\nIn the first part, we set environmental variables.\n(We assume that you downloaded and decompressed the datasets in `$HOME/DATA` )\n```bash\nexport SEED=0\nexport VERSION=8\n\n# To train LMs on List-question dataset\nexport LANGMODEL_DIR=\u003cPath to BioLM: it should be trained on SQuAD\u003e\nexport DATA_DIR=$HOME/DATA/BioASQ-SeqTagQA/${VERSION}b-list-20201030\nexport MAX_EPOCH=400\nexport LEARN_RATE=5e-6\nexport OUTPUT_DIR=bioasq8b-list\nmkdir $OUTPUT_DIR\n```\n\nThe following lines will run the training of the model.:\n```bash\npython run_eqa.py --model_struct=linear \\\n --do_train=true --do_eval=true --do_predict=false \\\n --model_name_or_path=$LANGMODEL_DIR \\\n --per_device_train_batch_size=18 --max_seq_length=512 \\\n --train_file=${DATA_DIR}/train_dev.tsv --validation_file=${DATA_DIR}/test.tsv --test_file=${DATA_DIR}/test.tsv \\\n --output_dir=$OUTPUT_DIR --save_total_limit=100 \\\n --learning_rate=${LEARN_RATE} --seed=${SEED} \\\n --num_train_epochs=${MAX_EPOCH} \\\n --save_steps=4000\n```\n\n#### Testing\nThen select the latest checkpoint and predict the test dataset using the checkpoint.\n```bash\nexport CKPT=84000\nexport SUB_OUTPUT_DIR=${OUTPUT_DIR}/checkpoint-${CKPT}\n\npython run_eqa.py --model_struct=linear \\\n  --do_train=false --do_eval=false --do_predict=true \\\n  --model_name_or_path=$SUB_OUTPUT_DIR \\\n  --per_device_train_batch_size=18 --per_device_eval_batch_size=32 --max_seq_length=512 \\\n  --test_file=${DATA_DIR}/test.tsv \\\n  --output_dir=$SUB_OUTPUT_DIR --save_total_limit=50 \\\n  --learning_rate=${LEARN_RATE} --seed=${SEED} \\\n  --save_steps=4000\n```\nPlease note that the parameter `model_name_or_path` is changed to the selected checkpoint.\n\nThen detokenize the output and transform it to BioASQ format.\n```bash\n# To Detokenize the output\npython detokenize-list.py \\\n  --test_path=${DATA_DIR}/test.tsv \\\n  --predictions_path=${SUB_OUTPUT_DIR}/predictions.txt \\\n  --original_test_path=${DATA_DIR}/original_test.json \\\n  --output_dir=${SUB_OUTPUT_DIR}\n```\n\nFinally, evaluate the output with the official BioASQ evaluation library. Please download it from [here](https://github.com/BioASQ/Evaluation-Measures). The library requires java.\n```bash\n# Example script of using official BioASQ evaluation library\nexport EVAL_LIB=\u003cCloned official eval code path\u003e/Evaluation-Measures\njava -Xmx10G -cp $CLASSPATH:$EVAL_LIB/flat/BioASQEvaluation/dist/BioASQEvaluation.jar evaluation.EvaluatorTask1b -phaseB -e 5 \\\n  $HOME/DATA/BioASQ-golden/${VERSION}B_total_golden.json ${SUB_OUTPUT_DIR}/NER_result_BioASQ.json \u003e\u003e $OUTPUT_DIR/total_BioASQ_eval.log\n```\n\n-----\n\nFor inquiries, please contact `wonjin.info (_at_) gmail.com`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fseqtagqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fseqtagqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fseqtagqa/lists"}