{"id":21441727,"url":"https://github.com/dmis-lab/kazu-ner-module","last_synced_at":"2025-09-03T12:36:48.978Z","repository":{"id":63613806,"uuid":"557763069","full_name":"dmis-lab/KAZU-NER-module","owner":"dmis-lab","description":"EMNLP 2022: Biomedical NER for the Enterprise with Distillated BERN2 and the Kazu Framework","archived":false,"fork":false,"pushed_at":"2024-08-29T05:38:36.000Z","size":111,"stargazers_count":11,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-17T23:04:41.770Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-26T08:58:34.000Z","updated_at":"2025-05-21T17:05:19.000Z","dependencies_parsed_at":"2024-08-20T22:46:27.987Z","dependency_job_id":"a4e67c64-1470-4492-9be6-db0d6c453789","html_url":"https://github.com/dmis-lab/KAZU-NER-module","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmis-lab/KAZU-NER-module","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FKAZU-NER-module","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FKAZU-NER-module/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FKAZU-NER-module/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FKAZU-NER-module/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/KAZU-NER-module/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FKAZU-NER-module/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273445246,"owners_count":25107148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T01:41:19.328Z","updated_at":"2025-09-03T12:36:48.900Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Multi-label token classification (using soft-label settings) - Modeling for NER task\n### Appendix code for: [Biomedical NER for the Enterprise with Distillated BERN2 and the Kazu Framework](https://aclanthology.org/2022.emnlp-industry.63) (EMNLP 2022)\n\n\u003cbr\u003e\n\nThis repository presents the train and evaluation codes for the NER module used in the initial release of the KAZU (Korea University and AstraZeneca) framework.\n* For the framework, please visit https://github.com/AstraZeneca/KAZU\n* For web demo, please visit http://kazu.korea.ac.kr\n* For details about the model, please see our paper entitled [`Biomedical NER for the Enterprise with Distillated BERN2 and the Kazu Framework (EMNLP 2022 Industry track)`](https://aclanthology.org/2022.emnlp-industry.63).\n\nOur models are available on Hugging Face framework: \n* [**KAZU-NER-module-distil-v1.0**](https://huggingface.co/dmis-lab/KAZU-NER-module-distil-v1.0): NER module for KAZU framework. Denoted as TinyBERN2 model in the paper. \n* [**Tiny-sized PubMedBERT-v1.0**](https://huggingface.co/dmis-lab/TinyPubMedBERT-v1.0): Distillated [PubMedBERT (Gu et al., 2021)](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract).\n* [**Tiny-sized SapBERT-v1.0**](https://huggingface.co/dmis-lab/TinySapBERT-from-TinyPubMedBERT-v1.0): Tiny-sized (4-layer) biomedical entity representations (language model). Training of this weights are initiated from TinyPubMedBERT-v1.0 and trained using [SapBERT (Liu et al., 2021)](https://github.com/cambridgeltl/sapbert) scheme.\n\nWe applied multi-label token classification settings for NER task to mitigate the problem from _nested entity_ recognition (Please check `Section 3.1 Model Architecture` of the paper).\n\n### Citation info\nJoint-first authorship of **Richard Jackson** and **WonJin Yoon**.\n\u003cbr\u003ePlease cite the paper using the simplified citation format provided in the following section, or find the [full citation information here](https://aclanthology.org/2022.emnlp-industry.63.bib)\n```\n@inproceedings{YoonAndJackson2022BiomedicalNER,\n  title=\"Biomedical {NER} for the Enterprise with Distillated {BERN}2 and the Kazu Framework\",\n  author=\"Yoon, Wonjin and Jackson, Richard and Ford, Elliot and Poroshin, Vladimir and Kang, Jaewoo\",\n  booktitle=\"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track\",\n  month = dec,\n  year = \"2022\",    \n  address = \"Abu Dhabi, UAE\",\n  publisher = \"Association for Computational Linguistics\",\n  url = \"https://aclanthology.org/2022.emnlp-industry.63\",\n  pages = \"619--626\",\n}\n```\nWe would like to recommend you to additionally cite [PubMedBERT (Gu et al., 2021)](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract) or [SapBERT (Liu et al., 2021)](https://github.com/cambridgeltl/sapbert) if you use our release of a distillated version of such models.\n\u003chr\u003e\n\n## How to train or evaluate a model.\n\n### Table of Contents\n- [Requirements](#requirements)\n- [Example codes](#example-codes)\n- [Input data format](#input-data-format)\n- [Dataset preparation](#dataset-preparation)\n- [How to eval KAZU\\-NER model](#how-to-eval-kazu-ner-model)\n- [How to train your own model using the code](#how-to-train-your-own-model-using-the-code-multi-label-ner-setting)\n- [Known issues](#known-issues)\n- [Contact information](#contact-information)\n\n### Requirements\n\n* \u003cb\u003ePlease install PyTorch before proceeding with the following steps.\u003c/b\u003e\n* You can install libraries using `pip install -r requirements.txt`.\n* Using Python v3.7.13 is strongly suggested for compatibility.\n\n#### Troubleshooting Installation Errors\nIf you encounter the error `error: can't find Rust compiler` while installing transformers, this may related to your python version.\nAs noted in [this comment](https://github.com/huggingface/transformers/issues/2831#issuecomment-2107017678), older version of tokenizers are not compatible with newer version of python.\n* \u003cb\u003eRecommended Solution:\u003c/b\u003e Using Python v3.7.13 is suggested for compatibility.\n* \u003cb\u003eAlternative Solution (Not Recommended):\u003c/b\u003e Alternatively, you can use Python v3.10.12 and install the libraries using the following command:\n```bash\n# Tested torch version: torch==2.1.0 CUDA 12.1\n#pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121\n\npip install transformers==4.16.2 tokenizers==0.12 datasets==1.18.3 seqeval==1.2.2\n```\n\n\u003cb\u003eTested version:\u003c/b\u003e Codes are tested using Python v3.7.13 and following libraries. \n```\ntorch==1.8.2, transformers==4.9.2, datasets==1.18.3, seqeval\u003e=1.2.2\n```\n\n### Example codes\n(Example CLI codes are in `example_run_ner.sh`.)\n\nThe following steps will provide a simple tutorial on how to produce predictions (and checkpoints if you are trying to train a model) in `${OUTPUT_DIR}`. All the codes are written in Linux bash script and tested on Ubuntu. \n\n### Input data format\n\n![image](https://user-images.githubusercontent.com/13089179/202068143-315bd182-872b-4a1b-b495-efae957d7c82.png)\nThe above image shows an example of input data. The example sentences are from test datasets of the NCBI-disease corpus [(Doğan et al., 2014)](https://pubmed.ncbi.nlm.nih.gov/24393765/) and we appended automatic annotations from predictions of one of our models.\n* Input data should be in tsv format **without a row for column names. In the image, column names are inserted to facilitate explanation**. \n* Analogous to the CoNLL format, each line includes a token (usually a word delimited by blank spaces or special characters: e.g. line 15-17) and annotation information for the token. \n* The datapoints (i.e. the sentences) are separated by a blank line (line 20 in the example).\n* The first column refers to tokens, and from the second to the column just before the last one indicates the probability of the token class, in the order of `labels.txt` that should be provided with the model (e.g. see [here](https://huggingface.co/dmis-lab/KAZU-NER-module-distil-v1.0/blob/main/labels.txt) for the KAZU-NER model). \n* The last column indicates a token class with the highest probability **which is for reference only and not used for the training**.\n* We provide scripts for the transformation from CoNLL format. Please see the next section (**Dataset preparation**). For test/inference time, if you do not have true labels, you can put `O` for all tokens, and the model will provide its prediction. Please be informed that in this case, you cannot evaluate the model with your input data. \n\nAs you can see in line number 8-11, this input format can train the model with multiple classes for a token. \n\u003cbr\u003eIn the example, `ataxia-telangiectasia` is indicated as a disease type entity and at the same time `ataxia-telangiectasia gene` is a gene type entity. (We apply a threshold value of 0.5 throughout the repo)\n\n### Dataset preparation\n\n\u003cb\u003eIn this tutorial, we demonstrate how to evaluate our model [(KAZU-NER-module-distil-v1.0)](https://huggingface.co/dmis-lab/KAZU-NER-module-distil-v1.0), which has already been trained on our dataset (as specified in the paper). \nThe example scripts are provided for `test.tsv` only. Please process `train.tsv` and `dev.tsv` (or their equivalents) if you need the train or dev splits. You will need to modify the flags to train the model on your own dataset.\u003c/b\u003e\n\nFirst, collect CoNLL format datasets. \n\u003cbr\u003eIn this tutorial, we will download pre-processed BC5CDR benchmark dataset from [cambridgeltl/MTL-Bioinformatics-2016](https://github.com/cambridgeltl/MTL-Bioinformatics-2016/blob/master/data/BC5CDR-IOB) repository.\n```bash\nexport DATA_DIR=${HOME}/KAZU-NER-exp/BC5CDR_test # Please use absolute path to avoid some unexpected errors \nmkdir -p ${DATA_DIR}\nwget -O ${DATA_DIR}/test.tsv https://github.com/cambridgeltl/MTL-Bioinformatics-2016/raw/master/data/BC5CDR-IOB/test.tsv\n```\n\nFor the inference (eval time), the model is required to use `labels.txt` identical to the one used for training. Please download `labels.txt` from [here](https://huggingface.co/dmis-lab/KAZU-NER-module-distil-v1.0/resolve/main/labels.txt) and place it in `${DATA_DIR}` folder with train or test files.\n```\nwget -O ${DATA_DIR}/labels.txt https://huggingface.co/dmis-lab/KAZU-NER-module-distil-v1.0/resolve/main/labels.txt\n```\n\n**Important:** please make it sure that all the tags in dataset (`test.tsv`) should be in `labels.txt` (case-sensitive). If not, we suggest to modify dataset to match tags in labels.txt. You can replace some non-supported tags (entity types) with `O` tags. \n\u003cbr\u003eFor example the following bash script can alter the tags in the dataset (works in sed (GNU sed) 4.2.2 - in some sed versions this solution may not work). Alternativly, we suggest you to write a simple python code to change tags.\n```bash\n#sed -i 's/search_string/replace_string/' filename\nsed -i 's/\\tB-Disease/\\tB-disease/' ${DATA_DIR}/test.tsv\nsed -i 's/\\tI-Disease/\\tI-disease/' ${DATA_DIR}/test.tsv\nsed -i 's/\\tB-Chemical/\\tB-drug/' ${DATA_DIR}/test.tsv\nsed -i 's/\\tI-Chemical/\\tI-drug/' ${DATA_DIR}/test.tsv\n```\n\nFinal step of dataset preparation is to transform conll format dataset to `*.prob_conll` format. `label2prob.py` is a simple script to change the format.\n```bash\nexport IS_IO=\"\" # set this for using IO taggings.\npython label2prob.py --label ${DATA_DIR}/labels.txt  --file_path ${DATA_DIR}/test.tsv --output_path ${DATA_DIR}/test.prob_conll ${IS_IO}\n```\nCheck that the final product has the same number of lines as the original dataset (unless you saw `n` duplicated empty lines from the stdout message of the previous step).\n```bash\nwc -l ${DATA_DIR}/*\n```\n\nPlease prepare dev set using the same procedure. \n\n### How to eval KAZU-NER model\n\n```bash\nexport BATCH_SIZE=256\nexport GRAD_ACCUMLATION=4 \nexport LEARN_RATE=3e-5\nexport SAVE_STEPS=100\n\n# Create basic folders\nexport CACHE_DIR=_tmp/cache\nrm -rf ${CACHE_DIR}\nmkdir -p ${CACHE_DIR}\nexport OUTPUT_DIR=_tmp/output/MultiLabelNER-test\nmkdir -p ${OUTPUT_DIR}\n\n# To eval (without training)\nexport BERT_MODEL=\"dmis-lab/KAZU-NER-module-distil-v1.0\"\n\npython3 run_ner.py \\\n --model_name_or_path $BERT_MODEL \\\n --max_length 128 \\\n --do_eval --validation_file ${DATA_DIR}/dev.prob_conll \\\n --evaluation_strategy steps --eval_steps 100 \\\n --per_device_eval_batch_size 256 \\\n --do_predict --test_file ${DATA_DIR}/test.prob_conll \\\n --cache_dir ${CACHE_DIR} \\\n --preprocessing_num_workers 8 \\\n --output_dir ${OUTPUT_DIR} \\\n --overwrite_output_dir \\\n --save_steps $SAVE_STEPS --save_total_limit 50 \\\n --return_entity_level_metrics \\\n --use_probs\n```\n\nYou will see evaluation results from stdout:\n```\n### Eval results:\n# cell_type   : precision: 0.0000, recall: 0.0000, f1: 0.0000, number: 0.0000, accuracy: 0.9967\n# drug        : precision: 0.9452, recall: 0.8655, f1: 0.9036, number: 5383.0000, accuracy: 0.9892\n# disease     : precision: 0.7691, recall: 0.6625, f1: 0.7118, number: 4424.0000, accuracy: 0.9744\n# gene        : precision: 0.0000, recall: 0.0000, f1: 0.0000, number: 0.0000, accuracy: 0.9810\n# cell_line   : precision: 0.0000, recall: 0.0000, f1: 0.0000, number: 0.0000, accuracy: 0.9984\n# species     : precision: 0.0000, recall: 0.0000, f1: 0.0000, number: 0.0000, accuracy: 0.9945\n```\nThe predictions (labels for tokens) are written in `${OUTPUT_DIR}/predictions.txt`.\n\n### How to train your own model using the code (multi-label NER setting)\n```bash\nexport BATCH_SIZE=256\nexport GRAD_ACCUMLATION=4 \nexport LEARN_RATE=3e-5\nexport SAVE_STEPS=100\n\n# Create basic folders\nexport CACHE_DIR=_tmp/cache\nrm -rf ${CACHE_DIR}\nmkdir -p ${CACHE_DIR}\nexport OUTPUT_DIR=_tmp/output/MultiLabelNER-test\nmkdir -p ${OUTPUT_DIR}\n\n# To train\nexport BERT_MODEL=\"microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract\" # or one of our distillated models\n#export BERT_MODEL=\"TinyPubMedBERT\" # an example of our distillated model\n\npython3 run_ner.py \\\n --model_name_or_path $BERT_MODEL \\\n --max_length 128 \\\n --do_train --train_file ${DATA_DIR}/train.prob_conll \\\n --learning_rate ${LEARN_RATE} --num_train_epochs 10 \\\n --per_device_train_batch_size ${BATCH_SIZE} \\\n --gradient_accumulation_steps $GRAD_ACCUMLATION \\\n --do_eval --validation_file ${DATA_DIR}/dev.prob_conll \\\n --evaluation_strategy steps --eval_steps 100 \\\n --per_device_eval_batch_size 256 \\\n --do_predict --test_file ${DATA_DIR}/test.prob_conll \\\n --cache_dir ${CACHE_DIR} \\\n --preprocessing_num_workers 8 \\\n --output_dir ${OUTPUT_DIR} \\\n --overwrite_output_dir \\\n --save_steps $SAVE_STEPS --save_total_limit 50 \\\n --return_entity_level_metrics \\\n --use_probs\n\n```\n\n### Known issues\n\n* **FileNotFoundError: Unable to find '\\\u003cFILE\\\u003e' at \\\u003cLOCATION\\\u003e/KAZU-NER-module/prob_conll** :\n  \u003cbr\u003eThis error occurs when one of `--train_file`, `--validation_file`, or `--test_file` is missing from the location you pass through the command line argument.\nThere is a minor issue about error log especially about the dataset folder path and we are working on this. The error message is sometimes not directly related to the error. \nWe suggest you to double check whether the required datasets (i.e. the location you pass to the script) are actully exist in the location (from our example codes,  ${DATA_DIR} folder).\n\n\n### Contact information\nFor help or issues using the codes or model (NER module of KAZU) in this repository, please contact WonJin Yoon (`wonjin.info (at) gmail.com`) or submit a GitHub issue.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fkazu-ner-module","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fkazu-ner-module","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fkazu-ner-module/lists"}