{"id":19520159,"url":"https://github.com/osu-nlp-group/bio-tokenization","last_synced_at":"2025-10-14T13:13:00.806Z","repository":{"id":179289860,"uuid":"663095369","full_name":"OSU-NLP-Group/Bio-Tokenization","owner":"OSU-NLP-Group","description":"Biomedical LMs are Robust to Sub-optimal Tokenization","archived":false,"fork":false,"pushed_at":"2023-07-08T18:44:29.000Z","size":6405,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-31T02:13:33.134Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2306.17649","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-06T14:39:36.000Z","updated_at":"2023-10-18T14:11:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"6b8cb2b9-d174-4fff-bcd9-d0f7819574f1","html_url":"https://github.com/OSU-NLP-Group/Bio-Tokenization","commit_stats":null,"previous_names":["osu-nlp-group/bio-tokenization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OSU-NLP-Group/Bio-Tokenization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FBio-Tokenization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FBio-Tokenization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FBio-Tokenization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FBio-Tokenization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/Bio-Tokenization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FBio-Tokenization/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018706,"owners_count":26086605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:06.766Z","updated_at":"2025-10-14T13:13:00.769Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Biomedical Language Models are Robust to Sub-optimal Tokenization\n\n\\\n**Our Paper:**\n- [Biomedical Language Models are Robust to Sub-optimal Tokenization](https://arxiv.org/abs/2306.17649) \u003cbr\u003eBernal Jiménez Gutiérrez, Huan Sun, Yu Su \u003cbr\u003e BioNLP @ ACL 2023 \n\n\n**Available HuggingFace Models:**\n- [BioVocabBERT](https://huggingface.co/osunlp/BioVocabBERT)\n- [PubMedBERT_Replica](https://huggingface.co/osunlp/PubMedBERT_Replica)\n\n## Data\n\n### SIGMORPHON Data\n\nThe datasets from the [SIGMORPHON 2022 Word-Level Morphological Segmentation Shared Task](https://github.com/sigmorphon/2022SegmentationST) \nthat are relevant for our work are added to this repository directly as follows. \n\n```\ndata/eng.word.train.tsv\ndata/eng.word.dev.tsv\n```\n\n### UMLS\n\nDue to the UMLS license, we are unable to distribute the MRCONSO.RRF file necessary for our work. After creating a UMLS account, \nyou can download the necessary UMLS version (we use the UMLS 2022AB) from this [archive](https://www.nlm.nih.gov/research/umls/licensedcontent/umlsarchives04.html). \nFor reproducibility, add the MRCONSO.RRF file under the ```data``` directory before running any scripts.\n\n### PubMed Pretraining\n\nWe use a previously processed set of PubMed abstracts made available by [BlueBERT](https://github.com/ncbi-nlp/bluebert/tree/master) from \nthis [link](https://ftp.ncbi.nlm.nih.gov/pub/lu/Suppl/NCBI-BERT/pubmed_uncased_sentence_nltk.txt.tar.gz). Download this corpus and \nadd it to the ```data``` directory before starting.\n\n### Downstream Tasks\n\nAll data for downstream performance evaluation can be obtained through the repositories used for evaluation. As explained in\nmore detail below, we refer to people interested in reproducing downstream results to those repositories for \n[NER](https://github.com/dki-lab/few-shot-bioIE/tree/main) \nand [entity linking](https://github.com/cambridgeltl/sapbert).\n\n## Installation\n\nRun the following commands to create a conda environment with the required packages. \n\n```\nconda create -n bio-tok python=3.11 pip\nconda activate bio-tok\npip install -r requirements.txt\n```\n\n## Evaluating Tokenization Quality\n\nWe use morphological segmentation as a way to evaluate how well-aligned biomedical tokenizers are to the segmentation \njudgements of biomedical expert.\n\n### Load UMLS Phrases\n\nWe first load biomedical phrases from the UMLS that are in English as well as not suppressed by the UMLS in order to get a \nhigher quality set of biomedical phrases. We run the ```src/parse_umls_file.ipynb``` Jupyter notebook to carry out this process. \nRun this notebook **only after** obtaining the MRCONSO.RRF file mentioned above.\n\n### Create Biomedical Subset of SIGMORPHON 2022\n\nNow that we have a higher quality set of biomedical phrases, we first extract biomedical terms and intersect them with the \ndevelopment SIGMORPHON 2022 dataset mentioned above in order to evaluation biomedical morphological segmentation quality \nspecifically. The Jupyter notebook ```src/create_biomedical_sigmorphon.ipynb``` carries out \nthis process and creates the file ```data/dev_bio_words.tsv``` which holds the biomedical subset used for evaluation.\n\n### Standard Tokenizer Evaluation\n\nWe use the evaluation metrics used by the SIGMORPHON 2022 Shared Task to evaluate standard and new tokenizers. The Jupyter \nnotebook ```src/evaluate_tokenizers_sigmorphon_task.ipynb``` uses the biomedical subset created above to assess tokenizer quality.\n\n**NOTE:** The final part of this notebook requires the fine-tuned CANINE model created in the following section. Return to this\nnotebook after you have created the supervised model to reproduce the results in our paper.\n\n## Creating a Well-Aligned Tokenizer \n\nGiven the poor performance of standard tokenizer on biomedical morphological segmentation, we try to create a new, better aligned\ntokenizer.\n\n### Fine-Tuning a Character LM for Segmentation\n\nWe first fine-tune a character-based language model using the SIGMORPHON 2022 word-level segmentation training set. To this we \nrun the following command:\n\n```\nCUDA_VISIBLE_DEVICES=0 python src/train_char_lm_sigmorphon.py\n```\n\nThis script will fine-tune a CANINE model and create a strong supervised segmentation model. The resulting models will be \nsaved under ```output/canine_exps```.\n\n### Constructing BioVocabBERT\n\nNow that we have a strong biomedical segmentation model, we use it to create a LM friendly tokenizer. Vocabulary free tokenizers\nlike our CANINE model have strong downsides such as needing a large amount of tokens for training and an out-of-vocabulary token \nfor inference which could hinder a LM's generalizability. We therefore create a high quality and high coverage vocabulary of \nbiomedical tokens using the following steps:\n1) CANINE segments all biomedical terms in UMLS.\n2) Filter only tokens which happen more than once.\n3) Add back the tokens from the BERT tokenizer to cover general domain data as well.\n4) Create BioVocabBERT!\n\nWe carry out these steps in the Jupyter notebook ```Create BioVocabBERT Tokenizer.ipynb```. The tokenizer will be saved as\n```../output/biovocabbert_tokenizer```.\n\n## Language Model Pre-Training\n\nOur pre-training procedure and code is closely based on the [academic-budget-bert](https://github.com/IntelLabs/academic-budget-bert/tree/main) code-base. \nIt allowed us to accelerate the pre-training process considerably. Check out their code and paper for more information on \nthe techniques that enable this speedup.\n\n### Creating the Pre-Training Data\n\nBefore starting the pre-training process, make sure that you have downloaded the corpus mentioned in the above [data](#Data) section\nand have created the biovocabbert tokenizer.\n\nWe follow the original repository's procedure by running the following commands.\nIt first shards the pre-training corpus and then generates samples for both PubMedBERT and BioVocabBERT tokenizers.\nSamples must be generated for the tokenizer of each LM you would like to pre-train. \n\n```\nbash shard_data.sh\nbash generate_sample_pubmedbert.sh #PubMedBERT Sample Creation \nbash generate_sample_biovocabbert.sh #BioVocabBERT Sample Creation\n```\n\n### Run Pre-Training\n\nWe can now finally run pre-training for both our PubMedBERT replica and the BioVocabBERT model. To start \npre-training, run the following scripts. Make sure to adjust the ```train_micro_batch_size_per_gpu``` parameter \ndepending on the size of your GPUs. \n\n```\nbash pretrain_pubmedbert_replica.sh\nbash pretrain_biovocabbert.sh\n```\n\n## Downstream Biomedical Evaluation\n\nFor evaluating our model's you can either obtain them directly from HuggingFace or follow the steps above to reproduce the \npre-training steps.\n\nWe refer the reader interested in reproducing our downstream results to other repositories which we used directly for \nfine-tuning and evaluation purposes. Datasets can be obtained by following instructions in each repository.\n\nFor **Named Entity Recognition (NER)**, we use the [Few-Shot-BioIE](https://github.com/dki-lab/few-shot-bioIE/tree/main) repository \nto run both our fully-supervised and low-resource fine-tuning. Make sure that you set the ```kfolds``` parameter to 1 in order\nto use the true development set as we have in our work.\n\nFor Entity Linking (EL), we use the [SapBERT](https://github.com/cambridgeltl/sapbert) repository. We only use the code under \nthe ```evaluation``` directory which evaluates the zero-shot entity linking performance of biomedical language models.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fbio-tokenization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Fbio-tokenization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Fbio-tokenization/lists"}