{"id":30183617,"url":"https://github.com/dmis-lab/angel","last_synced_at":"2025-08-12T11:15:09.799Z","repository":{"id":295365043,"uuid":"849136137","full_name":"dmis-lab/ANGEL","owner":"dmis-lab","description":"Learning from Negative samples for Biomedical Generative Entity Linking","archived":false,"fork":false,"pushed_at":"2025-05-25T05:40:51.000Z","size":1696,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-25T06:32:22.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-29T03:33:06.000Z","updated_at":"2025-05-25T05:40:54.000Z","dependencies_parsed_at":"2025-05-25T06:43:20.903Z","dependency_job_id":null,"html_url":"https://github.com/dmis-lab/ANGEL","commit_stats":null,"previous_names":["dmis-lab/angel"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmis-lab/ANGEL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FANGEL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FANGEL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FANGEL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FANGEL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/ANGEL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FANGEL/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270048482,"owners_count":24518075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-12T11:15:04.114Z","updated_at":"2025-08-12T11:15:09.793Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch3 align=\"center\"\u003e\n\u003cp\u003eANGEL\n\u003ca href=\"https://github.com/dmis-lab/ANGEL/blob/main/LICENSE\"\u003e\n   \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/badge/license-GPLv3-blue\"\u003e\n\u003c/a\u003e\n\u003c/h3\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003cp\u003eLe\u003cb\u003ea\u003c/b\u003erning from \u003cb\u003eN\u003c/b\u003eegative Samples in \u003cb\u003eG\u003c/b\u003eenerative Biomedical \u003cb\u003eE\u003c/b\u003entity \u003cb\u003eL\u003c/b\u003einking Framework\u003c/p\u003e\n\u003c/div\u003e\n\n![Main Image](https://github.com/dmis-lab/ANGEL/blob/main/assets/main_figure_git.png)\n\n---\n## Introduction\n**ANGEL** is a novel framework designed to enhance generative biomedical entity linking (BioEL) by incorporating both positive and negative samples during training. Traditional generative models primarily focus on positive samples during training, which can limit their ability to distinguish between similar entities. We address this limitation by using *direct preference optimization* with negative samples, significantly enhancing model accuracy.\n\nKey features of ANGEL include:\n- **Negative-aware Learning**: Enhances the model's ability to differentiate between similar entities by using both correct and incorrect examples during training.\n- **Memory Efficiency**: Retains the low memory footprint characteristic of generative models by avoiding the need for large pre-computed entity embeddings.\n\nFor a detailed description of our method, please refer to our paper: \n- **Paper**: [Learning from Negative Samples in Generative Biomedical Entity Linking](https://arxiv.org/abs/2408.16493).\n\n## Requirements\nANGEL requires two separate virtual environments: one for *positive-only training* and another for *negative-aware training*.\nPlease ensure that CUDA version 11.1 is installed for optimal performance.\n\nTo set up the environments and install the required dependencies, run the following script:\n\n```bash\nbash script/environment/set_environment.sh\n```\n\n## Dataset Preparation\nMost datasets (i.e., NCBI-disease, BC5CDR, COMETA, and AskAPatient) were used as provided by GenBioEL, and MedMentions was processed in a similar manner using the GenBioEL code. \nIf you need the pre-processing code, you can find it in the [GenBioEL](https://github.com/Yuanhy1997/GenBioEL) repository. \nTo download these datasets and set up the experimental environment, execute the following steps:\n\n```bash\nbash script/dataset/process_dataset.sh\n```\n\n### Dataset Format\nFor training, prepare the data in the following format:\n\n- source: Input text including marked mentions with **START** and **END**.\n- ex) ['Ocular manifestations of **START** juvenile rheumatoid arthritis **END**.']\n- target: Pair of the prefix ['mention is'] and the ['target entity'].\n- ex) ['juvenile rheumatoid arthritis is ', 'juvenile rheumatoid arthritis']\n\nA Prefix Tree (or **Trie**) is a type of tree data structure used to efficiently store and manage a set of strings. \nEach node in the Trie represents a single token, and entire strings are formed by tracing a path from the root to a specific node.\nIn this context, we tokenize target entities and build a Trie structure to restrict the output space to the target knowledge base. \nTo construct the Trie, you need to create a target_kb.json file formatted as a dictionary like below.\n\n```json\n{\n  \"C565588\": [\"epidermolysis bullosa with diaphragmatic hernia\"], \n  \"C567755\": [\"tooth agenesis selective 6\", \"sthag6\"], \n  \"C565584\": [\"epithelial squamous dysplasia keratinizing desquamative of urinary tract\"],\n  ...\n}\n```\n\nTo experiment with your own dataset, preprocess it in the format described above.\n\n\n## Pre-training\n\n#### Positive-Only Training\n\nWe conducted positive-only pre-training using the code from [GenBioEL](https://github.com/Yuanhy1997/GenBioEL). \nIf you wish to replicate this, follow the instructions provided in the GenBioEL repository.\n\n #### Negative-Aware Training\n\n\u003c!-- Negative-aware pre-training was conducted using the code from [alignment-handbook](https://github.com/huggingface/alignment-handbook). \nThis step refines the model’s ability to differentiate between closely related entities by learning from negative examples. --\u003e\n\nNegative-aware pre-training code will be open-sourced shortly.\nThe pre-trained model utilized in this project is uploaded on Hugging Face. \nIf you want to use negative-aware pre-trained model, you can use the model with the following script:\n\n```python\nfrom transformers import AutoModel\n\n# Load the model\nmodel = AutoModel.from_pretrained(\"dmis-lab/ANGEL_pretrained\")\n```\n\n## Fine-tuning\n\n### Positive-Only Training\n\nTo fine-tune downstream dataset using positive-only training, run:\n```bash\nDATASET=ncbi # bc5cdr, cometa, aap, mm\nLEARNING_RATE=3e-7 # 1e-5, 2e-5\nSTEPS=20000 # 30000, 40000\n\nbash script/train/train_positive.sh $DATASET $LEARNING_RATE $STEPS\n```\nThe script for other datasets is in the train_positive.sh file.\n\n### Negative-Aware Training\n\nFor negative-aware fine-tuning on the downstream dataset, execute:\n```bash\nDATASET=ncbi # bc5cdr, cometa, aap, mm\nLEARNING_RATE=2e-5 # 1e-5, 2e-5\nBATCH_SIZE=16 # 8, 16, 32\n\nbash script/train/train_negative.sh $DATASET $LEARNING_RAT $BATCH_SIZE\n```\nThe script for other datasets is in the train_negative.sh file.\n\n## Evaluation\n\n### Running Inference with the Best Model on Huggingface\n\nTo perform inference with our best model hosted on Huggingface, use the following script:\n```bash\nDATASET=ncbi # bc5cdr, cometa, aap, mm\n\nbash script/inference/inference.sh $DATASET\n```\n\n### Output format\n\nThe results file in your model folder contains the final scores:\n```json\n{\n  \"count_top1\": 92.812,\n  \"count_top2\": 94.062,\n  \"count_top3\": 95.208,\n  \"count_top4\": 95.625,\n  \"count_top5\": 95.729,\n  ...\n}\n```\n\nAdditionally, the file lists candidates for each mention, indicating correctness:\n```json\n{\n  \"correctness\": \"correct\",\n  \"given_mention\": \"non inherited breast carcinomas\",\n  \"result\": [\n    \" breast carcinomas\",\n    \" breast cancer\",\n    ...\n  ],\n  \"cui_label\": [\n    \"D001943\"\n  ],\n  \"cui_result\": [\n    [\"D001943\"],\n    [\"114480\",\"D001943\"],\n    ...\n    ]\n}\n```\n\n### Scores\n\nWe utilized five popular BioEL benchmark datasets: NCBI-disease (NCBI), BC5CDR, COMETA, AskAPatient (AAP), and MedMentions ST21pv (MM-ST21pv).\n\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"5\"\u003e\n  \u003cthead\u003e\n    \u003ctr\u003e\n      \u003cth rowspan=\"2\"\u003eModel\u003c/th\u003e\n      \u003cth colspan=\"5\"\u003eDataset\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eNCBI\u003c/th\u003e\n      \u003cth\u003eBC5CDR\u003c/th\u003e\n      \u003cth\u003eCOMETA\u003c/th\u003e\n      \u003cth\u003eAAP\u003c/th\u003e\n      \u003cth\u003eMM-ST21pv\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\u003ca href=\"https://github.com/dmis-lab/BioSyn\"\u003eBioSYN\u003c/a\u003e (Sung et al., 2020)\u003c/td\u003e\n      \u003ctd\u003e91.1\u003c/td\u003e\n      \u003ctd\u003e-\u003c/td\u003e\n      \u003ctd\u003e71.3\u003c/td\u003e\n      \u003ctd\u003e82.6\u003c/td\u003e\n      \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\u003ca href=\"https://github.com/cambridgeltl/sapbert\"\u003eSapBERT\u003c/a\u003e (Liu et al., 2021)\u003c/td\u003e\n      \u003ctd\u003e92.3\u003c/td\u003e\n      \u003ctd\u003e-\u003c/td\u003e\n      \u003ctd\u003e75.1\u003c/td\u003e\n      \u003ctd\u003e89.0\u003c/td\u003e\n      \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\u003ca href=\"https://github.com/Yuanhy1997/GenBioEL\"\u003eGenBioEL\u003c/a\u003e (Yuan et al., 2022b)\u003c/td\u003e\n      \u003ctd\u003e91.0\u003c/td\u003e\n      \u003ctd\u003e93.1\u003c/td\u003e\n      \u003ctd\u003e80.9\u003c/td\u003e\n      \u003ctd\u003e89.3\u003c/td\u003e\n      \u003ctd\u003e70.7\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd\u003eANGEL (\u003cb\u003eOurs\u003c/b\u003e)\u003c/td\u003e\n      \u003ctd\u003e\u003cb\u003e92.8\u003c/td\u003e\n      \u003ctd\u003e\u003cb\u003e94.5\u003c/td\u003e\n      \u003ctd\u003e\u003cb\u003e82.8\u003c/td\u003e\n      \u003ctd\u003e\u003cb\u003e90.2\u003c/td\u003e\n      \u003ctd\u003e\u003cb\u003e73.3\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n- The scores of GenBioEL were reproduced.\n- We excluded the performance of BioSYN and SapBERT on BC5CDR, as they were evaluated separately on the chemical and disease subsets, differing from our settings.\n\n## Model Checkpoints\n\n| Pre-trained | NCBI | BC5CDR | COMETA | MM-ST21pv | \n|:------------:|:-----:|:-----:|:-----:|:-----:| \n| [ANGEL_pretrained](https://huggingface.co/dmis-lab/ANGEL_pretrained)| [ANGEL_ncbi](https://huggingface.co/dmis-lab/ANGEL_ncbi) | [ANGEL_bc5cdr](https://huggingface.co/dmis-lab/ANGEL_bc5cdr) | [ANGEL_cometa](https://huggingface.co/dmis-lab/ANGEL_cometa) |  [ANGEL_mm](https://huggingface.co/dmis-lab/ANGEL_mm) |\n\n- The AskAPatient dataset does not have a predefined split; therefore, we utilized a 10-fold cross-validation method to evaluate our model. As a result, there are 10 model checkpoints corresponding to the AskAPatient dataset. Due to this, we have not open-sourced the checkpoints for this dataset.\n\n\n## Direct Use\n\nTo run the model without any need for a preprocessed dataset, you can use the run_sample.sh script. \n\n```bash\nbash script/inference/run_sample.sh ncbi\n```\n\nIf you want to modify the sample or input, you can change the script in run_sample.py.\n\n#### Define Your Inputs\n- input_sentence: The sentence that includes the entity you want to normalize. Make sure the entity is enclosed within **START** and **END** markers.\n- prefix_sentence: A sentence that introduces the context. This should follow the format \"**entity** is\".\n- candidates: A list of candidate entities that the model will use to attempt normalization.\n\n```python\nif __name__ == '__main__':\n    \n    config = get_config()\n    \n    # Define the input sentence, marking the entity of interest with START and END\n    input_sentence = \"The r496h mutation of arylsulfatase a does not cause START metachromatic leukodystrophy END\"\n    # Define the prefix sentence to provide context\n    prefix_sentence = \"Metachromatic leukodystrophy is\"\n    # List your candidate entities for normalization\n    candidates = [\"adrenoleukodystrophy\", \"thrombosis\", \"anemia\", \"huntington disease\", \"leukodystrophy metachromatic\"]\n    \n    run_sample(config, input_sentence, prefix_sentence, candidates)\n```\n\n```terminal\n    # Expected Output\n    Input  :  The r496h mutation of arylsulfatase a does not cause START metachromatic leukodystrophy END.\n    Output :  Metachromatic leukodystrophy is leukodystrophy metachromatic\n```\n\nBy modifying input_sentence, prefix_sentence, and candidates, you can tailor the examples used by the model to fit your specific needs.\n\n\n## Citations\n\nIf you are interested in our work, please cite:\n```bibtex\n@article{kim2024learning,\n  title={Learning from Negative Samples in Generative Biomedical Entity Linking},\n  author={Kim, Chanhwi and Kim, Hyunjae and Park, Sihyeon and Lee, Jiwoo and Sung, Mujeen and Kang, Jaewoo},\n  journal={arXiv preprint arXiv:2408.16493},\n  year={2024}\n}\n```\n\n## Acknowledgement\nThis code includes modifications based on the code of [GenBioEL](https://github.com/Yuanhy1997/GenBioEL). We are grateful to the authors for providing their code/models as open-source software.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fangel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fangel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fangel/lists"}