{"id":21361736,"url":"https://github.com/ganjinzero/biobart","last_synced_at":"2025-07-13T02:32:32.975Z","repository":{"id":39977585,"uuid":"465219414","full_name":"GanjinZero/BioBART","owner":"GanjinZero","description":"BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model [ACL-BioNLP 2022]","archived":false,"fork":false,"pushed_at":"2022-10-26T07:52:24.000Z","size":120,"stargazers_count":38,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-03-05T10:19:08.913Z","etag":null,"topics":["biomedical","generative","pretrained-language-model"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2204.03905","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GanjinZero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-02T08:29:27.000Z","updated_at":"2023-03-01T16:17:55.000Z","dependencies_parsed_at":"2022-09-01T01:02:07.286Z","dependency_job_id":null,"html_url":"https://github.com/GanjinZero/BioBART","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GanjinZero%2FBioBART","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GanjinZero%2FBioBART/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GanjinZero%2FBioBART/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GanjinZero%2FBioBART/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GanjinZero","download_url":"https://codeload.github.com/GanjinZero/BioBART/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225850218,"owners_count":17534067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["biomedical","generative","pretrained-language-model"],"created_at":"2024-11-22T06:11:15.424Z","updated_at":"2024-11-22T06:11:15.982Z","avatar_url":"https://github.com/GanjinZero.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BioBART\nBioBART: Pretraining and Evaluation of A Biomedical Generative Language Model [ACL-BioNLP 2022] [Paper](https://arxiv.org/abs/2204.03905)\n\nTsinghua University \\\u0026 International Digital Economy Academy.\n\n# Model Checkpoints\n\n## BioBART\n\n- Base Version (6 + 6 Layers): **GanjinZero/biobart-base** or **IDEA-CCNL/Yuyuan-Bart-139M** (same model)\n- Large Version (12 + 12 Layers): **GanjinZero/biobart-large** or **IDEA-CCNL/Yuyuan-Bart-400M** (same model)\n\nP.S. Yuyuan is a character in novel Fengshenbang. [Chinese Introduction](https://baike.baidu.com/item/%E9%A4%98%E5%85%83/968026) \\ [English Introduction](https://en.wikisource.org/wiki/Portal:Investiture_of_the_Gods/Chapter_75)\n\nTwo line usages:\n```python\nmodel = AutoModel.from_pretrained('GanjinZero/biobart-base')\n# model = AutoModel.from_pretrained('GanjinZero/biobart-large')\ntok = AutoTokenizer.from_pretrained('GanjinZero/biobart-base')\n```\n\n## BioBART-v2\n\nNew generative language model with domain-adaptive pre-training on biomedical corpus BioBART-v2 is released. \nCompared to BioBART, the main difference of BioBART-v2 is using a cross-domain vocabulary of 85,401 tokens and pre-training for longer steps. \n\nThe detailed implementation introduction and experiment results on bimedical downstream tasks are [here](BioBART-v2.pdf).\n\n- Base Version (6 + 6 Layers): **GanjinZero/biobart-v2-base**\n- Large Version (12 + 12 Layers): **GanjinZero/biobart-v2-large**\n\nTwo line usages:\n```python\nmodel = AutoModel.from_pretrained('GanjinZero/biobart-v2-base')\n# model = AutoModel.from_pretrained('GanjinZero/biobart-v2-large')\ntok = AutoTokenizer.from_pretrained('GanjinZero/biobart-v2-base')\n```\n\n# Citation\n```bibtex\n@inproceedings{yuan-etal-2022-biobart,\n    title = \"{B}io{BART}: Pretraining and Evaluation of A Biomedical Generative Language Model\",\n    author = \"Yuan, Hongyi  and\n      Yuan, Zheng  and\n      Gan, Ruyi  and\n      Zhang, Jiaxing  and\n      Xie, Yutao  and\n      Yu, Sheng\",\n    booktitle = \"Proceedings of the 21st Workshop on Biomedical Language Processing\",\n    month = may,\n    year = \"2022\",\n    address = \"Dublin, Ireland\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.bionlp-1.9\",\n    pages = \"97--109\",\n    abstract = \"Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fganjinzero%2Fbiobart","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fganjinzero%2Fbiobart","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fganjinzero%2Fbiobart/lists"}